15:10:09 <mboddu> #startmeeting RELENG (2020-05-19) 15:10:09 <zodbot> Meeting started Tue May 19 15:10:09 2020 UTC. 15:10:09 <zodbot> This meeting is logged and archived in a public location. 15:10:09 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:10:09 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:10:09 <zodbot> The meeting name has been set to 'releng_(2020-05-19)' 15:10:09 <mboddu> #meetingname releng 15:10:09 <zodbot> The meeting name has been set to 'releng' 15:10:09 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec 15:10:09 <mboddu> #topic init process 15:10:09 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz 15:10:21 <nirik> morning 15:10:38 <pingou> ó/ 15:10:56 <mboddu> Sorry for the late start 15:11:08 <pingou> we'll forgive you, this time 15:11:10 <pingou> :-p 15:11:22 <mboddu> Thanks pingou :) 15:11:31 <mboddu> Lets get started 15:11:33 <pingou> we're that generous today 15:11:38 <mboddu> #topic #9459 Provide installed_pkgs.log for builds 15:11:45 <mboddu> #link https://pagure.io/releng/issue/9459 15:11:52 <nirik> needs investigation. 15:12:00 <mboddu> pingou: Oh, how generous are we talking here? :P 15:12:02 <pingou> post colo-move? 15:12:04 <nirik> personally I'd push this off until after the move 15:12:19 <nirik> I sure don't have any cycles to look... but if someone else wants to, sure... 15:13:00 <mboddu> I can take a look at it, but after my fmc work as its almost done 15:13:23 <mboddu> Which I am targetting to deploy by end of this week 15:14:19 <mboddu> #info It needs some investigation, we will take a look at it when we get a chance, probably next week. 15:14:47 <pingou> that gives 1 week buffer before reduced capacity 15:15:33 <nirik> fmc work? 15:16:05 <mboddu> fedora minimal compose 15:16:20 <mboddu> Too many abbreviations :) 15:16:57 <nirik> I was reading fedora media creator... and wondering what you were doing with it. 15:17:02 <nirik> :) 15:17:08 <pingou> Fedora Media Center 15:17:12 <pingou> but that's another project ;-) 15:17:21 <mboddu> pingou: Ohhh, thats a good one :) 15:17:33 <pingou> though on topic considering these times :) 15:17:39 <mboddu> #topic #8243 Consider enabling safe-failed-tree plugin 15:17:45 <mboddu> #link https://pagure.io/releng/issue/8243 15:18:06 <mboddu> nirik: Any update on this? It came up in the pagure grooming meeting today. 15:18:30 <nirik> I'm fine with enabling it... did we actually test it in staging? 15:18:52 <mboddu> I dont think so, should we enable it in stg and test it? 15:18:58 <nirik> sure. 15:19:12 <nirik> we could enable it when we update koji/mock next 15:20:00 <mboddu> Should I assign the ticket to you? 15:20:26 * pingou would say no 15:20:50 <pingou> just to keep Kevin and Stephen off tickets for the moment :) 15:20:53 <mboddu> Actually, maybe we should shadow the koji upgrade process when it happens next 15:21:21 <mboddu> pingou: It can wait until colo move, but currently Kevin is the only one who can upgrade koji 15:21:42 <mboddu> ^^ we = anyone interested 15:22:10 <pingou> +1 15:22:22 <nirik> we should only assign things when they are being done. ;) 15:22:46 <nirik> but we should figure out when to do the koji upgrade I suppose. 15:23:49 <nirik> I guess we are still waiting on mock. 15:23:53 <mboddu> Okay, then I will leave it as is 15:24:22 * nirik is tempted to just do it with the move, but it does add more variables. doing it before means more work away from the move tho 15:24:59 <mboddu> nirik: I would suggest not to do it before the move, once we get to a stable state after the move, then we can do whatever upgrades we want 15:25:13 <pingou> so as an FYI Aoife and I are writing an email to be sent to devel-announce about the colo move and what it means for people 15:25:51 <pingou> which makes it as clear as possible that they should expect reduced capacity, longer reply-time on tickets/requests, longer time to deploy new releases & so on 15:26:14 <nirik> pingou: pretty please let me review that before you send? 15:26:34 <nirik> mboddu: the problem then becomes we have to install the older stuff in the new dc. 15:26:45 <nirik> older koji, f31 builders, older mock 15:26:48 <pingou> so I think we can leverage this announcement to tell people that we're working on that and thus while we've heard the request it'll take a little time for us to get to it 15:26:51 <pingou> nirik: of course 15:26:53 <nirik> some of which are not in updates anymore 15:27:24 <mboddu> pingou: Can you also cc me when you are sending it to nirik for review? Thanks. 15:27:46 <nirik> this request is... not high priorty. It's waited a year... and I filed it, and don't care when it's done. ;) 15:27:47 <pingou> nirik: mboddu: here is the current draft: https://hackmd.io/NiOi2IlZTl2VzVH0F2p5VQ 15:29:15 <mboddu> nirik: Hmmm, I see, I will let you decide on that matter then, you are the expert :) 15:29:23 <nirik> june 8th to june 12 is the week we move services. 15:29:36 <pingou> nirik: and start unrack no? 15:29:39 <nirik> june 15th will be the deracking and shipping (after everything is moved) 15:29:42 <nirik> no. 15:29:46 <pingou> ok 15:30:05 <nirik> we also wrote up https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?edit a while back... 15:30:12 <pingou> nirik: when is reduced capacity starting though? 15:30:44 <pingou> yeah, we may want to link to that doc 15:30:57 <nirik> it needs work, we made it a while back and need to update it. 15:31:11 <nirik> the week of the 8th... depending on the service. 15:31:31 <nirik> we also should note the week of the 8th there will be outages. Stuff will be down as we move it. 15:31:35 <pingou> I'll reword a bit then but keep the week of the 8th 15:31:51 <pingou> that may make the rewording easier :) 15:31:54 <nirik> I'm hoping to come up with a more detailed doc about that plan. 15:33:27 <mboddu> nirik: Update the existing doc (https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?edit) and pingou can link that doc in the email 15:33:40 <pingou> +1 15:33:49 <pingou> sorry for derailing the topic 15:33:50 <nirik> sure, I'd like to. 15:34:30 <pingou> I wanted to bring it but at the end of the meeting, I just jumped on this as an opportunity to tell people: we'll look at this once that <announce> is done 15:34:38 <nirik> can try after this meeting, although I think I have more meetings. 15:35:12 <mboddu> nirik: Sure, definitely when you have time, but they can link the doc and ask them to check it for more updates 15:35:19 <mboddu> Or detailed info 15:35:20 <nirik> anyhow... not sure about koji. 15:35:28 <mboddu> Yeah 15:35:33 * pingou can't edit that note 15:35:48 <mboddu> pingou: Which note? 15:35:54 <pingou> the list of services impacted 15:36:10 <nirik> a) update as soon as mock is ready even tho it takes time out from dc move work and get it over with, b) wait and just move to new koji in the new dc... but then might be hard to tell a koji/mock bug from other issues... 15:37:34 <nirik> I think smooge has to add you... can you ask him? 15:37:47 <pingou> who can do a) ? 15:38:02 <nirik> IMHO, perhaps we could fold the two docs together? 15:38:46 <nirik> I guess anyone in main. ;) There's a trick or two... but also a playbook (with one bug to work around) 15:38:49 <mboddu> I suggest not to, we can keep updating your doc and pingou doc can be sent as an email 15:39:46 <nirik> ok. 15:39:58 <mboddu> Not to sound as a pessimist here, but if something goes wrong when someone in main does it, we might be asking for more time from nirik to fix it. 15:40:01 * nirik is leaning toward just doing the koji update in the new dc... 15:40:41 <nirik> f32 builders with new mock, etc. 15:41:07 <mboddu> nirik: As I said before, I trust you :) 15:41:19 <pingou> I am as torned as you 15:41:56 <pingou> I don't like having multiple changes at once, but I don't like taking your time away from the move 15:42:13 <pingou> so if that can't wait for post-move, than it's the less of two evils 15:42:37 <pingou> (least*?) 15:42:48 * mboddu agrees with pingou 15:43:05 <pingou> the advantage of doing it at the new DC is that we will be hands down in that at that time 15:43:21 <pingou> (ie: no/less context switch) 15:43:26 <nirik> yeah. 15:43:36 <nirik> and people will also hopefully be expecting/watching for issues. ;) 15:43:48 <mboddu> Right 15:43:52 <nirik> so lets do that... just move to the new stuff when we move to the new dc. 15:43:58 <mboddu> ack 15:44:09 <pingou> deal! 15:44:27 <mboddu> #info nirik is planning up upgrade koji, mock, ... when we move to the new dc 15:44:39 <mboddu> While we are on the topic of koji upgrades 15:45:12 <mboddu> #topic koji 1.22 doesn't support krb_login, it supports only gssapi_login 15:45:31 <mboddu> So, this affects fedpkg, releng scripts 15:45:35 <mboddu> And maybe bodhi? 15:45:38 <mboddu> pingou: ^? 15:45:47 <nirik> well, it's rpkg... I filed a bug there a while back 15:45:49 <pingou> bodhi uses a keytab so yes 15:45:56 <nirik> https://pagure.io/rpkg/issue/498 15:46:55 <pingou> so that should save fedpkg and the releng scripts? 15:47:06 <mboddu> pingou: Nope, only fedpkg 15:47:09 <nirik> didn't think about bodhi, but yes, it needs fixing 15:47:12 <mboddu> Releng scripts needs to be updated as well 15:47:31 * mboddu can create ticket with bodhi and releng 15:47:34 <pingou> thanks 15:47:50 <mboddu> And, am I missing any other tools/services? 15:48:12 <pingou> simple-koji-ci, likely the CI pipelines 15:48:19 <pingou> koschei 15:48:40 <pingou> anything that builds in koji I guess 15:48:55 <mboddu> Oh, I thought koschei uses fedpkg, but might be not 15:49:18 <pingou> to be checked 15:49:43 <nirik> koschei doesn't seem to have any krb_login in it's code 15:50:10 <nirik> it does scratch builds, so possibly it doesn't login ever anyhow. 15:50:20 <nirik> oh, I guess it must 15:50:22 <nirik> anyhow. 15:51:13 <mboddu> I see a test - https://github.com/fedora-infra/koschei/blob/6339f8bad9ad28afbbf44e134c9e9c6cfa0da0d0/test/test_config.cfg#L14 15:51:19 <mboddu> But nothing in the code though 15:51:35 * mboddu creates a ticket with koschei as well, just to check 15:51:45 <nirik> it uses ssl_login it seems 15:52:15 <nirik> ie, a cert... which I thought we stopped doing, but possibly koschei predates that 15:52:45 <mboddu> We can ask them to update to gssapi_login too :) 15:53:01 <nirik> sure 15:53:48 <mboddu> #info mboddu will be creating tickets against bodhi, releng, koschei to update to gssapi_login to authenticate to koji 15:53:54 <pingou> do we already support gssapi? 15:55:05 <pingou> which I think nirik just answered in -apps (yes) 15:55:31 <nirik> yeah, I think so... 15:56:31 <nirik> thanks for bringing this up mboddu 15:57:05 <mboddu> np :) 15:57:06 <mboddu> Moving on 15:57:10 <mboddu> #topic Open Floor 15:57:16 <mboddu> siddharthvipul: You got something? 15:57:16 <siddharthvipul> finally 15:57:18 <siddharthvipul> \o/ 15:57:20 <siddharthvipul> https://pagure.io/releng/issue/8105 15:57:24 <siddharthvipul> yes, what is this lol 15:57:50 <siddharthvipul> I see comment from Miro that the workaround nirik gave won't work 15:57:51 <mboddu> siddharthvipul: That was me :) 15:58:03 <siddharthvipul> and I am not sure of a way to do this 15:58:25 <siddharthvipul> mboddu: I see, thank you.. but still I have no clue what to document 15:58:27 <nirik> so hum, yeah... if gating checks for newer/older? I don't know if it does 15:59:06 <nirik> pingou: would you know? 15:59:18 <mboddu> They can still tag the older nvr to -updates-candidate tag and it will follow the gating 15:59:44 <siddharthvipul> is there something I should try and play around with to know, you see! I am not a very good packager yet (and definitely not someone who knows copr/koji/bodhi very well) 15:59:51 <nirik> say rawhide has foo-1.0-1.fc33 in it, I build foo-2.0-1.fc33, but it's horrible and broken, can I tag foo-1.0-1.fc33 in updates-candidate and have it go thru and become last tagged in f33? 15:59:56 <pingou> the comment from Miro doesn't say if the whole idea is wrong or just the tag 16:00:28 * mboddu guessing just tagging 16:00:34 <nirik> I'm leary of opening rawhide for untagging... we are not supposed to untag things that have already gone out in rawhide composes... 16:00:52 <mboddu> Yes, I am not okay with it either 16:00:53 <nirik> but this has a similar effect. 16:01:55 <pingou> I'd be inclined to say that this should be caught by CI 16:02:42 <nirik> the 'older' version you mean? or whatever issue caused them to want to untag the new build? ;) or both! 16:02:50 * mboddu does untagging when requested, but I check if the build was shipped out or not 16:02:55 <pingou> the later :) 16:03:47 <siddharthvipul> I guess before documenting the process, we would like to clear if the process should happen at all? 16:04:02 <mboddu> pingou: Maybe not always - https://pagure.io/releng/issue/9461 16:04:26 <mboddu> siddharthvipul: Yes, I wanted to talk to you about it, but good you brought it up now 16:04:41 <siddharthvipul> mboddu: I don't mind taking it somewhere else later 16:05:05 <pingou> mboddu: fair 16:05:38 <nirik> yeah, I guess we should see if the process actually works still... and we should decide if it's what we want to do 16:05:48 <mboddu> pingou: So "say rawhide has foo-1.0-1.fc33 in it, I build foo-2.0-1.fc33, but it's horrible and broken, can I tag foo-1.0-1.fc33 in updates-candidate and have it go thru and become last tagged in f33?" works? 16:05:59 <mboddu> Or it needs to be tested? 16:06:31 <pingou> mboddu: bodhi may not be happy with this 16:06:41 <pingou> since there was already an update with this build 16:06:41 <mboddu> Okay 16:07:02 <pingou> worth investigating 16:07:07 <mboddu> Okay 16:07:10 <pingou> but not 100% sure it'll work 16:07:10 <mboddu> Lets test it 16:07:28 * pingou checking is cheatsheet: https://pingou.fedorapeople.org/gating_rawhide/Simplified_single_build_update-20191021.png 16:07:47 <mboddu> pingou: Haha, I opened https://docs.fedoraproject.org/en-US/rawhide-gating/single-builds/#_simplified_diagram_of_the_single_build_updates_workflow :) 16:07:52 <mboddu> Same cheatsheet :) 16:07:54 <pingou> close enough ;-) 16:08:24 <nirik> I wonder if we could make a hub policy that would let untag work for 1 day... 16:08:30 <mboddu> Anyway, I covered up for the 10 min delay :) 16:08:56 <pingou> 1 day means it'll be in a compose always no? 16:09:08 <mboddu> nirik: That could be trickey 16:09:23 <nirik> yeah, it could be. depends on when it built and when the compose ran 16:09:24 <pingou> (but that's bikeshedding, allowing to untag withing X amount of time may be interesting) 16:09:39 <mboddu> Yup, trickey :) 16:10:01 * siddharthvipul slowly slides out of conversation :P 16:10:09 <mboddu> Haha :) 16:10:42 <mboddu> #info We will investigate if tagging of old build to -updates-candidate work and if it does, we will document the process 16:11:57 <siddharthvipul> awesome :) thank you all 16:12:09 <siddharthvipul> mboddu: I will catch you at other places to see if I can actually help 16:12:15 <siddharthvipul> mboddu++ 16:12:26 <mboddu> siddharthvipul: Anytime :) 16:12:31 <mboddu> Thanks everyone for joining 16:12:41 <mboddu> See you all on other channels :) 16:12:43 <mboddu> #endmeeting