15:10:09 <mboddu> #startmeeting RELENG (2020-05-19)
15:10:09 <zodbot> Meeting started Tue May 19 15:10:09 2020 UTC.
15:10:09 <zodbot> This meeting is logged and archived in a public location.
15:10:09 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:10:09 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:10:09 <zodbot> The meeting name has been set to 'releng_(2020-05-19)'
15:10:09 <mboddu> #meetingname releng
15:10:09 <zodbot> The meeting name has been set to 'releng'
15:10:09 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec
15:10:09 <mboddu> #topic init process
15:10:09 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz
15:10:21 <nirik> morning
15:10:38 <pingou> ó/
15:10:56 <mboddu> Sorry for the late start
15:11:08 <pingou> we'll forgive you, this time
15:11:10 <pingou> :-p
15:11:22 <mboddu> Thanks pingou :)
15:11:31 <mboddu> Lets get started
15:11:33 <pingou> we're that generous today
15:11:38 <mboddu> #topic #9459 Provide installed_pkgs.log for builds
15:11:45 <mboddu> #link https://pagure.io/releng/issue/9459
15:11:52 <nirik> needs investigation.
15:12:00 <mboddu> pingou: Oh, how generous are we talking here? :P
15:12:02 <pingou> post colo-move?
15:12:04 <nirik> personally I'd push this off until after the move
15:12:19 <nirik> I sure don't have any cycles to look... but if someone else wants to, sure...
15:13:00 <mboddu> I can take a look at it, but after my fmc work as its almost done
15:13:23 <mboddu> Which I am targetting to deploy by end of this week
15:14:19 <mboddu> #info It needs some investigation, we will take a look at it when we get a chance, probably next week.
15:14:47 <pingou> that gives 1 week buffer before reduced capacity
15:15:33 <nirik> fmc work?
15:16:05 <mboddu> fedora minimal compose
15:16:20 <mboddu> Too many abbreviations :)
15:16:57 <nirik> I was reading fedora media creator... and wondering what you were doing with it.
15:17:02 <nirik> :)
15:17:08 <pingou> Fedora Media Center
15:17:12 <pingou> but that's another project ;-)
15:17:21 <mboddu> pingou: Ohhh, thats a good one :)
15:17:33 <pingou> though on topic considering these times :)
15:17:39 <mboddu> #topic #8243 Consider enabling safe-failed-tree plugin
15:17:45 <mboddu> #link https://pagure.io/releng/issue/8243
15:18:06 <mboddu> nirik: Any update on this? It came up in the pagure grooming meeting today.
15:18:30 <nirik> I'm fine with enabling it... did we actually test it in staging?
15:18:52 <mboddu> I dont think so, should we enable it in stg and test it?
15:18:58 <nirik> sure.
15:19:12 <nirik> we could enable it when we update koji/mock next
15:20:00 <mboddu> Should I assign the ticket to you?
15:20:26 * pingou would say no
15:20:50 <pingou> just to keep Kevin and Stephen off tickets for the moment :)
15:20:53 <mboddu> Actually, maybe we should shadow the koji upgrade process when it happens next
15:21:21 <mboddu> pingou: It can wait until colo move, but currently Kevin is the only one who can upgrade koji
15:21:42 <mboddu> ^^ we = anyone interested
15:22:10 <pingou> +1
15:22:22 <nirik> we should only assign things when they are being done. ;)
15:22:46 <nirik> but we should figure out when to do the koji upgrade I suppose.
15:23:49 <nirik> I guess we are still waiting on mock.
15:23:53 <mboddu> Okay, then I will leave it as is
15:24:22 * nirik is tempted to just do it with the move, but it does add more variables. doing it before means more work away from the move tho
15:24:59 <mboddu> nirik: I would suggest not to do it before the move, once we get to a stable state after the move, then we can do whatever upgrades we want
15:25:13 <pingou> so as an FYI Aoife and I are writing an email to be sent to devel-announce about the colo move and what it means for people
15:25:51 <pingou> which makes it as clear as possible that they should expect reduced capacity, longer reply-time on tickets/requests, longer time to deploy new releases & so on
15:26:14 <nirik> pingou: pretty please let me review that before you send?
15:26:34 <nirik> mboddu: the problem then becomes we have to install the older stuff in the new dc.
15:26:45 <nirik> older koji, f31 builders, older mock
15:26:48 <pingou> so I think we can leverage this announcement to tell people that we're working on that and thus while we've heard the request it'll take a little time for us to get to it
15:26:51 <pingou> nirik: of course
15:26:53 <nirik> some of which are not in updates anymore
15:27:24 <mboddu> pingou: Can you also cc me when you are sending it to nirik for review? Thanks.
15:27:46 <nirik> this request is... not high priorty. It's waited a year... and I filed it, and don't care when it's done. ;)
15:27:47 <pingou> nirik: mboddu: here is the current draft: https://hackmd.io/NiOi2IlZTl2VzVH0F2p5VQ
15:29:15 <mboddu> nirik: Hmmm, I see, I will let you decide on that matter then, you are the expert :)
15:29:23 <nirik> june 8th to june 12 is the week we move services.
15:29:36 <pingou> nirik: and start unrack no?
15:29:39 <nirik> june 15th will be the deracking and shipping (after everything is moved)
15:29:42 <nirik> no.
15:29:46 <pingou> ok
15:30:05 <nirik> we also wrote up https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?edit a while back...
15:30:12 <pingou> nirik: when is reduced capacity starting though?
15:30:44 <pingou> yeah, we may want to link to that doc
15:30:57 <nirik> it needs work, we made it a while back and need to update it.
15:31:11 <nirik> the week of the 8th... depending on the service.
15:31:31 <nirik> we also should note the week of the 8th there will be outages. Stuff will be down as we move it.
15:31:35 <pingou> I'll reword a bit then but keep the week of the 8th
15:31:51 <pingou> that may make the rewording easier :)
15:31:54 <nirik> I'm hoping to come up with a more detailed doc about that plan.
15:33:27 <mboddu> nirik: Update the existing doc (https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?edit) and pingou can link that doc in the email
15:33:40 <pingou> +1
15:33:49 <pingou> sorry for derailing the topic
15:33:50 <nirik> sure, I'd like to.
15:34:30 <pingou> I wanted to bring it but at the end of the meeting, I just jumped on this as an opportunity to tell people: we'll look at this once that <announce> is done
15:34:38 <nirik> can try after this meeting, although I think I have more meetings.
15:35:12 <mboddu> nirik: Sure, definitely when you have time, but they can link the doc and ask them to check it for more updates
15:35:19 <mboddu> Or detailed info
15:35:20 <nirik> anyhow... not sure about koji.
15:35:28 <mboddu> Yeah
15:35:33 * pingou can't edit that note
15:35:48 <mboddu> pingou: Which note?
15:35:54 <pingou> the list of services impacted
15:36:10 <nirik> a) update as soon as mock is ready even tho it takes time out from dc move work and get it over with, b) wait and just move to new koji in the new dc... but then might be hard to tell a koji/mock bug from other issues...
15:37:34 <nirik> I think smooge has to add you... can you ask him?
15:37:47 <pingou> who can do a) ?
15:38:02 <nirik> IMHO, perhaps we could fold the two docs together?
15:38:46 <nirik> I guess anyone in main. ;) There's a trick or two... but also a playbook (with one bug to work around)
15:38:49 <mboddu> I suggest not to, we can keep updating your doc and pingou doc can be sent as an email
15:39:46 <nirik> ok.
15:39:58 <mboddu> Not to sound as a pessimist here, but if something goes wrong when someone in main does it, we might be asking for more time from nirik to fix it.
15:40:01 * nirik is leaning toward just doing the koji update in the new dc...
15:40:41 <nirik> f32 builders with new mock, etc.
15:41:07 <mboddu> nirik: As I said before, I trust you :)
15:41:19 <pingou> I am as torned as you
15:41:56 <pingou> I don't like having multiple changes at once, but I don't like taking your time away from the move
15:42:13 <pingou> so if that can't wait for post-move, than it's the less of two evils
15:42:37 <pingou> (least*?)
15:42:48 * mboddu agrees with pingou
15:43:05 <pingou> the advantage of doing it at the new DC is that we will be hands down in that at that time
15:43:21 <pingou> (ie: no/less context switch)
15:43:26 <nirik> yeah.
15:43:36 <nirik> and people will also hopefully be expecting/watching for issues. ;)
15:43:48 <mboddu> Right
15:43:52 <nirik> so lets do that... just move to the new stuff when we move to the new dc.
15:43:58 <mboddu> ack
15:44:09 <pingou> deal!
15:44:27 <mboddu> #info nirik is planning up upgrade koji, mock, ... when we move to the new dc
15:44:39 <mboddu> While we are on the topic of koji upgrades
15:45:12 <mboddu> #topic koji 1.22 doesn't support krb_login, it supports only gssapi_login
15:45:31 <mboddu> So, this affects fedpkg, releng scripts
15:45:35 <mboddu> And maybe bodhi?
15:45:38 <mboddu> pingou: ^?
15:45:47 <nirik> well, it's rpkg... I filed a bug there a while back
15:45:49 <pingou> bodhi uses a keytab so yes
15:45:56 <nirik> https://pagure.io/rpkg/issue/498
15:46:55 <pingou> so that should save fedpkg and the releng scripts?
15:47:06 <mboddu> pingou: Nope, only fedpkg
15:47:09 <nirik> didn't think about bodhi, but yes, it needs fixing
15:47:12 <mboddu> Releng scripts needs to be updated as well
15:47:31 * mboddu can create ticket with bodhi and releng
15:47:34 <pingou> thanks
15:47:50 <mboddu> And, am I missing any other tools/services?
15:48:12 <pingou> simple-koji-ci, likely the CI pipelines
15:48:19 <pingou> koschei
15:48:40 <pingou> anything that builds in koji I guess
15:48:55 <mboddu> Oh, I thought koschei uses fedpkg, but might be not
15:49:18 <pingou> to be checked
15:49:43 <nirik> koschei doesn't seem to have any krb_login in it's code
15:50:10 <nirik> it does scratch builds, so possibly it doesn't login ever anyhow.
15:50:20 <nirik> oh, I guess it must
15:50:22 <nirik> anyhow.
15:51:13 <mboddu> I see a test - https://github.com/fedora-infra/koschei/blob/6339f8bad9ad28afbbf44e134c9e9c6cfa0da0d0/test/test_config.cfg#L14
15:51:19 <mboddu> But nothing in the code though
15:51:35 * mboddu creates a ticket with koschei as well, just to check
15:51:45 <nirik> it uses ssl_login it seems
15:52:15 <nirik> ie, a cert... which I thought we stopped doing, but possibly koschei predates that
15:52:45 <mboddu> We can ask them to update to gssapi_login too :)
15:53:01 <nirik> sure
15:53:48 <mboddu> #info mboddu will be creating tickets against bodhi, releng, koschei to update to gssapi_login to authenticate to koji
15:53:54 <pingou> do we already support gssapi?
15:55:05 <pingou> which I think nirik just answered in -apps (yes)
15:55:31 <nirik> yeah, I think so...
15:56:31 <nirik> thanks for bringing this up mboddu
15:57:05 <mboddu> np :)
15:57:06 <mboddu> Moving on
15:57:10 <mboddu> #topic Open Floor
15:57:16 <mboddu> siddharthvipul: You got something?
15:57:16 <siddharthvipul> finally
15:57:18 <siddharthvipul> \o/
15:57:20 <siddharthvipul> https://pagure.io/releng/issue/8105
15:57:24 <siddharthvipul> yes, what is this lol
15:57:50 <siddharthvipul> I see comment from Miro that the workaround nirik gave won't work
15:57:51 <mboddu> siddharthvipul: That was me :)
15:58:03 <siddharthvipul> and I am not sure of a way to do this
15:58:25 <siddharthvipul> mboddu: I see, thank you.. but still I have no clue what to document
15:58:27 <nirik> so hum, yeah... if gating checks for newer/older? I don't know if it does
15:59:06 <nirik> pingou: would you know?
15:59:18 <mboddu> They can still tag the older nvr to -updates-candidate tag and it will follow the gating
15:59:44 <siddharthvipul> is there something I should try and play around with to know, you see! I am not a very good packager yet (and definitely not someone who knows copr/koji/bodhi very well)
15:59:51 <nirik> say rawhide has foo-1.0-1.fc33 in it, I build foo-2.0-1.fc33, but it's horrible and broken, can I tag foo-1.0-1.fc33 in updates-candidate and have it go thru and become last tagged in f33?
15:59:56 <pingou> the comment from Miro doesn't say if the whole idea is wrong or just the tag
16:00:28 * mboddu guessing just tagging
16:00:34 <nirik> I'm leary of opening rawhide for untagging... we are not supposed to untag things that have already gone out in rawhide composes...
16:00:52 <mboddu> Yes, I am not okay with it either
16:00:53 <nirik> but this has a similar effect.
16:01:55 <pingou> I'd be inclined to say that this should be caught by CI
16:02:42 <nirik> the 'older' version you mean? or whatever issue caused them to want to untag the new build? ;) or both!
16:02:50 * mboddu does untagging when requested, but I check if the build was shipped out or not
16:02:55 <pingou> the later :)
16:03:47 <siddharthvipul> I guess before documenting the process, we would like to clear if the process should happen at all?
16:04:02 <mboddu> pingou: Maybe not always - https://pagure.io/releng/issue/9461
16:04:26 <mboddu> siddharthvipul: Yes, I wanted to talk to you about it, but good you brought it up now
16:04:41 <siddharthvipul> mboddu: I don't mind taking it somewhere else later
16:05:05 <pingou> mboddu: fair
16:05:38 <nirik> yeah, I guess we should see if the process actually works still... and we should decide if it's what we want to do
16:05:48 <mboddu> pingou: So "say rawhide has foo-1.0-1.fc33 in it, I build foo-2.0-1.fc33, but it's horrible and broken, can I tag foo-1.0-1.fc33 in updates-candidate and have it go thru and become last tagged in f33?" works?
16:05:59 <mboddu> Or it needs to be tested?
16:06:31 <pingou> mboddu: bodhi may not be happy with this
16:06:41 <pingou> since there was already an update with this build
16:06:41 <mboddu> Okay
16:07:02 <pingou> worth investigating
16:07:07 <mboddu> Okay
16:07:10 <pingou> but not 100% sure it'll work
16:07:10 <mboddu> Lets test it
16:07:28 * pingou checking is cheatsheet: https://pingou.fedorapeople.org/gating_rawhide/Simplified_single_build_update-20191021.png
16:07:47 <mboddu> pingou: Haha, I opened https://docs.fedoraproject.org/en-US/rawhide-gating/single-builds/#_simplified_diagram_of_the_single_build_updates_workflow :)
16:07:52 <mboddu> Same cheatsheet :)
16:07:54 <pingou> close enough ;-)
16:08:24 <nirik> I wonder if we could make a hub policy that would let untag work for 1 day...
16:08:30 <mboddu> Anyway, I covered up for the 10 min delay :)
16:08:56 <pingou> 1 day means it'll be in a compose always no?
16:09:08 <mboddu> nirik: That could be trickey
16:09:23 <nirik> yeah, it could be. depends on when it built and when the compose ran
16:09:24 <pingou> (but that's bikeshedding, allowing to untag withing X amount of time may be interesting)
16:09:39 <mboddu> Yup, trickey :)
16:10:01 * siddharthvipul slowly slides out of conversation :P
16:10:09 <mboddu> Haha :)
16:10:42 <mboddu> #info We will investigate if tagging of old build to -updates-candidate work and if it does, we will document the process
16:11:57 <siddharthvipul> awesome :) thank you all
16:12:09 <siddharthvipul> mboddu: I will catch you at other places to see if I can actually help
16:12:15 <siddharthvipul> mboddu++
16:12:26 <mboddu> siddharthvipul: Anytime :)
16:12:31 <mboddu> Thanks everyone for joining
16:12:41 <mboddu> See you all on other channels :)
16:12:43 <mboddu> #endmeeting