18:00:01 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
18:00:01 <zodbot> Meeting started Mon Jun  1 18:00:01 2020 UTC.
18:00:01 <zodbot> This meeting is logged and archived in a public location.
18:00:01 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:01 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:02 <nirik> #chair cverna mboddu nirik smooge
18:00:02 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting
18:00:02 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops
18:00:02 <zodbot> Current chairs: cverna mboddu nirik smooge
18:00:02 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:02 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily
18:00:03 <nirik> #topic Tickets needing review
18:00:04 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
18:00:06 <smooge> here
18:00:10 * pingou 
18:00:14 <nirik> .ticket 8957
18:00:15 <zodbot> nirik: Issue #8957: create a jbangdev group - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8957
18:00:21 * mboddu is here
18:00:40 <nirik> anyone want to be ticket updater today?
18:00:45 <smooge> i will
18:00:50 <pingou> I've this one on my to-process list, but the long and short of it is that I don't know how copr works with groups
18:00:54 <pingou> so I can't quite help there
18:01:08 <nirik> it's the same as pagure... on login.
18:01:13 <nirik> so, ask them to logout/login?
18:01:20 <pingou> sounds fine
18:01:34 <nirik> smooge: ask them that and close?
18:01:44 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8957: low-gain and medium-trouble https://pagure.io/fedora-infrastructure/issue/8957
18:01:45 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8957 https://pagure.io/fedora-infrastructure/issue/8957
18:01:53 <cverna> o/
18:02:09 <nirik> .ticket 8965
18:02:10 <zodbot> nirik: Issue #8965: Bodhi-backend is trying to connect directly to pagure.io fedmsg - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8965
18:02:12 <nirik> .ticket 8964
18:02:14 <zodbot> nirik: Issue #8964: docs too big to fit on proxies anymore - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8964
18:02:15 <nirik> sorry... missed one
18:02:27 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8957: "create a jbangdev group" https://pagure.io/fedora-infrastructure/issue/8957#comment-655673
18:02:33 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8957 https://pagure.io/fedora-infrastructure/issue/8957
18:02:34 <fm-admin> pagure.issue.edit -- pingou edited the priority fields of ticket fedora-infrastructure#8957 https://pagure.io/fedora-infrastructure/issue/8957
18:02:35 <fm-admin> pagure.issue.comment.added -- pingou commented on ticket fedora-infrastructure#8957: "create a jbangdev group" https://pagure.io/fedora-infrastructure/issue/8957#comment-655675
18:02:46 <nirik> so... if this could use a hardlink copy it might help a lot?
18:03:00 <nirik> and/or our sync could just exclude all those build dirs
18:03:15 <nirik> and/or we could just serve it from openshift and stop trying to put it on proxies.
18:03:33 <smooge> well part of it is that their build script seems to repeat the entire build for ALL the zones and then copies out one zone, then removes that, then builds for all the zones, and copies out the next one, and
18:03:38 <nirik> perhaps this would be good foer a list discussion?
18:04:11 <nirik> unless theres some urgent doc change, I would prefer to just leave it off for now and revist after next week...
18:04:35 <nirik> thoughts?
18:04:44 <smooge> ok high-trouble high-gain groomed waiting on assignee
18:04:49 <cverna> sounds good to me, we could ask asamalik and misc to also have a look at the build script
18:05:18 <nirik> smooge: +1 ack
18:05:24 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8964: groomed, high-gain, and 2 others https://pagure.io/fedora-infrastructure/issue/8964
18:05:25 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8964 https://pagure.io/fedora-infrastructure/issue/8964
18:05:27 <nirik> cverna: yeah. might be some easy wins
18:05:40 <nirik> .ticket 8965
18:05:41 <zodbot> nirik: Issue #8965: Bodhi-backend is trying to connect directly to pagure.io fedmsg - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8965
18:06:09 <nirik> so... is pagure.io still on it's on message bus? I can't recall
18:06:25 <nirik> so, there's 1 last thing using fedmsg on bodhi-backend01
18:06:30 <pingou> not anymore since fedora-messaging
18:06:44 <pingou> and when we upgrade to rhel8 we should clean things up
18:06:58 <nirik> it's the koji owner sync thing.
18:07:24 <nirik> ie, watches for new packages and adds them to koji tags (I think still incorrectly as 'releng' owned too)
18:07:36 <smooge> so could be turned off?
18:07:49 <nirik> only if we don't want anyone to build any new packges. ;)
18:07:58 <pingou> nicolas17: can you open a ticket so we can port it?
18:08:01 <pingou> nirik: ^
18:08:04 <pingou> sorry nicolas17 :)
18:08:09 * nicolas17 hides
18:08:15 <nirik> sure. I can... I am not sure how it works in phx2 either.
18:08:50 <cverna> We can turn it off in IAD2 until we have a fedora-messaging version ?
18:08:55 <nirik> well, we could reuse this ticket? or close this one and do a new one?
18:09:03 <nirik> cverna: we can...
18:09:19 <pingou> ticket wise, either way wfm
18:09:45 <smooge> nirik, I should have been clearer: if pagure didn't use fedmsg anymore then having it defined in the configs that bodhi is using to connect to it could go away?
18:09:46 <cverna> not sure how much effort there is to port it to fedora-messaging, if it is just a consumer that should be fairly straight forward
18:10:30 <fm-admin> pagure.issue.new -- kevin opened a new ticket fedora-infrastructure#8967: "port koji-owner-sync script to fedora-messaging and make it work better" https://pagure.io/fedora-infrastructure/issue/8967
18:10:52 <smooge> nirik, close this one as with that as the replacement?
18:10:55 <nirik> smooge: bodhi doesn't.;. but this script that runs there does
18:11:03 <nirik> smooge: ack +1
18:11:22 <nirik> the script runs there because it needs koji admin perms, and it just reuses bodhi's keytab...
18:11:25 <fm-admin> pagure.issue.edit -- smooge edited the close_status and status fields of ticket fedora-infrastructure#8965 https://pagure.io/fedora-infrastructure/issue/8965
18:11:25 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8965: "Bodhi-backend is trying to connect directly to pagure.io fedmsg" https://pagure.io/fedora-infrastructure/issue/8965#comment-655680
18:11:35 <pingou> couldn't we just point fedmsg to the gateway?
18:11:52 <pingou> instead of pagure01
18:11:54 <nirik> thats not how it works is it?
18:12:13 <pingou> for fedmsg, w/ the bridge it should I think
18:12:34 <nirik> oh, I see what you are saying... point it to our busgateway
18:12:38 <pingou> yes
18:12:39 <nirik> yeah, we could.
18:12:52 <pingou> pagure01> fedora-messaging -> bugateway -> fedmsg -> bodhi
18:12:54 <nirik> I really want to finish porting away from fedmsg tho. it's really a headache having both
18:12:59 <pingou> +!
18:13:01 <pingou> +1*
18:13:15 <nirik> but we could just punt and do that if we can't port in a timely manner
18:13:24 <nirik> .ticket 8966
18:13:24 <pingou> just as a temporary fix until we get the fedora-messaging client, which should be straight forward
18:13:25 <zodbot> nirik: Issue #8966: Staging Anitya is unable to send fedora messaging messages - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8966
18:13:44 <smooge> I have a fix for this
18:13:45 <pingou> Michal is ok w/ this waiting post move
18:13:51 <smooge> that was my fix
18:14:17 <nirik> yeah, odd that it broke now tho
18:15:09 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8966: backlog, groomed, and 3 others https://pagure.io/fedora-infrastructure/issue/8966
18:15:10 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8966 https://pagure.io/fedora-infrastructure/issue/8966
18:16:25 <nirik> ok... thats all the infra needs review.
18:16:33 <nirik> anything on releng side mboddu ?
18:16:43 <nirik> or any other tickets we want to discuss?
18:16:44 <mboddu> nirik: Yes
18:17:00 <mboddu> .releng 9497
18:17:02 <zodbot> mboddu: Issue #9497: Package stuck in f33-build-side-23466-testing-pending - releng - Pagure.io - https://pagure.io/releng/issue/9497
18:17:14 <smooge> brb
18:17:25 <mboddu> pingou: ^ on that, I can untag the builds, but the tag should be removed as well after that?
18:17:33 <mboddu> side tag*
18:18:02 * pingou checks the ticket
18:18:12 <mboddu> The doc doesn't say anything, I can look up history of other side tags or the code, but thought it might be easier to ask you :)
18:18:31 <pingou> mboddu: I don't think they need up
18:18:33 <pingou> us*
18:18:45 <pingou> they can just do fedpkg remove-side-tag
18:18:56 <pingou> it'll remove it w/ merging the packages
18:19:12 <pingou> then they can just re-tag the build to the proper tag if they wish to
18:19:13 <mboddu> The builds are already merged
18:19:33 <mboddu> I will try fedpkg remove-side-tag and see what happens
18:19:43 <nirik> mboddu: BTW, if you have a document for what to do at branching can you add 'add new rawhide tag to koji-gc' to it?
18:19:45 <pingou> mboddu: I'd let them do it
18:20:05 <mboddu> nirik: Sure, it should be the releng doc as I have updated it very recently
18:20:08 <mboddu> pingou: Okay
18:20:20 <mboddu> Other tickets, I already worked on them
18:20:25 <mboddu> Thats it on releng side
18:20:33 <nirik> I just noticed the other day it was missing and.... fedora_koji is painfully near full.
18:21:25 <nirik> ok, quickly on the dc move... I updated https://hackmd.io/Eqsf5wFoQRGYhAVwSKJvIA?edit with our plans.
18:21:37 <mboddu> nirik: Sure, and sorry I missed it
18:21:42 <nirik> so thats the place to look for up to date info...
18:21:44 <nirik> basically:
18:22:10 <nirik> today/tomorrow/wed we are going to be testing iad2 stuff more, making sure playbooks complete, making sure everything is installed, etc.
18:22:28 <nirik> thursday we are gonna cut over zodbot/value
18:22:38 <nirik> friday we are gonna cut over batacave
18:23:03 <pingou> poor zodbot
18:23:11 * mboddu weeps for batcave and zodbot
18:23:14 <nirik> then next week: monday is taking down all *stg.phx2 and other stuff we won't be moving,
18:23:33 <nirik> tuesday is fedora-messaging cluster, mirrormanager, pdc and the auth stuff
18:23:47 <nirik> sorry, monday ^
18:24:02 <nirik> tuesday is build stack... src, koji, etc
18:24:26 <nirik> wed is openshift apps. mailman/lists, datagrepper/datanommer
18:24:58 <nirik> thursday is the build openshift stuff (websites, docs, etc), blockerbugs, and elections (after the current elections close)
18:25:08 <nirik> friday is fix things day.
18:25:10 <mboddu> When does batcave will be brought back up?
18:25:23 <mboddu> in IAD2
18:25:38 <pingou> mboddu: it's already there
18:25:48 <nirik> mboddu: this friday... the plan is to do a cut over friday and then everyone should start using it instead of the phx2 one (which will be turned off)
18:26:08 <mboddu> pingou: ^ thats the question, but worded poorly :D
18:26:10 <nirik> we have to switch the fedora_app volume then to be rw in iad2.
18:26:30 <nirik> yeah, it's there now, but it's got ro storage mounts and is manually synced.
18:26:42 <mboddu> nirik: Okay, thanks
18:27:02 <nirik> I plan to send a message to devel-announce and infra lists later today with the above ^ ie, general days when things move.
18:27:13 <nirik> we still need to isolate more exact times for things.
18:27:36 <nirik> (because we need storage folks to flip over the storage and need to tell them when to be around)
18:27:54 <nirik> does that all make sense? any problems or issues? things I forgot?
18:28:20 * mboddu reading back about compose boxes for rawhide
18:28:34 <nirik> so, some things will be stopped while we move.
18:28:40 <mboddu> Right
18:28:52 <mboddu> rawhide composer will be brought back up on Monday or ... ?
18:29:18 <nirik> so, tuesday morning we disable rawhide and bodhi.
18:29:37 <nirik> tuesday evening if all goes well we try those manually and confirm they work
18:29:50 <mboddu> Oh okay
18:30:01 <nirik> also, sunday night we ask people to stop processing unretirements or adding new packages
18:30:24 <nirik> because thats going to potentially fail if done at the wrong time. So that holds until we say "ok, try doing one now"
18:30:40 <mboddu> Okay, makes sense
18:31:02 <nirik> we might want to disable rawhide monday morning... since it's been taking a long time to finish
18:31:29 <mboddu> Tues morning or Mon morning?
18:31:46 <mboddu> Well, we could disable after our Mon compose
18:32:25 <nirik> yep.
18:32:45 <nirik> if you see things that need doing at a specific time on this plan: please add a checkbox for them!
18:33:07 <nirik> ok, thats all I had and we are over time...
18:33:11 <nirik> anything else?
18:33:23 <mboddu> Nothing from me
18:33:59 <nirik> #endmeeting