18:05:35 <cverna> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
18:05:35 <zodbot> Meeting started Mon May 25 18:05:35 2020 UTC.
18:05:35 <zodbot> This meeting is logged and archived in a public location.
18:05:35 <zodbot> The chair is cverna. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:05:35 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:05:35 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:05:35 <cverna> #chair cverna mboddu nirik smooge
18:05:35 <cverna> #meetingname fedora_infrastructure_ops_daily_standup_meeting
18:05:35 <zodbot> Current chairs: cverna mboddu nirik smooge
18:05:35 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:05:50 <cverna> #info meeting is 30 minutes MAX. At the end of 30, its stops
18:05:51 <cverna> #info agenda is at https://board.net/p/fedora-infra-daily
18:06:04 <cverna> #topic Tickets needing review
18:06:04 <cverna> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
18:07:42 * nirik isn't really here since it's a holiday, but I can be lurking. ;)
18:08:18 * cverna wait and see who is here then :P
18:08:34 <nirik> we can do a short meeting. I am just working on iad2...
18:08:52 <nirik> .ticket 8937
18:08:53 <zodbot> nirik: Issue #8937: Rawhide builds are not pushed to stable - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8937
18:09:02 <nirik> that seemed to fix itself?
18:10:12 <cverna> yeah I did not do anything to this, I tried to look at kibana but the logs were gone
18:10:30 <nirik> the logging pods keep getting OOMed.
18:10:31 <nirik> :(
18:10:41 <cverna> :(
18:10:45 <nirik> I made the nodes in iad2 with more memory, hopefully that will help that
18:11:01 <cverna> I ll try to monitor the pods and to see if I see some specific traceback
18:11:16 <nirik> is there an upstream ticket on this? or ?
18:11:54 <cverna> I don't think this is upstream, I think this is more of a problem with celery and rabbitmq but I am not 100% sure
18:12:53 <smooge> hello sorry was poking disk space
18:13:32 <cverna> I think we can close that and I ll keep an eye on the bodhi in the next few days
18:13:38 <nirik> ok, so keep it open and loook a while, then close if nothing can be found?
18:13:40 <cverna> I ll update that ticket
18:13:43 <nirik> or that
18:13:44 <nirik> +1
18:14:18 <fm-admin> pagure.issue.new -- kevin opened a new ticket fedora-infrastructure#8941: "get ipad01/02.iad2 replicating with ipa01/02.phx2" https://pagure.io/fedora-infrastructure/issue/8941
18:14:27 <nirik> .ticket 8939
18:14:28 <zodbot> nirik: Issue #8939: centos-ci not processing fedora-messaging messages? - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8939
18:14:58 <nirik> this does not seem to be fixed.
18:15:01 <nirik> RABBITMQ_QUEUE WARNING - messages WARNING (24480), messages_ready OK (24480) messages_unacknowledged OK (0) consumers OK (0)
18:15:22 <fm-admin> pagure.issue.edit -- cverna edited the close_status and status fields of ticket fedora-infrastructure#8937 https://pagure.io/fedora-infrastructure/issue/8937
18:15:23 <fm-admin> pagure.issue.comment.added -- cverna commented on ticket fedora-infrastructure#8937: "Rawhide builds are not pushed to stable" https://pagure.io/fedora-infrastructure/issue/8937#comment-654181
18:15:52 <fm-admin> pagure.issue.comment.added -- kevin commented on ticket fedora-infrastructure#8939: "centos-ci not processing fedora-messaging messages?" https://pagure.io/fedora-infrastructure/issue/8939#comment-654183
18:16:11 <nirik> I guess we can move it to wiating on asignee and see what they say to my last comment there?
18:16:53 <cverna> ok doing it
18:17:18 <fm-admin> pagure.issue.edit -- cverna edited the priority fields of ticket fedora-infrastructure#8939 https://pagure.io/fedora-infrastructure/issue/8939
18:18:21 <fm-admin> pagure.issue.new -- kevin opened a new ticket fedora-infrastructure#8942: "rabbitmq cluster in iad2 not clustering" https://pagure.io/fedora-infrastructure/issue/8942
18:18:33 <nirik> ok, the next set are iad2 related.
18:19:04 <nirik> .ticket 8940
18:19:05 <zodbot> nirik: Issue #8940: port python-tag2distrepo koji hub plugin to python3 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8940
18:19:07 <nirik> .ticket 8941
18:19:08 <zodbot> nirik: Issue #8941: get ipad01/02.iad2 replicating with ipa01/02.phx2 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8941
18:19:08 <nirik> .ticket 8942
18:19:11 <zodbot> nirik: Issue #8942: rabbitmq cluster in iad2 not clustering - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8942
18:19:35 <nirik> I'm hoping puiterwijk could help with some or all of these, but if someone else can, feel free.
18:19:55 <cverna> do we need help from abompard for the rabbitmq cluster ?
18:20:15 <cverna> we can see if he has a bit of time to help us
18:20:28 <nirik> yeah, I mentioned him there too. if he has time, great!
18:21:05 <cverna> will he have access to the box tho ?
18:21:41 <nirik> yep. Just use bastion-iad01.fedoraproject.org in your .ssh/config for iad hosts... and the same people who have access to stuff in phx2 should have access to the iad2 counterpart
18:21:59 <cverna> cool
18:22:09 <nirik> I have almost all the non build stuff built now. The exceptions are: mailman01 and notifs
18:22:15 <cverna> I ll try to catch him tomorrow and see if he can help us or not :)
18:22:25 <nirik> I was thinking we might want ti migrate those vm's and reconfigure them rather than reinstall...
18:23:19 <nirik> we could also try doing new installs... but I am not sure how far we will get... mailman01 has some... interesting deps. notifs-backend01 is.... fedora 27 I think?
18:23:45 <nirik> so I am sure modern fedora would break it due to python2 if nothing else.
18:23:49 <cverna> oh my
18:24:17 <cverna> I think nils  started to look at mailman but this are quite some work there to package it
18:24:34 <nirik> I'm working on build sys side now... hopefully will have that setup in the next few days.
18:24:57 <nirik> then I want to make a checklist and have people go and validate things.
18:25:24 <cverna> nirik: for tag2distrepo is it the package that needs to be python3 or there is an error in the code base ?
18:25:55 <nirik> the package. its python2 currently.
18:26:06 <nirik> https://koji.fedoraproject.org/koji/buildinfo?buildID=1272405
18:26:12 <cverna> ok
18:26:21 <cverna> the code base looks sane
18:26:26 <cverna> for python3
18:27:06 <nirik> https://pagure.io/releng/tag2distrepo/commits/master
18:27:13 <nirik> yeah, hopefully it will be easy...
18:27:14 <cverna> I ll check with siddharthvipul if he wants to do a PR for the package to have it build for python3
18:27:28 <nirik> thats blocking me configuring the koji hubs
18:28:01 <cverna> ok yeah would be nice to have soon
18:28:11 <nirik> cverna: you want me to do osbs in iad2? or could you take that on?
18:28:40 <cverna> happy to do it
18:28:53 <nirik> cool. Or I can start it and hand it to you to finish?
18:29:08 <cverna> yeah sure that works too
18:29:25 <cverna> I would like to move prod to rhel7 instead of fedora
18:29:25 <nirik> that way I can get all the ips/vmhosts all set...
18:29:39 <cverna> in phx2 I did stg but no prod
18:29:48 <nirik> ok. I thought there was some blocker for that...
18:29:50 <cverna> +1
18:29:53 <nirik> but I can't recall what it was..
18:30:14 <cverna> no the blocker is for rhel8 because no docker
18:30:23 <cverna> rhel7 seem to work fine :)
18:30:30 <fm-admin> pagure.issue.comment.added -- msrb commented on ticket fedora-infrastructure#8939: "centos-ci not processing fedora-messaging messages?" https://pagure.io/fedora-infrastructure/issue/8939#comment-654187
18:30:41 <nirik> ahhh. ok.
18:30:50 <nirik> fine with me then.
18:31:24 <plarsen> cverna, no docker as in Moby yes. But crio and podman are there. You should be able to run/translate what-ever you have to use podman very easily (just alias docker => podman)
18:32:55 <cverna> plarsen: yep but no :) OSBS was writting using the docker python binding API and moving to something else is a lot of work.
18:33:07 <cverna> plarsen: Last I talked with the upstream this was not there priority for now
18:33:14 <cverna> their*
18:34:05 <cverna> ok we are done for the 30mins
18:34:10 <cverna> #endmeeting