16:00:05 <nirik> #startmeeting Infrastructure (2022-03-31)
16:00:05 <zodbot> Meeting started Thu Mar 31 16:00:05 2022 UTC.
16:00:05 <zodbot> This meeting is logged and archived in a public location.
16:00:05 <zodbot> The chair is nirik. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:05 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:05 <zodbot> The meeting name has been set to 'infrastructure_(2022-03-31)'
16:00:05 <nirik> #meetingname infrastructure
16:00:05 <nirik> #chair nirik siddharthvipul mobrien zlopez pingou bodanel dtometzki jnsamyak computerkid
16:00:05 <nirik> #info Agenda is at: https://board.net/p/fedora-infra
16:00:05 <zodbot> The meeting name has been set to 'infrastructure'
16:00:05 <zodbot> Current chairs: bodanel computerkid dtometzki jnsamyak mobrien nirik pingou siddharthvipul zlopez
16:00:05 <nirik> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:06 <nirik> #topic greetings!
16:00:14 <eddiejennings> .hello
16:00:14 <zodbot> eddiejennings: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:00:24 <mkonecny> .hello zlopez
16:00:24 <eddiejennings> .hello eddiejennings
16:00:24 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@psmail.xyz>
16:00:27 <zodbot> eddiejennings: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:00:31 <lenkaseg> .hi
16:00:32 <zodbot> lenkaseg: lenkaseg 'Lenka Segura' <lenka@sepu.cz>
16:00:39 <VipulSiddharth[m> .hello siddharthvipul1
16:00:39 <nirik> avast and ahoy all ye land lubbers!
16:00:42 <zodbot> VipulSiddharth[m: siddharthvipul1 'Vipul Siddharth' <siddharthvipul1@gmail.com>
16:00:44 <dtometzki> .hi
16:00:46 <zodbot> dtometzki: dtometzki 'Damian Tometzki' <linux@tometzki.de>
16:01:10 <nirik> #topic New folks introductions
16:01:10 <nirik> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:01:10 <nirik> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:01:53 <nirik> will weigh anchor here for a few to see if any new folks would like to introduce themseves and in case more folks come back from shore to join us.
16:02:15 <nirik> oh, we should replace that gettingstarted with a new docs link. Did we move that doc over yet?
16:02:30 <eddiejennings> Not sure, but the new docs site looks great
16:03:07 <mobrien> .hi
16:03:08 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
16:03:16 <mkonecny> nirik: I don't think we moved it yet
16:03:24 <nirik> yeah seems not
16:03:29 <mkonecny> But I looked at the wiki and updated what needed update
16:03:47 <nirik> it would be nice to retire all the wiki pages...
16:03:54 <eddiejennings> Looks like all the SOPs are at the new site
16:04:10 <nirik> yeah, but they all need going over. ;)
16:04:24 <nirik> anyhow, no new crew today?
16:04:30 <petebuffon> .hello petebuffon
16:04:31 <zodbot> petebuffon: petebuffon 'Peter Buffon' <pabuffon@gmail.com>
16:05:04 <nirik> Lets sail on then...
16:05:08 <nirik> #topic Next chair
16:05:08 <nirik> #info magic eight ball says:
16:05:08 <nirik> #info chair 2022-03-31 - nirik
16:05:08 <nirik> ##info chair 2022-04-07 - ??
16:05:08 <nirik> ##info chair 2022-04-14 - ??
16:05:25 <eddiejennings> One moment while I check my calendar
16:05:26 <dtometzki> i will do it on 04-07
16:05:26 <nirik> anyone want to step up to the wheel for 4-07 and/or 4-14?
16:05:40 <mobrien> I can do 04-14
16:05:40 <eddiejennings> I should be able to do 4/14, one sec while I confirm
16:05:42 <nirik> dtometzki: done
16:06:03 <dtometzki> thanks
16:06:13 <eddiejennings> put me down for 4/21 then
16:06:24 <nirik> sold.
16:07:06 <nirik> alright, listen up me hearties to these announcements:
16:07:09 <nirik> #topic announcements and information
16:07:09 <nirik> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:07:09 <nirik> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:07:09 <nirik> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:07:11 <nirik> #info Fedora 36 Beta is out
16:07:13 <nirik> #info Fedora Final freeze starts on 5th April
16:07:15 <nirik> #info thread on fedoraplanet on infrastructure list, chime in if you have thoughts on it
16:07:17 <nirik> #info mass updates and reboots of Fedora Infrastructure may cause some outages https://pagure.io/fedora-infrastructure/issue/10613
16:07:22 <nirik> any other announcements? yell em out!
16:08:02 <mkonecny> #info Fedora 36 Beta freeze is no longer in effect
16:08:21 <nirik> yeah, but not much of a break... :)
16:09:09 <nirik> ok, lets see who's in the crows nest and what they have seen:
16:09:11 <nirik> #topic Oncall
16:09:11 <nirik> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:09:11 <nirik> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:09:11 <nirik> ## .oncalltakeeu .oncalltakeus
16:09:11 <nirik> #info eddiejennings on call from 2022-03-25 to 2022-03-31
16:09:12 <nirik> #info petebuffon on call from 2022-04-01 to 2022-04-07
16:09:14 <nirik> ##info ?? on call from 2022-04-08 to 2022-04-14
16:09:40 <nirik> someone want to climb the rigging for the 2022-04-08 to 2022-04-14 stint?
16:09:55 <mobrien> Aye aye cap'n
16:10:00 <mobrien> put me down
16:10:11 <nirik> alrighty
16:10:28 <nirik> #info Summary of last week: (from current oncall )
16:10:43 <nirik> what seen ya from the crows nest mr eddiejennings ?
16:10:44 <eddiejennings> I'll be willing to do 4/15 - 4/21.  I like ending on-call when I chair a meeting.
16:11:00 <nirik> fair. will put you down
16:11:28 <eddiejennings> I was summoned once for mailman having an issue.  Ultimately it was solved by a non-apprentice
16:11:38 <eddiejennings> I think its service needed to be restarted.
16:11:54 <nirik> ok
16:12:16 <eddiejennings> Other than that, it was quiet :)
16:12:29 <petebuffon> .oncalltakeus
16:12:29 <zodbot> petebuffon: Kneel before zod!
16:12:56 <nirik> great!
16:13:03 <nirik> #topic Monitoring discussion [nirik]
16:13:03 <nirik> #info https://nagios.fedoraproject.org/nagios
16:13:03 <nirik> #info Go over existing out items and fix
16:13:33 <nirik> so, nothing much new on the nagios front.
16:14:17 <nirik> we are doing a mass update/reboot cycle finishing later today... will see where we stand then.
16:14:33 <eddiejennings> Just in time for petebuffon ;)
16:14:59 <nirik> mobrien: did you see that worker05-mgmt.stg appeared again?
16:16:00 <nirik> weird, but I'll take it. ;)
16:16:17 <mobrien> nirik: I did not, the computer gods are smiling on us
16:16:42 <petebuffon> hah nice
16:16:45 <nirik> well, we also have download-rdu01 down, so... the gods give, the gods take
16:17:30 <nirik> anyhow, so, I was scheduled for a ansible 5 talk today. I can still do that if we like... but I ran into problems trying to switch batcave01 over, so we will not be switched until after final freeze...
16:17:49 <nirik> or I could wait and give the talk after we finish moving over?
16:18:23 <mkonecny> I would like to hear the talk with the issue that happened
16:18:35 <eddiejennings> +1 about the issue
16:18:42 <nirik> ok, fair enough... and actually I need to help on them anyhow. ;)
16:18:53 <nirik> #topic Ansible 5 and fedora infrastructure [nirik] - 2022-03-31
16:19:19 <nirik> So first a bit of history/background. Many of you may already know this, but some of it might be new. ;)
16:19:51 <nirik> ansible has always been a single package with 'batteries included' in the past. The idea being you could install the one package and be ready to use it...
16:20:41 <nirik> The problem became that ansible became massively popular. So, everything out there wanted to make sure and have an ansible module for their thing.
16:21:39 <nirik> Some things were iterating rapidly, so they needed ansible to release often. There were so many PR's against ansible, the core maintainers couldn't come close to reviewing / merging them all.
16:21:46 <nirik> This caused changes to be slow to land...
16:22:01 <eddiejennings> The sad reality :(
16:22:26 <nirik> additionally the ansible engine had to try and deal with the rapid releases. It's hard to make big changes when you release super often...
16:23:01 <nirik> So, ansible folks tried various things. They setup community maintainers to merge things in some areas. They tried to automate reviews/ci/PR's
16:23:19 <nirik> but that still didn't solve the slow/fast engine/modules.
16:23:49 <nirik> They also had problems like obscure module X has a CVE, so you have to do a complete release, even though almost no one was using that module.
16:24:09 <nirik> So, they decided split things up.
16:24:43 <nirik> ansible-core would be the engine only with a bare few modules. This would allow you to work on the engine without having to worry about all the modules/collections, etc.
16:25:02 <eddiejennings> Ah, modularizing Ansible, eh? :)
16:25:24 <nirik> 'ansible' would become a collection of collections. Popular things that agree to release at the same time.
16:25:46 <nirik> This way someone could install the 'ansible' collections and 'ansible-core' and still have everything (and more) included.
16:26:15 <nirik> In this setup all the collections are seperate upstreams. They can have their own maintainers that care about them and merge things as they like.
16:26:50 <nirik> Some users may only want to install ansible-core and some specific collections to have a smaller footprint.
16:27:03 <nirik> the 'ansible' collection of collections has a LOT of them.
16:28:04 <nirik> So, thats where we are today. Currently fedora infra is using 'ansible classic' ie, ansible 2.9.x before the split.
16:28:32 <nirik> I did some testing and for the most part things are pretty complatible playbook wise.
16:29:00 <nirik> I tried to swap us over to ansible-core yesterday, but ended up reverting it. ;( This is for several reasons:
16:29:40 <nirik> actually... let me back up a sec.
16:29:55 <nirik> ansible-core is landing in RHEL8.6 and 9.
16:30:07 <nirik> so, it's in CentOS stream 8 currently.
16:30:34 <nirik> However, ansible-core-2.12.3 (the current one) requires... python 3.8
16:30:35 * austinpowered sneaks in quietly
16:30:39 <austinpowered> .hi
16:30:42 <zodbot> austinpowered: austinpowered 'T.C. Williams' <fedoraproject@wootenwilliams.com>
16:30:52 <austinpowered> sorry I'm late
16:31:05 <nirik> so, the rhel8/centos-stream8 ansible-core is built against python3.8. This causes some issues for us sadly. :(
16:31:18 <nirik> When I swapped in ansible-core, it broke two things:
16:32:00 <nirik> * ansible_utils (the is the package that provides rbac_playbook). I tried rebuilding against python3.8 but it still was broken. Something deeper needs changing. :(
16:32:51 <nirik> * fedora messaging. our ansible has a callback plugin to emit a fedora-messaging message on start/end of playbooks. But fedora-messaging is built against python 3.6, so ansible-core can't import it. ;(
16:33:09 <nirik> so, we will need to solve those issues before we can switch to it.
16:33:37 <mkonecny> python 3.6 is EOL already, so this is not good
16:33:50 <nirik> well, rhel is maintaining it for 8 still. ;)
16:34:09 <mkonecny> I meant the official upstream
16:34:17 <nirik> yeah...
16:34:42 <nirik> one extreme thing we could do would be to switch batcave01 to a fedora install. But then we would need to upgrade it a lot more. ;(
16:35:10 <mkonecny> What is the Batcave01 running now? RHEL 8?
16:35:10 <eddiejennings> This means the dev(s) who wrote fedora-messaging must alter the app to be able to use python 3.8?
16:35:17 <mobrien> I would think batcave should stay on rhel
16:35:21 <nirik> mkonecny: yes, 8
16:35:42 <nirik> eddiejennings: I'm not sure. ;) Is there some way to import something from one fedora version into another?
16:35:52 <mkonecny> I think it shouldn't be that hard to migrate fedora-messaging to 3.8
16:35:56 <nirik> or some way it could emit the message without loading the python bindings
16:36:46 <mkonecny> I can look at the fedora-messaging if you want
16:37:36 <nirik> ok, if we change the official epel8 one to use python3.8, it might cause some messyness... since anything that used it would need to enable the python3.8 module...
16:39:07 <nirik> not sure. Anyhow, I can update the ticket with all this and we can ponder on the best way forward.
16:39:17 <mkonecny> The Fedora messaging 2.0.0 and newer should be fine, the-new-hotness only supports python >= 3.8 and it has fedora-messaging as requirement
16:39:21 <nirik> Did anyone have any questions about the ansible background or change?
16:40:42 <mkonecny> I'm actually interested in rbac-playbook, what it does?
16:40:48 <nirik> mkonecny: I guess we could just do a version of fedora-messaging for batcave01 that uses 3.8...
16:41:10 <nirik> it's the wrapper around sudo and ansible-playbook that lets particular groups run playbooks.
16:41:43 <mkonecny> So it's basically just checking permissions of the user?
16:41:49 <nirik> so, for example it's config may say sysadmin-web can run the proxies playbook. They can run 'sudo rbac-playbook groups/proxies.yml' and it will run that playbookk if the user is in that group
16:41:53 <nirik> yep
16:42:02 <Ebeneezer_Smooge> EL 9 should be out by summer
16:42:44 <mkonecny> And how the wrapping around sudo works?
16:42:57 <nirik> if someday we ever deploy AWX it might be able to handle the use cases, but we don't have that now.
16:43:33 <nirik> https://bitbucket.org/tflink/ansible_utils/src/master/
16:46:42 <nirik> I'm not sure exactly how it works. ;)
16:47:17 <nirik> Oh, one last thing... I found that we don't use that many collections really... so I think we can go with the ansible-core and just install those specific collections we use.
16:47:46 <nirik> it's 7 collections currently
16:48:14 <nirik> 'ansible' metacollection has like hundreds.
16:48:25 <eddiejennings> Yeah, leaner = generally better in my opinion
16:50:37 <nirik> also one curious thing... I reverted back after the failed switchover... and ansible_utils wasn't working. I had to disable email sending... I don't understand how it worked before. ;(
16:51:52 <nirik> ok. Lets sail on then...
16:52:02 <nirik> #topic Open Deck
16:52:12 <nirik> anything anyone wants to shout out?
16:52:50 <mkonecny> Do we have any other topic for the future?
16:53:20 <eddiejennings> yes, one moment
16:53:33 <eddiejennings> .ticket 9549
16:53:35 <zodbot> eddiejennings: Issue #9549: Tor hidden service for update metadata - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9549
16:54:00 <eddiejennings> I'm working with petebuffon on this (will be shooting Pete an E-mail either this evening or tomorrow).
16:54:07 <nirik> great!
16:54:18 <eddiejennings> Should we be looking at this from the perspective or "how to make it happen" or "does it make sense to consider this?"
16:54:52 <eddiejennings> perspective of . . .
16:55:53 <nirik> well, either or both? ;) I mean, I guess what it would take to do it from a technical side and then we can factor in if it makes sense?
16:56:42 <eddiejennings> That seems reasonable.  If the technical challenge is great, then we consider if it's worth it.
16:57:04 <petebuffon> sounds good
16:57:25 <nirik> cool.
16:57:54 <nirik> Thanks for coming everyone, fair winds and following seas to you all!
16:58:01 <nirik> #endmeeting