16:00:35 <eddiejennings> #startmeeting Infrastructure (2022-04-28)
16:00:35 <zodbot> Meeting started Thu Apr 28 16:00:35 2022 UTC.
16:00:35 <zodbot> This meeting is logged and archived in a public location.
16:00:35 <zodbot> The chair is eddiejennings. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:35 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:35 <zodbot> The meeting name has been set to 'infrastructure_(2022-04-28)'
16:00:50 <eddiejennings> #meetingname infrastructure
16:00:50 <zodbot> The meeting name has been set to 'infrastructure'
16:00:51 <eddiejennings> #chair nirik siddharthvipul mobrien zlopez pingou bodanel dtometzki jnsamyak computerkid
16:00:51 <eddiejennings> #info Agenda is at: https://board.net/p/fedora-infra
16:00:51 <zodbot> Current chairs: bodanel computerkid dtometzki eddiejennings jnsamyak mobrien nirik pingou siddharthvipul zlopez
16:00:51 <eddiejennings> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:51 <eddiejennings> #topic greetings!
16:00:56 <aheath1992> .hello aheath1992
16:00:57 <zodbot> aheath1992: aheath1992 'Andrew Heath' <aheath1992@gmail.com>
16:00:59 <eddiejennings> .hi
16:01:01 <zodbot> eddiejennings: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:01:01 <nirik> morning
16:01:07 <eddiejennings> Greetings all! :D
16:01:22 <nb> .hi
16:01:23 <zodbot> nb: nb 'Nick Bebout' <nick@bebout.net>
16:02:12 <eddiejennings> #topic New folks introductions
16:02:12 <eddiejennings> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:02:12 <eddiejennings> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:02:31 <eddiejennings> Anyone new with us this week?  If so, say "hi" and introduce yourself!
16:02:36 <mkonecny> .hello zlopez
16:02:37 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@psmail.xyz>
16:02:42 <eddiejennings> We won't bite.  Not even nirik :)
16:03:18 <petebuffon> .hello petebuffon
16:03:18 <zodbot> petebuffon: petebuffon 'Peter Buffon' <pabuffon@gmail.com>
16:03:23 <mobrien> .hi
16:03:24 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
16:03:28 <nirik> 🙋
16:03:33 <aheath1992> zodbot is a bit sketchy
16:04:02 <eddiejennings> Yeah.  It only accepts greetings through kneeling :D
16:04:04 <zlopez> There is a lag between matrix and libera.chat, I see only few messages on matrix
16:04:17 <eddiejennings> zlopez :(
16:05:27 <eddiejennings> Looks like we don't have any new folks this week.  Let's move along.
16:05:50 <Saffronique> .hello dkirwan
16:05:51 <zodbot> Saffronique: dkirwan 'David Kirwan' <davidkirwanirl@gmail.com>
16:06:00 <eddiejennings> #topic Next chair
16:06:00 <eddiejennings> #info magic eight ball says:
16:06:16 <eddiejennings> ##info chair 2022-04-28 - eddiejennings
16:06:16 <eddiejennings> ##info chair 2022-05-05 - dtometzski
16:06:16 <eddiejennings> ##info chair 2022-05-12 -
16:06:28 <eddiejennings> Any volunteers for May 12th?
16:06:31 <mobrien> Is it just me or has matrix stopped working?
16:06:44 <eddiejennings> zlopez said it was lagging
16:06:51 * nirik hands the matrix bridge a plunger
16:06:58 <mkonecny> It doesn't work well today
16:07:11 <mkonecny> I see only third of the messages
16:07:24 <VipulSiddharth[m> .hi siddharthvipul1
16:07:24 <zodbot> VipulSiddharth[m: Error: Missing "]".  You may want to quote your arguments with double quotes in order to prevent extra brackets from being evaluated as nested commands.
16:08:08 <eddiejennings> Didn't know Zod would respond to commands from Matrix.  That's curious. :)
16:08:45 <eddiejennings> After May 2nd, I'll know what my new work schedule will be, but until then I can't volunteer for meeting chair :(
16:08:55 <nirik> ha ha ha.
16:09:07 <nirik> it's because VipulSiddharth[m has a missing ] in nick... funny
16:09:48 <eddiejennings> Any takers for chairing 2022-05-12?  We have next week covered, so worst case we decide it then, but it's nice to have 2 weeks out.
16:10:00 <zlopez> It's too long for IRC :-D
16:10:25 <zlopez> eddiejennings: I can take it
16:10:46 <eddiejennings> Sold to zlopez!
16:11:01 <eddiejennings> ##info chair 2022-05-12 - zlopez
16:11:12 <eddiejennings> Thank you!
16:11:27 <eddiejennings> #topic announcements and information
16:11:27 <eddiejennings> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:11:27 <eddiejennings> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:11:27 <eddiejennings> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:11:29 <eddiejennings> #info Fedora 36 Beta is out
16:11:31 <eddiejennings> #info Fedora Final freeze is on going
16:11:33 <eddiejennings> #info thread on fedoraplanet on infrastructure list, chime in if you have thoughts on it
16:11:35 <eddiejennings> #info please help us with improving contribution to fedora infra https://discussion.fedoraproject.org/t/improving-contribution-to-fedora-infrastructure/38294/8
16:12:06 <eddiejennings> Any other new announcements?
16:12:23 <eddiejennings> I've just opened the above link, so I *will* get to reading it and responding today :D
16:12:42 <nirik> I need to still get back to the planet thread. Been burried.
16:14:01 <eddiejennings> We're still in Final Freeze for F36, right?
16:14:22 <zlopez> Yes, but we have RC compose :-)
16:15:06 <eddiejennings> Looks like there are no other new announcments.  Let's keep moving.
16:15:24 <eddiejennings> #topic Oncall
16:15:24 <eddiejennings> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:15:24 <eddiejennings> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:15:24 <eddiejennings> ## .oncalltakeeu .oncalltakeus
16:15:44 <eddiejennings> #info dtometzki on call from 2022-04-22 to 2022-04-28 (eddiejennings covering 2022-04-28)
16:15:44 <eddiejennings> #info nirik on call from 2022-04-29 to 2022-05-05
16:16:08 <eddiejennings> Any takers for 2022-05-06 - 2022-05-12?
16:16:17 <nirik> .oncalltakeus
16:16:17 <zodbot> nirik: Kneel before zod!
16:17:49 <eddiejennings> As with the meeting chair, it's nice to have on-call planned at least 2 weeks out.
16:18:34 <mobrien76> I can do that one
16:18:43 <eddiejennings> Thank you mobrien76
16:19:34 <eddiejennings> #info mobrien76 on call from 2022-05-06 to 2022-05-12
16:19:58 <eddiejennings> Speaking of on-call
16:21:02 <eddiejennings> #info Summary of last week: (from current oncall )
16:21:21 <eddiejennings> I'm not sure if dtometzki is with us, but I don't recall seeing any pings.  And there weren't any for the time I covered this morning.
16:22:46 <nirik> There was one yesterday about spammers... but nb took that on
16:22:59 <eddiejennings> Ah, thanks nb :)
16:23:01 <nb> what a rabbit hole that is
16:23:18 <nb> Which, if we have any apprentices that have some spare time maybe?
16:23:35 <nb> It'd help if you could look through https://pagure.io/browse/projects/ and make a list of spammy-looking ones
16:24:06 <eddiejennings> I may be able to help with that once new work schedule is solid.
16:24:35 <eddiejennings> Moving along.
16:24:37 <eddiejennings> #topic Monitoring discussion [nirik]
16:24:46 <eddiejennings> #info https://nagios.fedoraproject.org/nagios
16:24:46 <eddiejennings> #info Go over existing out items and fix
16:24:55 <eddiejennings> nirik the stage is your's!
16:24:57 <nirik> I think not much change here...
16:25:09 <nirik> we continue to get badges and resultsdb alerts anoyingly.
16:26:23 <nirik> I think we can move on.
16:26:31 <nirik> after freeze I want to try and clean up all the little things...
16:26:46 <eddiejennings> Noted, and sounds like a plan!
16:27:22 <eddiejennings> I know two weeks ago petebuffon and I did the learning topic on Tor services.  So last week would've been ticket backlog.
16:27:39 <eddiejennings> Was there a learning topic planned for this week?  If not, we can look at tickets again.
16:28:24 <nirik> I thought we had one, but I could be wrong.
16:28:34 <nirik> perhaps we should try and brainstorm up a few ?
16:28:40 <petebuffon> weird thing was immediately after the infra meeting I had a meetup about tor
16:29:22 <petebuffon> i think last time we just came up with the last two learning topics
16:29:34 <eddiejennings> Something that I want to learn about is making and restoring backups / having a DR plan using btrfs snapshots.
16:29:57 <nirik> eddiejennings: sounds like a nice talk... when would you like to give it? ;)
16:29:59 <eddiejennings> So I'd be happy to experiment with that in lab, and in two weeks present a learning topic about it
16:30:39 <petebuffon> ya i like that, btrfs commands can be a bit tricky to remember
16:31:31 <eddiejennings> The general (maybe crazy) idea I'm wanting to test, is have my KVM VMs on a btrfs volume, somehow have that replicating to another physical server, and if disaster strikes, basically mount the snapshot on the other server and turn on the VMs.
16:31:41 <eddiejennings> Then replicate back to the original server when original server is fixed.
16:32:03 <nirik> reminds me of drbd. ;)
16:32:10 <aheath1992> eddiejennings, so like hyperconvirge with BTRFS
16:32:24 <mobrien76> I could do a talk on terraform and aws some week if there was any interest?
16:32:40 <eddiejennings> mobrien76, I would be quite interested in that.
16:32:53 <petebuffon> terraform++
16:33:18 <nirik> sure. +1
16:33:27 <mobrien76> ok put me down at the next empty date
16:33:39 <cmurf> eddiejennings: We can talk about it offline if you want.
16:33:41 <petebuffon> maybe a deep dive on an essential fedora app?
16:33:48 <eddiejennings> aheath1992, Something like that.  The idea being that my VM storage is being replicated to another server fairly often so if something goes sideways, I can simply power on the VM from the other server until the original host is fixed.
16:34:03 <aheath1992> ack
16:34:49 <eddiejennings> So learning topic for 2022-05-12 will be BTRFS replication / backup / restore by eddiejennings
16:35:32 <eddiejennings> and 2022-05-26 will be mobrien76 with into to AWS and Terraform?
16:35:42 <nirik> sounds good to me.
16:35:46 <mobrien76> +1
16:36:07 <cmurf> Ping me when you get started. There are some caveats, e.g. btrfs send/receive on nodatacow images.
16:36:18 <eddiejennings> I shall cmurf :)
16:37:17 <eddiejennings> #topic Upcoming learning topics
16:37:18 <eddiejennings> ## info BTRFS replication / backup / restore [eddiejennings] 2022-05-12
16:37:46 <eddiejennings> ## info Intro to AWS and Terraform [mobrien] 2022-05-26
16:38:27 <eddiejennings> Did we find out if there was a learning topic scheduled for today (with the 20 minute we have left)?
16:38:48 <zlopez> I didn't find any
16:38:58 <nirik> yeah, seems not.
16:39:42 <zlopez> We can probably look at some tickets now
16:39:48 <eddiejennings> Sounds good.
16:40:02 <eddiejennings> #topic Fedora Infra backlog refinement
16:40:03 <eddiejennings> #info Refine oldest tickets on Fedora Infra tracker
16:40:03 <eddiejennings> #link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc
16:40:25 <eddiejennings> .ticket 10099
16:40:26 <zodbot> eddiejennings: Issue #10099: bvmhost-x86-01.stg.iad2.fedoraproject.org lost a disk - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10099
16:40:53 <nirik> yeah, I haven't done anything here... we have 3 machines that are needing disk replacement.
16:41:23 <eddiejennings> Noted.
16:41:36 <eddiejennings> .ticket 10528
16:41:37 <zodbot> eddiejennings: Issue #10528: src.fp.o: Cannot activate the "Prevent creating new branches by git push" hook - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10528
16:41:38 <nirik> I guess I can try and deal with it... (he says, hoping mobrien76 might take it)
16:41:55 <eddiejennings> doh, jumped the gun.  My apologies
16:42:10 <cmurf> Is there an SOP for disk replacement documented somewhere?
16:42:21 <cmurf> I'm curious what the steps look like...
16:42:48 <mobrien76> nirikI have to drop offline now but I can have a look tomorrow. Bad news is that machine is out of warranty
16:43:00 <nirik> cmurf: call dell, send them junk they ask for, they send a new disk to the dc, get tech approved to go in and replace it.
16:43:20 <nirik> mobrien76: yeah, hit me tomorrow morning and we can look at the 3 and figure out what to do.
16:44:02 * nirik can update that ticket so it's not on the top still
16:44:18 <cmurf> Yeah I mean software end of things :) like is it a reinstall or restore of some kind.
16:44:28 <nirik> ah, zlopez beat me to it
16:45:16 <zlopez> I added the backlog refinement comment :-)
16:45:16 <nirik> cmurf: oh, no, it's reboot to make sure it's seen by the controller (optional), then mdadm commands to readd it to the array
16:45:35 <cmurf> Ahh ok
16:46:03 <nirik> that part is easy. The calling and talking to people and getting thru paperwork is the anoying part. :)
16:46:12 <eddiejennings> speaking of, I'll probably be posting in ask Fedora soon about mdadm and some failed experiments in my lab :P
16:46:13 <cmurf> Always
16:46:47 <eddiejennings> How about 10528, any insight on that one?  Looks like nirik was the last comment on it
16:47:22 <nirik> ah yeah, I was hoping pingou would chime in there... I don't understand the error. Perhaps zlopez could look?
16:47:27 * aheath1992 interested in mdadm experiments
16:48:17 <nirik> I can ping pingou again in the ticket perhaps?
16:48:47 <eddiejennings> pinging anyone with ping in their name seems to be the only prudent course of action :D
16:48:57 <zlopez> I can try to look at it, but my knowledge of pagure is not that deep
16:49:30 <nirik> hum, looking at it again... I see why it's trying to do that... but not sure why it's denied.
16:50:06 <zlopez> Does the hook has correct ownership?
16:50:41 <nirik> hum, not really.
16:50:46 <nirik> it's a broken sym link. ;)
16:50:54 <nirik> lrwxrwxrwx. 1 root packager 62 Nov 20  2018 /srv/git/repositories/requests/rpms/python-u-msgpack-python.git/hooks/pre-receive -> /usr/lib/python2.7/site-packages/pagure/hooks/files/hookrunner
16:51:04 <zlopez> This is probably your answer :-)
16:51:15 <nirik> yeah.
16:51:17 <zlopez> python 2.7?
16:51:25 <nirik> especially since pagure is using python3.6
16:51:27 <nirik> yeah
16:51:46 <eddiejennings> Makes sense :)
16:51:48 <nirik> let me fix that and see if it fixes the issue.
16:51:51 <zlopez> Is this symlink created by pagure or by ansible playbook
16:51:55 <zlopez> ?
16:52:20 <zlopez> It seems that we probably found the fix for the issue :-D
16:52:53 <eddiejennings> one more ticket
16:53:08 <eddiejennings> .ticket 10541
16:53:09 <zodbot> eddiejennings: Issue #10541: Nagios checks for pagure.io - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10541
16:53:46 <nirik> zlopez: I think it's historical data from before we moved to 3.6. I don't know why it didn't get converted tho.
16:54:10 <nirik> so on this one we were waiting for what to monitor.
16:54:12 <zlopez> eddiejennings: This one seems to be waiting for reporter
16:54:31 <nirik> perhaps we should just brainstorm what makes sense here? but we don't know what it was that they noticed.
16:54:34 <zlopez> I will switch the status to Waiting on Reporter
16:55:17 <eddiejennings> Yeah. Status change would be appropriate
16:55:34 <zlopez> But https://pagure.io/fedora-infrastructure/issues makes sense to monitor, because if this doesn't work, people can't file tickets
16:56:08 <nirik> well, monitor how? that it returns a 200?
16:56:30 <eddiejennings> That would be reasonable
16:56:35 <eddiejennings> look for HTTP 200
16:56:43 <aheath1992> That should be simple enough
16:57:13 <aheath1992> start with 200 get more advance if needed
16:57:25 <zlopez> +1
16:57:57 <aheath1992> eddiejennings, I can take this one
16:58:17 <eddiejennings> Excellent
16:58:24 <eddiejennings> And for our last couple of minutes.
16:58:35 <eddiejennings> #topic Open Floor
16:58:42 <nirik> ok, note that nagios is "fun" :) but it can do this.
16:58:50 <eddiejennings> Any other comments, announcements, etc?
16:59:45 * nirik has another meeting. Thanks folks.
17:00:11 <petebuffon> eddiejennings++
17:00:13 <aheath1992> If someone can provide some background and information on https://pagure.io/fedora-infrastructure/issue/10509 i can get started on this ticket as well
17:00:16 <zlopez> eddiejennings++
17:00:16 <zodbot> zlopez: Karma for eddiejennings changed to 7 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:00:36 <eddiejennings> And we've hit 13:00 on the east coast of the United States; thus. . .
17:00:37 <eddiejennings> #endmeeting