18:07:28 <phsmoura> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
18:07:29 <zodbot> Meeting started Thu Oct 20 18:07:28 2022 UTC.
18:07:29 <zodbot> This meeting is logged and archived in a public location.
18:07:29 <zodbot> The chair is phsmoura. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
18:07:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:07:29 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:07:29 <phsmoura> #chair nirik mobrien aheath1992 smooge
18:07:29 <zodbot> Current chairs: aheath1992 mobrien nirik phsmoura smooge
18:07:29 <phsmoura> #meetingname fedora_infrastructure_ops_daily_standup_meeting
18:07:29 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:07:29 <phsmoura> #info meeting is 30 minutes MAX. At the end of 30, its stops
18:07:29 <phsmoura> #info agenda is at https://board.net/p/fedora-infra-daily
18:07:40 <phsmoura> sorry I was late
18:07:52 <phsmoura> #info reminder: speak up if you want to work on a ticket!
18:07:52 <phsmoura> #topic Tickets needing review
18:08:11 <phsmoura> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
18:08:54 <Ebeneezer_Smooge> hrllo phsmoura
18:09:07 <Ebeneezer_Smooge> hope you are doing ok
18:09:39 <phsmoura> hi Ebeneezer_Smooge, yes Im fine thanks and you?
18:10:01 <Ebeneezer_Smooge> I am doing ok. I can work on 10950
18:10:40 <phsmoura> .ticket 10950
18:10:41 <zodbot> phsmoura: Issue #10950: read-only /var/tmp/ on buildvm-ppc64le-23 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10950
18:10:47 <phsmoura> cool
18:10:56 <phsmoura> low,low,ops?
18:11:17 <Ebeneezer_Smooge> yep
18:11:42 <phsmoura> ok
18:11:56 <phsmoura> .ticket 10948
18:11:57 <zodbot> phsmoura: Issue #10948: CentOS Stream 9 VM Request for Minetest game server - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10948
18:12:15 <phsmoura> Im not sure about this one. low gain med trouble ops?
18:12:47 <Ebeneezer_Smooge> low/high/ops/meeting
18:13:46 <phsmoura> meeting?
18:14:41 <Ebeneezer_Smooge> sorry probably the wrong flag. It needs someone with amazon AND it needs approval from bcotton or mattdm that this is important
18:15:13 <Ebeneezer_Smooge> s/with amazon/with amazon rights/
18:15:30 <Ebeneezer_Smooge> low gain/high trouble/ops
18:15:47 <phsmoura> we have aws flag, the reporter added mattdm in cc
18:16:44 <phsmoura> updated with aws flag
18:17:26 <Ebeneezer_Smooge> ok so buildvm-ppc64le-23 has a bad disk
18:18:26 <phsmoura> you solving it already
18:18:35 <Ebeneezer_Smooge> I am shutting it down as it is not possible to repair without a reboot. that is 10950
18:18:44 <Ebeneezer_Smooge> s/reboot/rebuild/
18:19:10 <phsmoura> yep. Im going move to releng we have 1 ticket there
18:19:13 <phsmoura> #info https://pagure.io/releng/issues?status=Open
18:19:24 <phsmoura> .releng 11102
18:19:25 <zodbot> phsmoura: Issue #11102: Remove of private-c99-port branches - releng - Pagure.io - https://pagure.io/releng/issue/11102
18:20:42 <phsmoura> low trouble high gain ops I guess?
18:20:49 <Ebeneezer_Smooge> yep.
18:20:55 <Ebeneezer_Smooge> med trouble maybe
18:21:04 <Ebeneezer_Smooge> I don't know how easy it is to remove branches from releng
18:21:24 <jednorozec> med
18:21:58 <phsmoura> done med high ops
18:22:39 <phsmoura> #topic Planning, Upcoming work and Open floor
18:23:01 <jednorozec> f37 was nogo
18:23:04 <phsmoura> anything for open floor?
18:23:09 <jednorozec> so there will be new RC request
18:23:49 <jednorozec> there was a glitch in koji and some packages went unsigned, so I did some clenup today and updates* composes are out for all releases
18:24:10 <Ebeneezer_Smooge> yesterday there was an extended outage on a Power9 system bvmhost-p09-03
18:24:47 <Ebeneezer_Smooge> nirik got it back up but the guests seem to be reporting btrfs errors
18:25:16 <Ebeneezer_Smooge> I would suggest we get them all out of of koji as usable
18:25:17 <phsmoura> what time that happened Ebeneezer_Smooge, do you know?
18:25:57 <Ebeneezer_Smooge> i think kevin got it rebooted at 00:00 UTC or so
18:26:22 <Ebeneezer_Smooge> I had it power cycled earlier in the day but it was stuck on needing a password until he did the final work
18:26:48 <Ebeneezer_Smooge> jednorozec, what is needed to disable builders in koji as I am not sure these boxes are 'working' in any good sense
18:27:05 <phsmoura> I think I had a script running about that time I was gathering info from our infra so we can update the firmware it lasted about 2 hrs to run
18:27:17 <phsmoura> but I only hit the mgmt machines
18:27:17 <Ebeneezer_Smooge> I think -23 was disabled earlier by sharkcz but I just checked -21 and it is also complaining about btrfs errors
18:27:41 <jednorozec> koji disable-host show do the job
18:29:49 <Ebeneezer_Smooge> ok I am going to reboot a couple of these vm's and see if they are recoverable. -23 is definitely a rebuild
18:30:44 <Ebeneezer_Smooge> phsmoura, do you have ability to ack things in nagios?
18:31:24 <phsmoura> sorry, what do you mean ack things?
18:31:53 <Ebeneezer_Smooge> go into nagios and acknowledge an alert so that it doesn't ping #fedora-noc every 10 minutes
18:32:58 <phsmoura> no I dont, but can do it if theres a sop or something
18:35:27 <Ebeneezer_Smooge> ok it might need an FBR to allow you to do that .. I will see if I can still do myself and then talk with you in #fedora-noc about what I am doing
18:35:41 <Ebeneezer_Smooge> that wya you can do it once whatever change is needed
18:36:43 <phsmoura> ok. Im also looking for a sop about that
18:37:07 <phsmoura> we passed a bit the time today thanks all for attending
18:37:07 <Ebeneezer_Smooge> yeah I may never have read one
18:37:15 <Ebeneezer_Smooge> s/read/written/
18:38:28 <phsmoura> I see, ok. if you ping me we can work together
18:39:13 <phsmoura> #endmeeting