Meeting started at 2026-08-20 15:00:11 UTC

The Meeting name is 'Infrastructure (2026-08-20)'

!chair @nirik:matrix.scrye.com @zlopez:fedora.im @jnsamyak:matrix.org @james:fedora.im @gwmngilfen:fedora.im @patrikp:matrix.org

The Meeting Name is now infrastructure

Welcome everyone

!hi

Michal Konečný: Michal Konecny (zlopez)

hello

morning

How it is going?

!hi

all good. i was recently added to the fi-apprentice group... thank kevin for adding me

hope to learn more and contribute

hmm zodbot isn't talking to me apparently

I hope he doesn't hold any grudge against you

i think he gave up

a bit early to tell. ;)

sniff the logs ig

Let's go to first topic

Anybody new around?

@sheikhlimon:matrix.org I think

I honestly just kinda lurk around meeting rooms to see what is going on

nothing wrong with that, it's why we do stuff in public :)

Ive been here for some time.. actually.. nice to meet you

ah

mb

It doesn't seem that there is anyone, so let me skip to next topic

We have a comfortable chair and looking for people who want to sit in it

I could probibly take it... haven't done it in a while...

I don't see anybody else raising hand, so it's yours

Anybody else for 2026-09-03?

i can do that, it's probably my turn

I just realized how little of vacation I have left.

The summer is slipping away. ;)

schools already went back here :)

I think that is enough for now

so is time to study

Let's move on

Anything else to announce?

special celebration when?

🎉

i think if you do 3, there is an animation on element

🎉🎉🎉

yup

we really need to try and engage ipa folks about that... see if they have any ideas.

Oh, it's so close O.O

does that mean development for F45 is paused?

or blocker review time?

https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/#beta-to-final-freeze

I agree, I'm not sure why this is only happening for accounts.fedoraproject.org and IPA web UI

yes, blocker reviews will start up

yeah, it's strange. Also the weird caching in the ipa web ui... possibly related? possibly not?

Anything else to announce?

it does only seem to affect folks who have an otp enrolled...

OTP cache?

first login goes through. i noticed when entering wrong otp by mistake can trigger it actually. also quickly logging in and logging out

I saw some threads failing for otpd

But that didn't seem related

yeah, it's a puzzler.

if i investigate noggin would i be finding something?(hopping there where login stuff happens now)

if i investigate noggin would i be finding something?(hoping there where login stuff happens now)

Let's move on to monitoring

so firstly I realised this week that we rarely look at the stg zabbix. we probably should at some point ;)

It seems to affect noggin and also the ipa web ui, so it seems to be something in ipa... the lower layer.

staying on prod for now though, i don't see anything too crazy in the top 100

plenty of stuff related to the reboot cycles, and the usual scraper-based alerts

We had issue with badges consumer yesterday, but it's dealt with now

scrapers have really been mitigated by those ua blocks. It's lovely for however long it lasts.

ryanlerch++

so yeah, there will be more noise from the updates/reboots later today, but then hopefully it looks better next week

i *think* i fixed the ipa01 free-ids check with a new keytab

oh, do we want to bump up those openqa worker alerts... they were really busy, but fine...

hmm, yeah, we can do that

high memory I think it was?

right

looks like its set to 90, we could bump it to 95

looks like its set to 90%, we could bump it to 95%

seems to peak around 92

sounds good.

Sheikh Limon: do you want to send a PR for that? since you want to do more zabbix :P

it should be a oneliner, but I can point you to where to look

Oh, vmhost-p09-copr03 seems pretty haunted. I guess thats not monitoring really tho...

yeah i can do that

zabbix thinks it's dead

but i guess it didn't come back because of the disk?

it is currently. I rebooted it, it didn't come back. I rebooted it again and got re-installed. then it didn't come back from the reinstall.

¯\\\_(ツ)\_/¯ not sure what to do with it. ;(

get the macro name from the trigger, add it here: https://forge.fedoraproject.org/infra/ansible/src/branch/main/inventory/group_vars/openqa_workers#L31-L33

sounds like my experience with the power9

on it

Yeah, I'll poke at it some more as time permits I guess.

anyhow, I don't see much else monitoring wise. Oh wait...

is there a reason why the el9/10 test hosts are out?

i can go poke them if not, just spotted it

I rebooted them, but they should have been back...

OK. also i wanted to ask you want to do with the new NFS disk space alerts. I assume we want a higher threshold?

I was gonna show a graph of the download servers... switching them to BBR really helped them pump out more bits. ;)

possibly yes.

https://zabbix.fedoraproject.org/history.php?action=batchgraph&itemids%5B67769%5D=67769&itemids%5B67801%5D=67801&itemids%5B67819%5D=67819&itemids%5B67874%5D=67874&itemids%5B67906%5D=67906&itemids%5B70272%5D=70272&itemids%5B136105%5D=136105&graphtype=1

```

FS [/mnt/fedora_koji/koji/compose/ostree]: SSpace used: 97.6%

FS [/mnt/fedora_koji/koji/ostree]: Space used: 97.6%

FS [/mnt/fedora_koji]: Space used: 97.2%org Linux:

FS [/fedora_backups]: Space used: 97.3%

```

```

FS [/mnt/fedora_koji/koji/compose/ostree]: SSpace used: 97.6%

FS [/mnt/fedora_koji/koji/ostree]: Space used: 97.6%

FS [/mnt/fedora_koji]: Space used: 97.2%

FS [/fedora_backups]: Space used: 97.3%

```

Sheikh Limon: has offered to work on the time-until-full ticket, so this may be shortlived anyway

well... mostly I wanted them monitored for the graphing... the netapp autoresizes them automatically... so amount is not too useful.

oh, ok

I guess we could make it 98% since it resizes around then?

or perhaps "less than 50Gb remians" or something like that

I need to cleanup backups. Thats on my todo

so we get a warning if the netapp does not do its things

so we get a warning if the netapp does not do its thing

yeah, that would work too, since it probibly wouldn't happen

yep

ok, let me make a ticket so i don't forget

thats probably a useful check for a lot of filesystems, but not all

do i do 95 or increase it?

I'd suggest 98% or 'less than 50G'

less than X would be a new check

i've logged 13516 for that

the % is configurable per host/fs though so we can target the netapp and make it 98% for now

Sheikh Limon: if you're doing that too, you'll want an example of how to target an LLD macro ;)

https://forge.fedoraproject.org/infra/ansible/raw/branch/main/inventory/host_vars/pagure01.fedoraproject.org

last line

got it

ok, we do need to review stg zabbix at some point, but we're already at :45

oh, last point

If there isn't anything for open floor I think we can look at it now

I've enabled notifications from Zabbix to #forge-notifications:fedora.im so the forge team get their alerts

obviously we still get them too, but just fyi, they should be being proactive about that

ok, all done :)

So what now? We have 13 minutes

Open floor?

Backlog refinement?

open floor sounds nice.

+1

Oh, one thing I realized:

Anything we want to talk about here?

oh, already? yay

i don't have anything. i'm looking at how to flip pagure to the ro version ;)

That is being worked on

I'm reinstalling vmhost-p09-copr03 again just for fun.

I finished the ticket where we lost the release-monitoring.org updates when the bugzilla API token expired

And I also finished the quay.io preparation for flatpaks 🙂

kevin gave a cookie to zlopez. They now have 140 cookies, 9 of which were obtained in the Fedora 44 release cycle

awesome

oh, it'll talk to YOU

but still not to me. weird

So I didn't get one cookie from Gwmngilfen 😕

ok one done.. hope i did it right

maybe I'm out of cookies :)

looks ok

shall we wrap up?

The quay.io switch will probably happen during F45 preparation as there will be mass rebuild of flatpaks for F45

linter failed but i ran lint

And we are at the end of time