Meeting started at 2026-06-04 15:00:02 UTC

The Meeting name is 'Infrastructure (2026-06-04)'

!chair @nirik:matrix.scrye.com @zlopez:fedora.im @jnsamyak:matrix.org @james:fedora.im @gwmngilfen:fedora.im @patrikp:matrix.org

The Meeting Name is now infrastructure

Good afternoon to all.

Hey

!hi

morning

Michal Konečný: Michal Konecny (zlopez)

!hi

Gwmngilfen: Greg Sutcliffe (gwmngilfen) - he / him / his

Looks like it's just the usual suspects here so I'll skip the new folks introductions.

Anybody want next week?

I can take it

I'll be here next week, but... packing up to head to the airport/flock, so I don't think I want to run the meeting. ;)

Have a good time at Flock!

I'm going to Flock as well, but leaving on Saturday

i wish I was 😕

make sure to have an extra good time, you have to keep the average up 😉

Kevin, will you be coming to DevConf as well?

will miss you (and lots of others) there...

sadly nope, heading back after flock.

You plan on having a bad time next week then? 😀

We'll see how much folks from our team will be at Flock

yeah

Any other announcements?

every day is a day of learning!

Checking the dates here and I made a mistake, it's not next week.

if you're lucky!

Unless there's nothing else, monitoring time!

🖥️ing!

so, firstly status updates

Nagios migration is a little behind but still aiming for all-done by end of next week

the reason I'm behind is because I had a lot of good chat with the COPR folks

that lead to me working on fixing up per-team notifications (at least in Matrix) and that's working for them now

anything in the "Copr" hostgroup, or tagged "team: copr" will report to both #buildsys and #noc

i will experiment further at some point with *excluding* things we don't need to worry about, but it's progress

i'm just wrapping up moving their remaining Nagios items today, then back to the rest of it tomorrow

ok, reporting

nothing jumps out at me that's live right now, did I miss anything?

nothing too much. it's kinda noisy due to update/reboots... (changed packages and brief downs)

yeah. i wonder how much value the package changes have, given our busy infra

bvmhost-s390x-01 has alerted on memory... I guess I need to rebalance things there... but it's complicated

but all that tweaking is off limits until I'm done with Nagios 😉

The httpd stuff still seems to be happening, but much less so since my change this weekend.

what changed? I missed that...

the bodhi stuff is because when it restarts it's down for a minute... ;(

https://forge.fedoraproject.org/infra/ansible/commit/8bbcceafdd2609d36b40699ae982e3589a9d0bf7

I bumped max connections up some

nice

is the countme alerts expected? they surprised me

my theory was that while it wasn't hitting max workers, it was having problems because there were so many connections anubis was rejecting... it couldn't clean up those fast enough.

I don't know if it's true, but it's a theory. ;)

i wish I lived in Theory, everything works there 😛

May I interject with an Anubis question? Or during open floor would be better?

In theory, theory and practice are the same. In practice, they are not.

sure, fire away

Should probably up the timeouts on those ... give them a couple more hours.

sure, easy to do

But they usually go away after I do a vacuum too, so maybe fine to keep as is and I need to get them vacuum'd automatically.

There were some in the past week or so, where something actually went wrong (python security update broke everything importing sqlite). So that's why they popped up for a couple of days.

I completely understand the issues with the scrapers and how annoying they are. But I was wondering if, in theory, there was anything that could be done about this?

So, one of the big advantages we have in terms of our work is that all of the tickets and docs are on the open internet. Times past, you could paste the URLs directly into for example Claude and it'd parse them. But lately Anubis has been blocking every attempt.

ah, maybe thats all it is then

I don't use it too much but sometimes it'd be very helpful.

I think there are ways around that, but I haven't tried/looked at details. Going directly should work if there's a useragent and a accept header, so perhaps there's some way to tell it to use that when doing direct queries?

needs investigation I would say

Hmm, thanks for the info. Anything else for the monitoring section?

the only other thing i see is the fmn queue had a few alerts, but perhaps it was just busy?

the python 3.15 mass rebuild is going on... that caused a lot of notices.

that'll be it I guess

I fixed bugzilla2fedmsg few times this week, thanks the zabbix for alerts 🙂

cool

theres a stg cert up for renewal but we'll get to that

all good from me this week I think

whats left in nagios now? just some small kibble?

yup

25 left

my crusade continues

25 services in main nagios, looks like... 9 in nagios-external. ;) nice

porting the copr checks has been ... messy. so hopefully the rest will be easier 😉

move on to next thing?

Shall we go to backlog refinement or does anyone have a learning topic, discussion, or something of that sort?

Backlog refinement it is.

Oh, I missed in announcements...

!ticket 11683

**infra/tickets #11683** (https://forge.fedoraproject.org/infra/tickets/issues/11683):**discourse s3 backup buckets no longer active**

● **Assignee:** Not Assigned

● **Opened:** 2 years ago by mattdm

● **Last Updated:** 3 months ago

oh yeah, this... lets see

I can make a note to ask mattdm about this I guess. I _think_ it may just be done/closable... but it would be nice to confirm that.

Maybe ask him on Flock?

I know there's backups, I just don't know if they have the image uploads too.

I can try and remember to, yes.

I'll update the ticket.

oh, we should add points / pri ?

low 3?

+1

Yeah, we should probably re-label these as we go over them. I think is the third thing the powers that be care about is "Work item", which would be "Task" for this one.

relabled.

Next one...

!ticket 12426

● **Assignee:** Not Assigned

● **Last Updated:** 3 months ago

● **Opened:** a year ago by lnvso

**infra/tickets #12426** (https://forge.fedoraproject.org/infra/tickets/issues/12426):**fedora 40 and above missing on images on data-analysis.fedoraproject.org/csv-reports/images**

No idea what to do with this one. ;) I guess we need to do some work, but not clear to me what... and the old pre f40 stuff is using the old counting I think, so it can't be made to work anymore...

i hear mattdm is rewriting brontosaurusifier, maybe give it to him? 😛

Data on EOL releases is needed too? After they've gone EOL?

Yeh, realistically I think the new FPL probably needs to get some resources allocated to it. But it's not going to be easy.

yes.

Small possibility that someone does a PoC for day of learning or something, that then lives in prod. for 15 years.

so I guess we could just close this... but then it won't be tracked anywhere.

I guess lets let it sit more and hope for someone to work on it?

Should we relabel it? Doesn't seem like "low trouble".

high trouble, 5 points ? or 33?

high trouble, 5 points ? or 3?

perhaps someone could vibe code a replacement. ;)

I think we need to investigate a little first as it's not really clear what needs to be done

yeah, its difficult to get requirements out of that

(at least to me)

You are not alone

and also the data that we have changes right? pre f40 it was one thing, post f40 its different.

My guess is anything that turns countme data into graphs is going to solve at least 90% of the problem.

(or am I wrong there)

i would definitely ping mattdm on that one, he is working on new analysis stuff anyway

oh wow. there's a docs page:

https://docs.fedoraproject.org/en-US/infra/sysadmin_sops/dnf-counting/

(or at least he said he was to me earlier today 😛)

Didn't Matthew generate shiny graphs for his presentations when he was FPL using countme?

smooge generated them for him. ;)

There's smooge's pre countme data ... which uses awk and other things I don't really understand, and all of the graphs are generated from that. But that's hard coded to stop at f39. So technically someone can update that code for newer releases, which is how the original ticket was sized.

I can try and mention this to mattdm also when I see him.

For countme AFAIK there's never been anything automated to generate graphs.

cats are now growling at each other. Gonna go feed them. ;) I didn't have anything else for open floor or anything.

Less than 5 minutes to go, open floor time!

so i guess I should go post something about the imminent demise of Nagios to Discourse, right?

i kinda lost track of updating that topic...

Blog post would be nice as well

definitely a blog post once it's done

Probably worth a blog post, not sure what good things will happen on discourse though.

I like reading the Discourse posts. In my experience 99% of the time people behave very well and make completely reasonable points. When compared to the rest of the internet...

mainly i want a wide net for possible people who are expecting email from nagios. copr folks missed a disk-ull alert last week because they didn't know that check was no longer in nagios 😉

mainly i want a wide net for possible people who are expecting email from nagios. copr folks missed a disk-full alert last week because they didn't know that check was no longer in nagios 😉

this is now fixed *for them* but there will be others

You could try some of the mailing lists also.

If you want a wide net.

in other words, i already know the answer, I just need to go write it

Fair enough ... I assumed all the people who would be impacted would be in noc a bunch anyway.

i know the ELN folks are interested but I'm sure there's more

anyway, fair enough, i will write words

We're at time.