Meeting started at 2026-02-26 17:00:06 UTC

The Meeting name is 'Infrastructure (2026-02-26)'

!chair @nirik:matrix.scrye.com @zlopez:fedora.im @jnsamyak:matrix.org @james:fedora.im @gwmngilfen:fedora.im @patrikp:matrix.org

The Meeting Name is now infrastructure

Will wait for folks to arrive... and if there's any new folks, please introduce yourself.

!hi

Greg Sutcliffe (gwmngilfen) - he / him / his

!hi

None (patrikp)

morning everyone.

!hi

Michal Konecny (zlopez)

ok, I guess lets go ahead and dive in...

anyone like to take the chair on the 12th?

i'd love to but as ever I'll have to go at :30 so probably a bad idea

I can

cool. Sold.

Any other announcements this week?

Not really an announcement yet, but I have the new secure-boot signing working. Still need to test and write up docs, but great progress on it.

it's the oldest open ticket we have I think. I know there's a email about it in my mailbox thats many years old. ;)

That is probably everything I can think of right now

alright, moving on then...

vit wasn't able to be here, but provided:

one ping for a playbook run

broken httpd on people01

mailq complaints from zabbix

!oncall

● @nirik:matrix.scrye.com (kevin) Current Time for them: 09:10 (US/Pacific)

The following people are oncall:

If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)

Does someone want to take it next week?

I will be chairing, so I can take it as well

ok.

so, a few interesting things here

zabbix correctly caught a bunch of issues with httpd on people01, which was nice

ssl keeps failing on checking whatcanidoforfedora.org - I just opened a PR which should help, but I also plan to try and find out why it's failing for that domain

I looked at the people01 issue. Might be solved. I think it was scrapers of cgit downloading archives of every commit... so I disabled those links in cgit and also increased some capacity in apache config there and it's been ok since then.

i had a quick look at postfix on smtp-mm-ib01, seemed like it's just an accumulation of connection-fresued junk

i had a quick look at postfix on smtp-mm-ib01, seemed like it's just an accumulation of connection-refused junk

I would like to ask about the toddler-unretire-packages queue, that seems to be stale for some time, but it's acked on nagios

I'll note that the asknot openshift app logs nothing at all. No way to tell if it's hitting maxclients or something...

this is openssl s_client which I think terminates at the proxy, right?

Anton Medvedev was working on getting that working, but I think in the mean time until it's being tested in prod we should delete that queue and it's monitoring.

monitoring will auto-remove if you delete the queue

yeah, I am not sure where it's failing. ;) Might be the proxy. I did get it to hang once here...

in zabbix yes, not in nagios sadly. ;(

So should be staging only then

well i'm thinking it's time for removing the queue checks in nagios anyway, they are shown to be working in zabbix now

yeah, I think so...

They are much better in zabbix

although... this is not alerting in zabbix?

it is. just not in #noc because it's Warning level

oh, it is. right

https://zabbix.fedoraproject.org/tr_events.php?triggerid=57645&eventid=1415295

yep. Found it.

it would alert in noc if it kept increasing

huh, I am not sure what created that queue... it's not in ansible...

perhaps it was at some point and was removed?

oh, hmm, no. there's only a warn level trigger right now - and now i recall I made a ticket to fix that, and I think it's in this sprint too. will sort tomorrow 😉

anyhow, to remove it is just rabbitmqctl on rabbitmq01 deleting the queue, then rm the .cfg file on noc01 under /etc/nagios and restarting it.

i can PR the nagios bit in ansible, np - but is it still technically broken in ansible because of the rdu3-iso hosts?

there's nothing to change in ansible that I can see... it adds that .cfg monitoring file when it adds a queue, but I don't see that queue actually being added right now.

oh, hmm

ok

I fixed nagios config before freeze, so ansible should correctly deploy it.

i'm too used to all of nagios being in the monolithic role, clearly 🙂

anyhow, I can do that or someone else can if they want.

i can deal with it, sure

any other monitoring stuff?

moving on, I don't see a great deal of other interesting things in the top100 report

lots of load-avg and disk as usual, the rest seems stuff we've discussed

yeah, we should sort out the disk stuff, probibly after freeze.

wiki01/02 seems to be high on load list again this week

yeah, scrapers love them. ;(

indeed

anyway, seems quiet. move on?

So, shall we do backlog refinement, some learning thing TBD or end early?

+1 for backlog

ok, lets do at least a few. ;)

https://forge.fedoraproject.org/infra/tickets/issues/12727

Vit isn't here, so not sure the status...

Does anyone know?

No idea

ok, I'll ask for status in ticket.

Moving on...

https://forge.fedoraproject.org/infra/tickets/issues/12750

thats mine, well at least i logged it

i can't fix it though, so unless we find someone to work on it ...

The only one who might have bot experence I can think of here is ryan...

tbh we probably should rewrite the whole bot in something not 4+ years old, but thats a wider discussion 😉

i stood it up to see if we all liked the workflow, but we all know what happens to temporary things 🙂

Is it worth trying our friend AI to see if it can come up with a fix?

i did

The standupbot is really useful

oh, 100%, but the code is .... not nice

we could probably write a maubot plugin faster than fixing it

but it's not urgent, what we have works - it just cant use a fedora.im account , thats all

We can add it to next sprint and get somebody working on it

note that auth has changed too when we switched to MAS...

you no longer get a access token from a client

(see my flailings in the ticket on moderation bot)

hmm, maybe I should retry it. the current issue is that it *cannot* use an access token, it has to use user/pw

ah yeah, that would still need to change

thought so. tbh, a maubot plugin is almost certainly the way to go

since we already have that working

yeah

maybe mark it for next sprint, i think i can still write maubot code

so, what should we do here... rescope this to be a re-write? but until we commit to doing it the ticket is kinda pointless.

I could try that as well 🙂

ok. we should also when we triage these give them priority and points...

yeah i logged it so as not to forget that we were still using an ansible.im account for the bot, and to put my findings somewhere, but some idea of where we ant to go is a good idea

yeah i logged it so as not to forget that we were still using an ansible.im account for the bot, and to put my findings somewhere, but some idea of where we want to go is a good idea

ok, let me take a stab

you want to mod the ticket?

i've commented for now, I will probably write a new ticket. on my todo for tomorrow

ok, shall I just leave that one alone for now? or how about we close it and you can open a new one?

yeah, i'll close it tomorrow

ok.

and link the two tickets

sounds good.

ok, on to next ticket?

and Gwmngilfen is probibly leaving?

https://forge.fedoraproject.org/infra/tickets/issues/12759

i got time for one more 🙂

so, I am not sure how to do this one in a clean way.

I'd be inclined to just close it, but we could also update it and ask for any further ideas.

We can say, there is zabbix available now and just close it

Zabbix provides more details for those who want them

and can send alerts to multiple rooms, if desired

sure, but thats not something consumable for end users/contributors

true

They wanted a way for contributors to easily see status of those services.

Do you think the current status is missing something?

zabbix api -> auto generated page? 😛

I don't think so... others might

-1

i'm not serious, to be clear

We don't want users to be overwhelmed when visiting the page

people keep asking for that tho... not realizing how bad an idea it is in the end.

anyhow, how about I update the ticket and ask for any further ideas, if nothing close early next week?

i do like what you said about a secondary page (or pages) with more detail

i wonder if that *could* be generated. perhaps something like uptime-kuma could present that

but it would not be the frontpage of status.fpo

+1 from me

yeah, it would be something else...

i'd be ok with closing too, tbh. I think we're all clear that status.fpo is fine, at the least

I suppose for contributors (who are hopefully more interested in detail, we could have a zabbix dashboard of those things they might care about)

oh, thats a good plan

i have "making more dashboards" on my todo anyway

but still I worry about 'koji01 is down' and if you don't know that there is a koji02 and they are load balanced, you would think the service is down, etc.

ie, it's hard to know the details of what things mean sometimes

well, that plays into some of the service-level stuff i've ben researching

i.e not getting 0 alerts from the proxes when one haproxy backend goes down

i.e not getting 30 alerts from the proxes when one haproxy backend goes down

All of that sounds like a quality of life improvements

the service says something like "alert when more than N of these is alerting" and you could build a dashboard on that

sure, but I think this all depends on what the audience is for the data... ie, how it's presented.

oh, yes

I mean I care if one proxy shows something down, depending on what it is it may not affect users of that thing at all.

i think you're right to see if we can clarify/close the ticket, and investgate dashboards in the background

because we need them anyway

anyhow, how about I close for now and also perhaps redirect to discussion if we want to try and discuss further?

if it looks like something we can do in the future... we can always revisit

+1

and i have to run 🙂

thanks Gwmngilfen

so, I guess lets go to open floor...

We are almost at end anyway

Spring is almost here! ☀️

yep

So I had some luck with the registry.fp.o to quay.io flatpak migration, but it seems that I will need to discuss the changes with somebody from flatpak team

I already got contact, so will try to reach to them

cool.

Thanks for working on that

One flatpak is on quay.io https://quay.io/repository/fedora-flatpak-testing/0ad

But I found out that the quay.io doesn't support nested organization, so it needs to be outside fedora organization 😕

That is what I want to discuss

hum... can they not just be repos in an existing org?

although I suppose we could split all the flatpaks out into a seperate org

so you can't have a org/repo/repo right? or can you?

They can, but that is another problem, bot account can't have organisation wide permissions

No, that was throwing 401 error for me 😕

we typically only allow the bot upload perms...

but then we need to create the repos

The quay.io bot has permission per repository

right, so we would need a process to make the repo and grant the perms to the bot

Yes, this is why I need to discuss it with flatpak folks

yeah, makes sense.

It's possible, but will need some automation on top of it, or some documented process for adding new flatpak repository

perhaps it could be in the new package toddler? if it's a flatpak, create repo, etc.

flatpaks are added via the scm-request thing...

I wrote that toddler, so I know, it's possible to just listen for messages

(of course we still would need a one off to create all the existing ones)

As I said, it would need some discussion, but the changes on bodhi should work

excellent.

anything else for the last minute? ;)

Have fun with rest of your day 🙂

always.

Thanks for coming!