Meeting started at 2026-08-27 15:00:47 UTC

The Meeting name is 'Infrastructure (2026-08-27)'

!chair @nirik:matrix.scrye.com @zlopez:fedora.im @jnsamyak:matrix.org @james:fedora.im @gwmngilfen:fedora.im @patrikp:matrix.org

The Meeting Name is now infrastructure

!hi

Michal Konečný: Michal Konecny (zlopez)

morning everyone.

hello

will wait a few for more folks to wander in

!hi

Gwmngilfen: Greg Sutcliffe (gwmngilfen) - he / him / his

oh hey, zodbot likes me this week

lucky

ok, I guess lets go ahead...

any new folks today?

Hi. 👋

ok, seems not.

anyone want the 10th? or shall we leave it to next time?

I can take it

excellent!

Any other announcements?

nothing here

ok, on to our fav section...

so we've had a few fun ones today for things that look related to the rhel upgrades

i think i sorted most of them, there's a bunch of PRs open we can get to at some point

i do not know what is going on with db-datanommer

it appears to be reporting that zabbix can't connect to it, so I'll try to troubleshoot that later

so... it's postgresql had been oom killed yesterday. It was restarted, but then it died again...

I gave the host more memory for now.

yeah, it has metrics, but the version string has a connection refused, weird

nt urgent, but definitely curious

not urgent, but definitely curious

just the postgresql connection?

yeah

i'll take a look, np

the rest of the recent live issues are accounted for I think

I have noticed these from time to time: SSH host bvmhost-x86-05.rdu3.fedoraproject.org not reachable on 22 (and bvmhost-x86-06).

It might be that they are under a lot of load (they run buildvm's)

true. i wonder if that check should be X fils in a row?

true. i wonder if that check should be X fails in a row?

its always a tradeoff between how fast do you want to know vs false positives

oh, and logserver also had a lot of space taken because the bugzilla toddler was spewing to logs.

ok - it just fired again because I'm running the synchtt-logs script atm

but it still seems high... so perhaps something else is happening

ok - it just fired again because I'm running the sync-http-logs script atm

so the dl servers seem under a lot of load

I think it was yesterday but i slightly saw openqa giving some kind of how do you say this error 90> or more. i think we increased that to 95% right. am I just seeing it wrong or its a different thing?

yeah, the openqa-workers...

huh, i thought i rolled that out

do you want me to make the ssh check more ribust nirik? say 3 fails in a row?

we could, sure...

or 2?

sure, 2 is probably enough

the item runs every 5m, maybe we could run it faster with N fails, then we get a similar window of alert (although with more network traffic,)

i'll make a ticket, it's a pretty easy one for someone to pickup

i guess the crashloop alerts were the toddlers too?

i can do it. I need to do the free disk thing too

i'm not seeing anything else scary in there, I don't think

oh, `Apache: Service is down` on proxy09 is back :/

I need to readd ipa servers to dns/ansible (but that will need a freeze break)

humf

in top tiggers?

I am not seeing it.

hmm, looks like a cluster of them on 21st Aug

might just be a blip

[horrible long zabbix url](https://zabbix.fedoraproject.org/zabbix.php?show=2&name=&acknowledgement_status=0&inventory%5B0%5D%5Bfield%5D=type&inventory%5B0%5D%5Bvalue%5D=&evaltype=0&tags%5B0%5D%5Btag%5D=&tags%5B0%5D%5Boperator%5D=0&tags%5B0%5D%5Bvalue%5D=&show_tags=3&tag_name_format=0&tag_priority=&show_opdata=0&show_timeline=1&filter_name=&filter_show_counter=0&filter_custom_time=0&sort=clock&sortorder=DESC&age_state=0&show_symptoms=0&show_suppressed=0&acknowledged_by_me=0&compact_view=0&details=0&highlight_row=0&action=problem.view&triggerids%5B%5D=56123)

there was a large chud of traffic then I think that hit the max connections on proxies.

aha

or I could be misremembering.

oh, the stg ocp cert needs doing if anyone wants to deal with it :)

zabbix01.stg.rdu3.fedoraproject.org: api.ocp.stg.fedoraproject.org SSL Certificate expires in 29 days: 18 days left

i can sort that next week if no one beats me to it

ah, so that 'friday' was last thursday, so it was at the tail end of the mass update/reboot outage...

I did a bunch of tweaking on ipsilon01/ipa yesterday. I hope it might mean our auth woes are in the rear view mirror... but time will tell.

I opened this to track the issues with IPA https://hackmd.io/_8K_2VCcT_afoTsrUY8a6A

It's hard to watch for failed logins... because I often see a failure, then a success and can only guess someone mistyped their password or something.

oh good. I can read it now. ;)

oh, so i found this ...

`openqa-x86-worker03.rdu3.fedoraproject.org` isn't in the openqa_workers group in inventory

it jumps from 03 to 05

so all the others got that macro set, but not 04

some of them are 'staging' (connected to openqa-lab01)...

(but not really staging)

Do we still want to reach to IPA folks for help? I wanted to first collect everything in the hackmd document before contacting them

aha, so I guess we can set that macro in the staging group too. sheikhlimon want to send a PR?

openqa_lab_workers should also be in the same group I guess?

yeah

sure

Michal Konečný: yeah. I think so... will look / add to that...

there's also a bunch of forgejo alerts that I need to go talk with the forge folks about. the monitoring works, but I;m not sure what action they want us to take (if any yet)

otherwise, I think we're done for zabbix?

I also see there's a user that tries to login every 15m and fails, since... forever. Perhaps I will mail them

yeah. I think so.

So, what shall we do next (vote on your choice):

1. backlog refinement

2. some discussion topic

3. go to open floor/end early

no strong preference - do we have any discussion topics?

nothing planned...

so, backlog it is. ;)

best escuse

best excuse

!ticket 12415

**infra/tickets #12415** (https://forge.fedoraproject.org/infra/tickets/issues/12415):**Update the “Common Bugs” footer URI to list the current version, rather than F38.**

● **Opened:** a year ago by rokejulianlockhart

● **Last Updated:** 4 months ago

● **Assignee:** Not Assigned

I guess I could do this one...

or someone else could.

It can be done in the package and build in f44-infra...

https://koji.fedoraproject.org/koji/buildinfo?buildID=2992828

or it could be done upstream and then the package updated.

although I am not sure where upstream is

i could use the packaging practice but i am wary of saying yes when I have enough on my todo

https://gitlab.com/fedora/websites-apps/themes/koji-theme-fedora

but that seems out of date.

I could just ping ryan again in the ticket?

lets do that.

!ticket 13288

**infra/tickets #13288** (https://forge.fedoraproject.org/infra/tickets/issues/13288):**Review / refresh the "request for resources" process**

● **Opened:** 4 months ago by adamwill

● **Last Updated:** 4 months ago

● **Assignee:** Not Assigned

so, this is updating docs/process... I think this is good to go... just have not had time to do it.

we should move them into docs from the wiki and update them...

but I guess we just need to do it? I can try... but not sure if I will get to it.

That seems to be a good thing to work on during freeze

yeah.

yeah

well, I can try... no promises tho. ;)

i haven't opened the links, is that openshift centric, or more general?

it's a more general process from 2009.

heh

we should definitely make it more openshift centric...

and discoverable

yeah, for sure. i'm just thinking that I have to do the stg matrix server sometime soon, which might a chance to walk through any docs we write up

We should probably update the PDR guidelines as well as we now use the tool Ryan created

yeah, and Vít Smolík is currently looking at adding that public-inbox instance in staging too.

so perhaps both of you could update it ?

as part of that...

yes, we definitely need to do that...

ok, let me comment/assign

done

Michal Konečný: Our sop needs some work too now... along with any places that sent people to pagure.io

one more?

!ticket 13069

**infra/tickets #13069** (https://forge.fedoraproject.org/infra/tickets/issues/13069):**Reserving a machine for koji `ci` channel**

● **Opened:** 7 months ago by lecris

● **Last Updated:** 4 months ago

● **Assignee:** Not Assigned

I'm inclined to just close this one now. We cannot afford dedicated ci machines on some arches...

and if we have more it would be less of a problem in general.

thats fair, for my limited understanding

anything anyone has?

i have a question about deployment regarding oraculum. currently needs to be **manually** deployed both to stg and prod. whats the standard for automatic deployment? i looked around other fedora apps and they're mostly automated i believe and 3 branch setup with main branch, stg and prod

Michal Konečný: oh, I was wondering if you might be willing to look at the mailman01 upgrade...

https://forge.fedoraproject.org/infra/tickets/issues/13519

we have a number of apps using s2i...

ie, when commits happen to their upstream repos, we build/deploy automatically

Oh, mailman01 is not RHEL10 yet?

no

There's missing packages, not sure how many

I can look at that

anything else?