Meeting started at 2026-06-25 15:00:09 UTC

The Meeting name is 'Infrastructure (2026-06-25)'

!chair @nirik:matrix.scrye.com @zlopez:fedora.im @jnsamyak:matrix.org @james:fedora.im @gwmngilfen:fedora.im @patrikp:matrix.org

The Meeting Name is now infrastructure

!hi

Gwmngilfen: Greg Sutcliffe (gwmngilfen) - he / him / his

!hi

Michal Konečný: Michal Konecny (zlopez)

Welcome everyone

today in its-too-hot-everyone ... 😅

today in its-too-hot-everywhere ... 😅

it is

morning

Hey

Anybody new around?

It seems no, let's move forward

next week is questionable for me, it's the last day of the school term...

I am going to out next thursday (and next friday is a us holiday) and the following monday. ;)

may have kids around

I will be off on next friday and monday as well

But I would like somebody else to take this over next week

or we just cancel I guess?

We can, if there is nobody to chair

i'll *probably* be here

so put me down, and if we have to cancel, so be it 😉

OK

Eh, seems better to cancel

But I guess we can decide that later too.

Let's occupy the week after if we have to cancel the next one

I suppose I could take that one?

Sold!

💰️

Let's go to next topic

I assume you meant !info and not !topic

oops. yes, not enough coffee.

Anything else to announce?

not from me

nope

In that case let's move forward

so, for open alerts, it's all just disk warnings atm i think

I think hopefully I solved (for now) those apache alerts.

ooh

nice

gwmngilfen has already given cookies to kevin during the F44 timeframe

i will hope to see them much lower in the top100 next week then 😉

Yeah, as far as I can tell it was hitting the max workers... and then just queuing requests until it went under the limit. I raised the limit and it seems to have helped.

I caused some alerts on staging today as I'm playing with moving IPA to RHEL10

np, thats why staging gets its own room

i've been testing some CI monitoring with Cristian Le but I don't know if we should go that way. maybe a topic for open floor, though

(the limit was 4k, I upped it to 6k). You can see when.

Living in clip ;)

data ftw

i dont see much else on the top100 that isn't explainable

i added zabbix_agent to a bunch more hosts this week, so there was some cleanup from that

and i still need to do the secure hosts

i had a chat about how to add templates for openshift apps with @amedvedev this morning too, I have a plan for where that goes in ansible

(zabbix templates that is)

cool.

and next week is the great nagios / collectd cleanup I guess?

i'll start it, certainly

hopefully it won't take too long, just a lot of "state:absent" and running playbooks

yep.

I guess we can retire noc02 probibly too then, I think it only has nagios on it.

but noc01 still does have many other things.

maybe. i do want to put "something" outside of rdu3 that can alert us if it thinks we're suddenly living in a buble

maybe. i do want to put "something" outside of rdu3 that can alert us if it thinks we're suddenly living in a bubble

but it can be a cron script, it doesn't need a ui

could go on any of our sponsored hosts I guess

i *think* we'd notice, but it's good depth

after that, on lower prio, is dashboards, service levels, and some smarter checks

monitoring fin, I think?

OK, let's move on

!ticket 13175

● **Assignee:** Not Assigned

● **Last Updated:** 4 months ago

● **Opened:** 4 months ago by zlopez

**infra/tickets #13175** (https://forge.fedoraproject.org/infra/tickets/issues/13175):**Migrate fedora-pdr repository to forge.fedoraproject.org**

This one is still blocked from what I remember

Or is there any progress on private tickets on forge.fedoraproject.org?

There is work being done, it's not done yet.

If it's not done by end of next month we will have to figure out something.

I'll think on it and see if I can think of anything.

Something that could be tracked, has FAS integration and allows to do it privately, that would be difficult

yeah.

Hopefully the private tickets will be resolved by then

Let's go to next one

!ticket 13224

● **Assignee:** Not Assigned

**infra/tickets #13224** (https://forge.fedoraproject.org/infra/tickets/issues/13224):**AWS Permissions Request: KMS Key for HashiCorp Vault Auto-Unseal**

● **Opened:** 3 months ago by mvadkert

● **Last Updated:** 3 months ago

hum, I guess I missed this one...

it doesn't have our points/trouble/etc?

3 / low / day-to-day ? and I can try and do it soon?

I think I just added the aws label as I wasn't sure about the points value

private discourse category?

but i won't derail, lets come back to it

will that allow people to post in it?

Added the missing labels

Thanks!

➡️ open floor, I'll sketch something

Gwmngilfen: Do you want to switch now?

i just mean i can talk more about that idea when we get there

up to you when that is 🙂

So let's do one or two more tickets and then switch

I need to go feed cats soon too... ;)

!ticket 13239

● **Last Updated:** 3 months ago

**infra/tickets #13239** (https://forge.fedoraproject.org/infra/tickets/issues/13239):**dns errors in openshift**

● **Opened:** 3 months ago by kevin

● **Assignee:** Not Assigned

Did we find the cause of this?

nope. :(

I was hoping rhel10 named would fix it, but no.

We should probably mark this as investigation

If anyone has cycles to investigate, please do

I guess we could also file a bind bug and see if we can get more info from them

Good idea

I can try and do that if no one else.

If you can do that, that would be great

Let's switch to open floor then

ok, so first, on monitoring

i've been working with Cristian Le on some Jenkins monitoring... which is nice but it targets hosts we don't control

that means (a) we can't help if there is an issue, and (b) there's nowhere really to put it in Ansible

I *think* that reduces us to basically being a conduit for Matrix notifications, and I don't know if thats something infra should offer? or offer some other way?

yeah, I wonder if they shouldn't have their own monitoring for that? which they can manage as they like?

i'm curious what folks think the edges of our monitoring look like

I think it's fine, if it goes to a channel for people who can do things?

ok, but where do we represent that in ansible?

right now we're putting templates in roles and then lettings hosts declare themselves and add themselves to templates. That doesn't work if there's no role and no host, we'd have to write something new?

I guess it would have to be on the zabbix server itself?

well, a role there yes. you could declare a "fake" zabbix host to hold the data in the ui, but the server would have to run the role

or we would need to make a zabbix-other-checks01 vm and run them there?

Yeh, if it's a lot of work for you then it seems fine to say it won't work. But creating a zabbix instance is a bunch of work too, so if it was easy to add stuff for people then it seems good to do so.

yeah, I'd agree with that...

i don't think we need a vm. this is http-endoint stuff

which, to be fair, we'll be doing for ocp apps too

but they *do* get a role in ansible, so it's less ... intrusive? not quite the right word, but you get my drift

ok, seems like we're ok with it if we can make it tidy? I'll think on it

any other thoughts on this one?

ok, so quickly then

the pdr thing - whats the requirement? people need to open a ticket, and then other people have to discuss it privately?

as in, the originator can't see it? or only those involved can see it?

i'm asking because this is essentially how Discourse do their support - private categories that you can interact with either through discourse, or via an email address

the former won't work, you have to be in the group, and that won't work for pdr. but an email target, from your fas adress?

would that work?

Right now the users create private ticket in pagure repository and then we process it and close it once that is done

the flow i'm thinking of is:

- the creator, plus those with access to the category, can see it

- that creates a topic in a private repo

- send a mail from your fas address to "pdr@whatever.fpo"

i would have to test it, but it might be enough?

Not everybody has fas alias from what I remember

Really?

there might be other discourse workflows too. perhaps a form in one category that posts to another or something

it feels .... almost in grasp

Only people who are in group that is in fedora_contributor metagroup

i don't mean you email from an fpo address, just that you mail from the email account in your profile

anyway, we are at the hour. maybe i should post the idea to the ticket?

Sure

That could work, but we want them to login through FAS to verify them, otherwise they could just sent e-mail to admin@fedoraproject.org

hmm true

Let me close the meeting than

maybe an in-discourse workflow then

i'll poke around

Thanks Gwmngilfen , we can talk this more in other channel