Meeting started at 2026-02-19 17:00:04 UTC

The Meeting name is 'Infrastructure (2026-02-19)'

!chair @nirik:matrix.scrye.com @zlopez:fedora.im @jnsamyak:matrix.org @james:fedora.im @gwmngilfen:fedora.im @patrikp:matrix.org

The Meeting Name is now infrastructure

morning

Hello, welcome.

!hi

Vít Smolík (smoliicek) - he / him / his

!hi

Greg Sutcliffe (gwmngilfen) - he / him / his

Anybody new that would like to make an introduction?

Seems not.

Would anybody like to host on March 5th?

I can take it.

!hi

Michal Konecny (zlopez)

Any announcements?

Tomorrow is a recharge day.

cool!

Got it running 30 minutes ago 🙂

I will probably look at the toddlers next

I'm writing lots of Zabbix docs (finally 😛)

So just a reminder: we have our next sprint starting next week/sprint planning... so if everyone could set anything they plan to work on then in the next sprint before heading out today that would be good.

I will do it on Monday morning, it's evening my time anyway

sure, that works too. ;)

I would like to help with the RHEL10 adoption. Would somebody be willing to break a chunk off to a separate ticket and help me scope it and give a few pointers, please?

https://forge.fedoraproject.org/infra/tickets/issues/12712

Nice, I see Kevin gave some more details.

patrikp: I started looking at that. I thought of some I can ask your help with testing.

Excellent news!

I can try and mark some specific ones and explain testing on them today if I can.

If you get the chance it'd be much appreciated. Anything else to announce?

note that since we are in freeze we can't do much in prod, but we can in stg of course.

I did the update for postfix for lmdb, the PR is ready

yes, I saw, will try and review.

I already did a PoC on mailman01.stg and it seems to work

if no one objects I plan to crank though the open monitoring tickets - I figure net-new checks are probably safe to deploy even in freeze?

ill work on the commit emails with public inbox

https://forge.fedoraproject.org/infra/tickets/issues/11641

Cool. Moving on to things to discuss...

Anybody for the week starting on Feb 27th?

I can take it?

Vít Smolík: What is your fas username? So I can setup you for oncall

smoliicek

!oncall

● @smoliicek:fedora.im (smoliicek) Current Time for them: 18:17 (Europe/Prague)

If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)

The following people are oncall:

OK, set

Did anything of note happen during last week's oncall?

I got few direct pings, but didn't saw any oncall ping whole week

I got few direct pings, but didn't saw any oncall pings whole week

I assume the freeze is to blame here

Cool. Next section?

Go for it

not much in nagios now...

a fmn queue and 2 down hosts that we know about

same for zabbix. quite a lot of space warnings that should probably be handled better, but otherwise pretty quiet

for the queue... it's the unretire packages one... but this is not live yet, so perhaps we should remove that queue for now?

In zabbix there are the dummy alerts for ssl certificate expiration

yeah, i need to understand why that one site triggers false positives when none of the others do

those bastion mail queue ones are weird too...

The storinator one was correct

yeah

so there's two checks on mail, the original "it's higher than X" which is boring, and a "it's been increasing for X checks" which seemed more useful when I wrote it...

but bastion is the mail gateway. it often has a large queue.

-- 47962 Kbytes in 9704 Requests.

looks like a lot of protonmail issues right now.

i can take a look - the corrent config for that check is "more than $WARN" *and* "has been steadily increasing"

i can take a look - the current config for that check is "more than $WARN" _and_ "has been steadily increasing"

which seems about right, but perhaps $WARN should be higher (it's currently 10k)

(delivery temporarily suspended: connect to mailsec.protonmail.ch[185.70.42.129]:25: Connection timed out), etc

yeah, might need tweaking

i also see a lot of load warnings for wiki01/02

>100 each in the last

>100 each in the last 7 days

\> 100 each in the last 7 days

yeah, I saw that... but haven't looked into it yet.

possibly scrapers? (wiki is not behind anubis)

ah, could be

i'm liking having these reports for less-urgent things that we can look into if we have chance, rather than getting message-bombed in chat

nothing more monitoring wise from me that I can think of.

btw I *thhink* I have a PoC for service-level stuff that could handle, eg, when a HAProxy goes down and we get 20 alerts....

need to test the Ansible code for it though

cool

Backlog refinement?

well, we could, or I can do a thing on our anubis setup... or both I guess if we think we have time

How long is your presentation? We can go for it first and if there's time left over do backlog refinement?

I suppose this command should be called with the name of the topic also?

well, I didn't pre-pare anything... :) So not sure.

Anyhow, a short guide to anubis.

anubis is a proxy application that 'weighs' the worth of connections, with a desire to block or drop connections from undesired clients.

https://github.com/TecharoHQ/anubis

it's written in go

The way it works is you proxy web connections to the application, it then uses a policy you set to decide what to do with them, then if they pass it proxies them back to your application.

The default policy has a bunch of things in it already, but you can use that and build on it to allow things you need to allow or weigh things differently.

The policies are pretty flexable.

Moving to our infra, we have actually 2 different anubis roles.

One (anubis-el) is container based. It is used on our rhel hosts, because there's not a epel version of the package for various reasons.

The second (anubis) is used on fedora hosts. It's using the fedora package of anubis.

anubis-el is used on pagure.io and dl*.fedoraproject.org currently and anubis is used on the proxies.

for pagure.io and dl machines, the proxing to/from it is just setup in the regaulr http config for those hosts.

for proxies, we have a subset of sites behind it. If a playbooks/includes/proxies-websites.yml website entry has 'anubis: true' the templates do all the work and set it up for that site.

Normally, we don't need to add legit users to anything, using Accept: and User-Agent: headers are usually enough to get leig requests through

In some rare cases we do have to add things and thats in the policy files. (either by user-agent header or something else distinctive)

zabbix is monitoring anubis, so you can see a lot of data about it there.

any questions or things I should expand on?

Desperate times, desperate measures...

yeah, it's not great, but without it we would just be offline probibly.

Good overview, thanks.

If there are no questions, backlog refinement time?

as a side note, I should probibly make sure we have a doc for it. ;)

sure, one or two?

Last comment from 6 months ago asking if it's still a problem. Shall I ping again?

yep. this is still happening. Someone needs to investigate more.

I think it may be a harmless thing in teardown of the fedora-messaging thing, but I don't know that for sure.

yeah, please update asking for status/if someone has time to look into it

also still an issue. Someone needs to figure out what flatpaks we can prune.

I guess this may be made moot by switching to quay.io (another ongoing ticket), but we still might need a solution for there too.

(feel free to add that image to the issue too. ;)

One approach could be to get all of them that exist and then exclude any that are ever tagged in stable releases or something.

I don't think I have anything else... or at least not that comes to mind right now. ;)

also nothing here

Coolio.