Meeting started at 2025-12-02 16:59:54 UTC

The Meeting name is 'FESCO (2025-12-02)'

The Meeting Name is now fesco

!group members fesco

Members of fesco: Dave Cantrell, Fabio Valentini, Fabio Alessandro Locati, Tomáš Hrčka, Jef Spaleta, Kevin Fenzi, ngompa (@conan_kudo:matrix.org, @ngompa:fedora.im, @pharaoh_atem:opensuse.org, @ngompa:kde.org, @ngompa:almalinux.im), salimma (@michel-slm:matrix.org, @salimma:fedora.im), Stephen Gallagher, Zbigniew Jędrzejewski-Szmek

!hi

Fabio Alessandro Locati (fale) - he / him / his

!hi

!hi

Tomáš Hrčka (humaton) - he / him / his

Zbigniew Jędrzejewski-Szmek (zbyszek)

morning

!hi

Kevin Fenzi (kevin) - he / him / his

Still 1 missing for quorum

!hi

Michel Lind (salimma) - he / him / his

sorry, was in a private chat

We have quorum 🙂. Let's wait a couple of minutes to see if other will arrive otherwise we start

!hi

Fabio Valentini (decathorpe) - he / him / his

!hi

Neal Gompa (ngompa) - he / him / his

!fesco 3512

● **Opened:** a week ago by catanzaro

**fesco #3512** (https://pagure.io/fesco/issue/3512):**Let's do something about ignored CVE bugs**

● **Last Updated:** 5 hours ago

● **Assignee:** Not Assigned

Let the fun begin!

I think this might benifit from a wider discussion... but I probibly always say that

We discussed about this last week and would make sense to see if we want to weight in or wait for the conversation to continue in the discussion forum

was there a discussion thread on it? I thought it was just in the ticket

I missed the follow up (was dealing with people at home taking turns getting sick) - was a discussion started?

iirc there's a follow-up action to start such a discussion

I closed the firefox bugs.

Can we do the same for the kernel?

the wider public discussion just started

as for the kernel, I thought jforbes (traveling) usually closes them after shipping an update

if someone has it, adding the link here would be useful for the meeting logs

I don't think we should. We could ask kernel maintainers about it I guess...

also fyi, the dashboard we discussed last week is now offline pending review, but the source is out

oh I mixed up threads

so it's looking much better

the thread pushed was about elections not this

Yep, and the council ticket has been created.

I think we need to get a handle on fixing the input data before we try and clean up things, or else it will just go back to the same state after a while

we probably need to make a thread about this, but I'd like ProdSec engaged first

Unfortunately we have no reply from the council or FPL yet.

just fyi, most of the us was out last week for thanksgiving holidays.

so that about cover this topic, right? we're still in a holding pattern

yep

(and for the kernel issue specifically Justin will be sporadic until after LPC)

yeah, could make sense to give one week additional time, given last week was a slow one

do we want to discuss it here first before starting the wider discussion? e.g. once we hear from prodsec/council

I forgot who agreed to take point on this last week

Here or in the ticket.

I think we should probably continue in the ticket

simply because it'll be easier to get other stakeholders up to speed

though Neal seems to be the one who is talking to prodsec :)

yeah, ticket might be better

look man, I don't know how this happens 😅

Neal wrote to the site owners, I wrote to the FPL.

no good deeds go unpunished ;)

!fesco 3507

● **Opened:** a week ago by bookwar

● **Assignee:** Not Assigned

● **Last Updated:** 7 hours ago

**fesco #3507** (https://pagure.io/fesco/issue/3507):**FESCo election policy which prioritizes first timers**

this discussion was the thread I was referencing earlier

There has been a lot of discussion on this one...

it's still ongoing

yeah, I have been reading it and mulling over... there's a number of ideas in there.

I think we should probably continue to engage and see how that goes

+1

+1

+1

ack

+1 for continuing to engage, right?

+1

yup

!fesco 3506

**fesco #3506** (https://pagure.io/fesco/issue/3506):**Change: BtrfsBootForCloud**

● **Opened:** a week ago by alking

● **Assignee:** ngompa

● **Last Updated:** 5 days ago

ping davdunc

I think we were waiting for David to provide additional use cases

heh we have a non-fesco vote in the ticket

David is at re:Invent this week

so I don't know if he will be able to make it

punt till next week?

postpone if he does not show up?

Yeah, makes sense to postpone it to next week

let's wait a minute or two and see if he shows up

but otherwise yeah

I think that even if David shows up, it would be better if the usecases are integrated in the Change proposal, more than explained in a FESCo meeting

good point

if he shows up it would be useful to run them by us first for some vetting, but yeah this can be done async and that's easier for David too

!fesco 3513

**fesco #3513** (https://pagure.io/fesco/issue/3513):**Change: UseKmsconVTConsole**

● **Assignee:** jfalempe

● **Last Updated:** 7 hours ago

● **Opened:** a week ago by alking

I see that Michel Lind UTC had concerned on this one. Do we want to discuss it or are we all comfortable with direct voting?

one thing that I was confused about here: how is this related to the "use foot cage" or whatever discussion that happened a few months ago?

this change makes me very nervous

that's the next step after this

this is just the kmscon basic tier

hmm, ok 🤔

yeah, the next step is going to be the 'fun' one

the fallback "should" scares me a little

but otherwise I am fine with letting it through

nervous because the code is still relatively new and untested?

if it blows up, we can revert it

this one is fine/nice. Cool to be able to increase font size on the fly, scrollup, use whatever fonts, etc.

no, that part doesn't bother me, it's the uncertainty about fallbacks that makes me nervous

I mean, testing would cover fallback mechanism too ;)

Yeah, it seems to work surprisingly well in the tests I did… I think if it turns out to have problems, we can easily revert or postpone.

my reason for worry is I don't know how we're going to have automated testing

because we're going to need it

one sec

but yeah my local testing was fine too

we definitely need strong testing for this one

The uncertain fallback is bit strange, but the change will bring some modernity

in general this looks looks fine, but would probably be great to have this land in rawhide as early as possible

I've had a machine testing both this and the kwin+konsole variant for a while

it's been very cool

so what scares me is if the systemd unit does not fail, but the console is somehow inoperative

I'd like to see documented kernel cmdline parameters to disable this that you can add to grub

I want a way to test that failure mode explicitly in automated testing

but this is something I expect someone to solve as part of implementing this

but I probably would be happy to vote to approve provided that is fixed later, the proposer seems responsive to change requests

Adam will probably yell as soon as OpenQA blows up

same with Neal

I hit a selinux issue, but it was promptly fixed...

otherwise it's been great.

my worry is that it never does :P

unless we induce breakage on purpose

we don't have a virtual chaosmonkey in openqa yet

just only real ones after the fact :P

would it make sense to vote the proposal? I'm referring to the people that have not voted in the ticket already

I think a vote would be good. AFAIK, there are no open questions, just some apprehensions, so waiting is not going to help.

I'm still +1 (but voted in ticket)

+1 too with what I have learned today ;)

oh jfalempe addressed my concerns, so yeah I think the override will be documented

+1 (I'll also add it in the ticket with a short follow up)

I already voted +1 in the ticket

I think we already have the in-ticket votes to pass it

I dont mind voting in ticket or here +1

pagure.io is 503 for me now :(. But I think I voted in the ticket.

pagure.io loads really slowly for me so just count me as +1

Pagure just started to return Service Unavailble errors, but it seems that we are +7,0,-0

oh wow what a time for it to get DDOSed agai

oh wow what a time for it to get DDOSed again

I managed to submit my comment fwiw

s/again/still

!fesco 3515

● **Assignee:** Not Assigned

● **Last Updated:** an hour ago

● **Opened:** a week ago by py0xc3

**fesco #3515** (https://pagure.io/fesco/issue/3515):**Change Wrangler tasks done by AI with partly unpredictable outcome**

This conversation today picked up momentum, so not sure if it makes sense to be discussed in this forum already or wait for the conversation to progress in the ticket

oh fun

yeah I think we should continue in the ticket

I don't think there's much for us to do. In general, the change wrangler needs to write better automation. Currently this is all still so fragile.

just as always, I have not read any comments made in the last day or so...

yes, the lack of actionability on our side is also something I was not too sure about. Might make more sense to be discussed in the Discussion forum rather than FESCo tracker

when in doubt I would have used pandoc, not Claude, for this task 🥲

Adam brought up exactly that and I +1-ed it

in general it does bring to mind that... if you can't do something yourself (vs too busy to do something) you probably shouldn't get AI to do it

Well what we can think/talk about is making the process simpler so it does not require so much automation?

Or the tools...

jednorozec: I wanted to ask about forgejo status in Open Floor…

let's make Change process PR based! with structured data! everything in XML with XSLT templates! well maybe not the last one ...

because the new forge uses wiki in .md format for example...

'cause I don't think it makes sense to discuss bigger changes right now since pagure is going away.

yeah, if we implement a new process needs to be forgejo based

whatever we do, the end result is that we need relatively static URLs where all changes are collected and featured

and they are presented in a pretty form

the ticket finally loaded for me and wall of text sooo I dont have much to add

yay ai

heh, I think we can all agree that using the Wiki for this process is a bit anachronistic :D

well, I don't mind it :)

it looks nice, it presents instantly

you've done it like 254 times so of course you know what to do ;)

not quite _that_ many times

i dont mind wiki but our wiki is complicated to *just publish* a template

but yes

there are rst and md based wiki systems these days too

we could definitely make the process better if we can put time/energy into doing so

I can explore forgejo wiki which is md based and uses git. And create some PoC from currently opened changes

I think we can rediscuss this in the coming weeks after we have more info on possible alternative processes on forgejo

I will take a forge process over more discourse

I'm not sure wiki -> wiki is a good thing, I'd rather move to a PR process ...

but meh

technically you could do it either way if it was git backed

I personally don't like PRs for this process

it's kind of annoying based on my experience with ALESCo

well if its git commit with md file? we will see what I can come up with

any takers?

I can run next week

any topics?

Forgejo status?

does "dinner" count?

new landing page https://forge.fedoraproject.org/

Fabio: I don't think you want to *discuss* what's for dinner.

+1 to dinner. ;)

bunch of team already in https://forge.fedoraproject.org/explore/organizations

where do we file tickets about the forge itself?

ha, the Rust SIG is on the front page! nice.

are we talking about the forge(our deployment) or forgejo?

yes

both

I have specific asks on both levels

https://forge.fedoraproject.org/forge/forge ?

Oh, DEI has a great icon.

"Explore all organizations" on the front page is 404

Well each in respective place. For the deployment https://forge.fedoraproject.org/forge/forge

should probably have s/orgs/organizations/ in the link ...

yeah

somebody messed up the link :)

*"somebody"*

I have no idea who did the landing page, but finally its there not the default 20y old landing page :)

What about private issues?

I think ryan did that. ;) looks nice.

So we have a PoC in our fork but that is it

its PoC

more work, and upstream cooperation ahead

You can check the status of things on the project board https://forge.fedoraproject.org/forge/forge/projects/185

might be a good idea to "lead by example" and move at least the docs repo

should we file an infra ticket about creating a fesco org?

well we are very late with leading by example :) releng is already using the new forge

true. other "docs" repos are already there so looks like that all works as it should :)

sure, we should move when someone has time to do the mirgation (but we should probibly also announce it so people know)

I am not sure the docs build pipelines are in place for them

oh ... well then maybe not *yet* :(

at the very least we need to make sure updating the fesco docs works so we can update the members after the election in a few weeks :D

Hmm, but are not the docs build pipelines just pulling from a bunch of different repos?

yeah

https://gitlab.com/fedora/docs/docs-website/docs-fp-o/-/merge_requests/32 appears to be the first. ;)

that looks simple enough

epel and server also are using it

though that's the "stg" branch

https://gitlab.com/search?group_id=15282537&project_id=38671593&repository_ref=prod&scope=blobs&search=forge

and the forge itself

so what do we do when the forge is down?

I assume we're going to have documented some kind of backup process

what do we do when pagure.io is down?

well that process sucks which is mostly bug you directly :P

well, I mean, thats a pretty general question...

I don't want that to be the process going forward

do you mean down for a short outage, or destroyed by an asteroid or ?

we could bother jednorozec instead :D

and it's a reality we have to endure as AI scrapers keep taking our stuff down

Conan Kudo: we have daily backups of the forge and set of SOPs to follow for sysadmin-main people

not planned short outages

anyhow, forge is fully backed up and could be redeployed in any other openshift cluster(s)

destroyed by an asteroid is a scenario in which we're all doomed anyway

I assume for the DR you have regular restores testing your backups

Conan Kudo: My goal is to have the SoP docs in a shape that any member of sysadmin-main will be able to redeploy it from scratch

and that you can restore _outside_ of OpenShift too

because if your cluster is toast you need a way to bring it up without it

it will take so me time to get there but we are on track

might make sense to have a forge outage fallback on codeberg.org too

for status.fp.o that's on GitHub ... https://github.com/fedora-infra/statusfpo

That would be more work I suspect, but yes, all the data would work wherever

The current site has been redeployed from backups a number of times during the initial setup. ;)

forgejo should be 1 binary + DB, so it shoul dbe spinnable on a non-conatiner platform fairly easily

just saying, we should be able to prove that hypothesis :)

you'd be surprised how hairy things get in an emergency :P

its not 1 binary and DB in prod deployment

there is cache there are proxies there is stuff

yeah, that is also true

its rather complex deployment but its declarative in helm chart

so, OCP/k8s is probably required at least today

https://codeberg.org/fedora/forgejo-helm

yup you need kube cluster, but that is not a problem now days.

...

no comment

Should we publish comunitty blog on somethign like "how to deploy your forge with VM and k8s" ?

at least that yeah

me personally I'll probably use the rpm and do the migrations accordingly

I don't want to be in a position where kubernetes is required for simple dev/prod stuff

That is your choice but your issues with that are your own

Hmm, I don't think it makes sense to monitor details of what fedora infra is doing, either here or in some public discussion forums.

the helm is used by us so its maintained and gets tuned

We *know* that forgejo can be deployed in a bunch of different ways, Fedora infra might be using a more complicated one.

yeah, I don't really think we should micromanage how this is run?

I don't want to get into my problems with this here, because we'll be here a long time if we do

but I don't care how it's done in prod for fedora-infra

I mostly care that fedora infra services are easily replicable

this is a huge problem we have right now

Or in other words, I don't think we need jednorozec or others to write a howto about forgejo. There are other sources for that.

well I dont see how we can make it easier than helm install for people to replicate EXACTLY what we run...

then at least write up how your deployment works for people to try

can we park this one and potentially have a more structured conversation id the replication of the services is a wider issue?

sure

I need to go anyway for food

any other open floor conversations? Otherwise I'll close in 2 mins

just FYI: pagure.io outage tomorrow afternoon and a dc move outage (that mostly doesn't affect things people care about) next monday

will ssh host key change?

(just asking because people have been confused for months after that happened for src.fp.o)

nope, I'll keep the existing one

(this is just a move of dc's... it's still rhel8, etc)