Meeting started at 2026-05-20 15:30:09 UTC

The Meeting name is 'fedora_coreos_meeting'

!hi

marmijo: Michael Armijo (marmijo)

!hi

!hi

thilofm: Thilo Fromm (tfm)

jbtrystram: Jean-Baptiste Trystram (jbtrystram) - he / him / his

!hi

Timothée Ravier: Timothée Ravier (siosm) - he / him / his

!hi

rapneset: Rolv Apneseth (rapneset)

! hi

!hi

No Fedora Accounts users have the @peytonrobertson:matrix.org Matrix Account defined

!hi

Yasmin Valim de Souza: Yasmin Valim de Souza (ydesouza)

!hi

dustymabe: Dusty Mabe (dustymabe) - he / him / his

I see that we have one action from the last meeting:

- `Update FCOS Meeting Checklist to the right time info`

I updated it in the checklist :)

acervera has already given cookies to ydesouza during the F44 timeframe

We're sitting in an idle period right now with the F45 schedule. We have just a little over a month before the first change checkpoint

The next checkpoint seems to be next month

!hi

spresti: Steven Presti (spresti)

I know we've had a discussion about one change consideration we'll be submitting. But we should make sure to get anything else submitted before these deadlines

I'll also start pulling the list of change considerations for F45 weekly so we can discuss them in this meeting from now on (if there are not more pressing topics).

Great, if there is nothing else to add here - we can movo onto the fedora coreos tracker issues, of which we have two to discuss today

I think this one is from Yasmin Valim de Souza

yep. she created it last week because we were having frustrations with our upgrade tests

I ended up doing some RCA on what was going on and opened https://github.com/coreos/fedora-coreos-tracker/issues/2149

I originally thought the problem was with the ostree repo being slow (we no longer use it, but when doing upgrade tests from old starting points it does get used)

but it was actually a bug in the skopeo stack (we think) that was causing the pain for us.

Nice work

that being said, it's still worth discussing if we want to change things up or not.

right now we run the upgrade test from as far back as we can. i'm not sure it makes sense to keep doing so.

we can also leverage cloud instances for upgrade tests (in cases where we haven't cleaned up the cloud images (i.e. AMIs)) to help parallelize them

While it's technically really cool to test this, I don't think it's bringing us enough value to go as far back

I'm assuming we dont have information about our user's starting points? We just know general information like how many nodes?

yeah, we don't really have much info on that front :(

Well, those tests are what makes sure that if you had installed a node 5 years ago it would still work today.

The upgrade tests have certainly help us catch some interesting bugs

and even now CI coverage of "upgraded from a machine deployed two years ago" isn't very good

the upgrade test covers "can it upgrade", but not really "does everything function after it upgraded"

Yes, there's definitely a lot of value in making sure these nodes can upgrade. I'm wondering what percentage of users would fall into this category If it's low, we *could* evaluate dropping support for it.

or at least not supporting a starting point so far back.

I dont have a strong preference, I'm just looking at it from multiple sides :)

Maybe we could implement a sliding window of upgrade tests

i.e. 10 fedora releases - or 8 fedora releases (some number N)

that way the upgrade tests don't just continue to increase over time

I like that idea!

so 10 right now would take us back to F34

currently we start at f32

given how easy it is to respin an FCOS node 5 years sounds like a lot

but given how stable it is, 5 years can go fast :)

jbtrystram: you'd be surprised at how much infra just stays in place if it 'just works'

I think we only start at f32 for x86_64. the other arches have later starting points IIRC

That's what I'm wondering. How many of our users fall into this.

marmijo: right. because the other architectures were only added later

my routre right now aleph was `build": "35.20220131.3.0",`

yeah yeah I know, I'm just seeing the two faces of the coin. Heck, Nemric respins every two weeks !

my router right now aleph was `build": "35.20220131.3.0",`

dustymabe: your router ? I want to learn more

so I either want to have a long window of testing here - or we make a statement that explicitly says "if your aleph is older than X, we won't consider your bug"

jbtrystram: haha

dustymabe: i'd say we will consider the bug, but you have to report it

instead of us catching it early

jbtrystram: yeah. I think we have to be ok with just saying reprovision your node

which I'm OK with, but still want to state somewhere some guidance for how old a machine can be

if the initial issue is not to do with this test, what is the reason to not run the test / limit the versions ? Is it because the tests can take ver long in the CI and cause issues as the time grows ?

Angel Cervera Roldan: yeah. it takes a long time to run them and each of them pulls a lot of bits across the network :)

I've thought about periodically saving VMs that started at X,Y,Z and then starting them back up to let them update. but that is extra maintenance too

not sure what the right balance is

Can we maybe create an instance from the first boot image, update it to say the first barrier before we switch to container only and then take a snapshot and use that as a reference?

Timothée Ravier: we can :)

We don't support using old boot images, so what we really should test is old instances and keeping such a snapshot should do it

correct. I've just always felt that was more maintenance work than just running the full upgrade

I did some work last week (when I thought the ostree repo was the bottleneck) to remove the ostree repo from the actual upgrade (i.e. local tarball with ostree repo contents for barrier releases).

We either make an AMI on AWS or create a one off QCOW2 image and upload it somewhere to use it

now that I know what the actual bug was i'm not sure if we should use that or not, but I at least know how to do it

"one off QCOW2" for each starting point

hum, what do you mean each starting point?

32 had a different bootloader than 33, than 34, etc

we update those automatically now don't we?

well on x86_64 & aarch64 at least

now we do, yes.

so maybe the value is minimized.. but I figure there's still some value

i.e. xfs filesystem created from 5 years ago, etc.

I think there's value in having tests from different starting points

that's why the upgrade tests are heavyweight for us right now. we run a test for 32, 33, 34, 35,.... for each prod release

Then let's make 10 QCOW2 / AMI and use those?

trade storage for faster testing?

Timothée Ravier: I'm happy with that if someone wants to take it on.

(I know easy to say and someone has to do it)

1. how to do it every time a new release comes out

2. how to have kola start with a VM that was already booted (ignition already ran)

we'd have to figure out

We are running sort of tight for time, there is one more issue to discuss. Should we move onto the next topic, and come back to this in the open floor if time allows it ?

happy to move on to the next topic.. can we capture some of this discussion in the ticket?

Sure - I'll add the context to the github issue after the meeting.

I think this one is from dustymabe

yep. basically chunkah is a new tool for chunking up images smarter to try to encourage layer re-use.

i'm interested to start using it to shake out any issues (maybe in `next` stream first` and also increase our number of layers (to get even more layer re-use))

anyone have any strong opinions on this?

or even opinions that aren't strong :)(

+1

I've been using chunkah for my toolbox container images and the sealed images without issues so far

jbtrystram: if we revive the `next-devel` for chunkah it might be good to also roll out the bootc install to filesystem change there too (and maybe even konflux, cc jcapitao )

yeah !

I am all for testing out things

this isn't a user-impacting change either so we should roll it out IMHO

anyone with opinions on the number of layers? I was going to make the number large (basically one layer per srpm), but interested to see if people want to challenge that

Can we reduce those later on ? I want to keep some space for when we support derivation

I'm not sure to well understand everything, but I'd like to remove some unsued GPU firmware from the image ... would that help me in any way ?

jbtrystram: I haven't hit a limit yet because fcos doesn't have enough SRPMS to really bump up into it.

jbtrystram: if you were set max layers to like 600 and then try to chunk a silverblue image you'd probably hit whatever the limit is

Nemric: no :/ i think you want to do a full rebuild for that

but yes, we'd want some number of layers still available for people to do derivations

I thought the limit was 256 because of the kernel overlayfs ?

jbtrystram: IIUC it's more like 500 now

I've done tests with a FCOS with 355 layers - so at least that works

What number did you have in mind? We can start with 120 or so

Desktop sealed images are currently 128 layers (+ config + UKI so 130): https://github.com/travier/fedora-atomic-desktops-sealed/blob/main/Containerfile#L49

Timothée Ravier: :) - max-layers = 400 - > right now yields an FCOS with 355 layers (or maybe that was RHCOS and FCOS was less)

but yeah. i want to test the limit here (we can start in rawhide)

There are diminishing returns with more layers, not sure it's worth it.

I think the number of srpms we have is small enough that it's OK for us

Timothée Ravier: that's true, maybe we adjust it back over time, I just kind of want to start at the max and then pull it back from there

if no one is opposed to that

let's give it a try

!proposed we will try to use chunkah for chunking up our container layers and experiment rolling out a relatively high limit for the number of layers. we'll start in rawhide and next to see if there are any issues.

It's easier to try once we have images up

Maybe we can do 5 different number for 5 consequtive daily builds and compare / do an update between those / between a base

Timothée Ravier: are you interested in layer re-use for those 5?

+1 to proposed

+1 to the proposed

+1 too

+1

I would say mostly how long it takes rpm-ostree to go from current to those

Ill give it a few more seconds and agree to the proposal if no one votes against.

as they will likely not have much differing, but we can look at that as well

or we rechunk existing images to try

(we are at time, sorry)

We are starting to go overtime, unfortunately we wont have time for the open floor this week.

Thank you very much everyone for attending,