Meeting started at 2024-01-03 16:30:05 UTC

The Meeting name is 'fedora_coreos_meeting'

.hi dustymabe

.hello c4rt0

.hello jdoss

!chair apiaseck jdoss

.hello jlebon

!chair Jonathan Lebon

.hello jbtrystram

!chair jbtrystram

welcome all - we'll get started soon. /me pings travier to see if he is around too

also - happy new year to all - we made it

\o/

.hello aaradhak

Oh - that's a good idea

jbtrystram: want to intro this one?

Sure :)

Some kola tests depend on external container images. so far there were only one : a `tang` image used for the LUKS tests. With the iSCSI boot stuff I had to add another one : a `target` service to serve an iscsi target.

now, the images are owned by the authors of the tests, which are fine, as long as they are around

I propose to create a new repository under github.com/coreos to host, build and publish those images

I had resisted this model (i.e. individual container images for tests) in the past because it's hard to manage/maintain. I didn't realize we had the tang one slip through

I created a POC repo here : https://github.com/jbtrystram/kola-container-images

In the past I've recommended we just build the container image on the host and then use it

but I agree it can be nicer to have a container image already in a registry somewhere

yeah, if the test is relying on internet access anyway, it seems more efficient to use ready made images

it could be paired with a dedicated org in quay

this makes sense to me overall hmm, i don't think it's worth another quay org to manage. we could put it under the existing coreos-assembler one

Jonathan Lebon: i forgot about this one, the coreos-assembler org is a great fit for this

I think part of the reason I wanted to not do this in the past was we didn't have a good way to build multi-arch container images, but we do have a mechanism for that now (we do it for cosa and for the fcos-buildroot containers)

so we'd need to have new jenkins jobs for the new containers OR we'd need to parameterize the job somehow

dustymabe: I reused coreos/actions-lib/build-container@main

jbtrystram: that's not going to work for multi-arch

an alternative btw when evaluating these is to just ship the tools necessary as part of cosa so we can reuse the cosa image itself. i think for the iSCSI stuff it'd probably add too much, but it might be a good fit for the tang one

we need something like https://github.com/coreos/fedora-coreos-pipeline/blob/main/jobs/build-fcos-buildroot.Jenkinsfile

Jonathan Lebon: true.

dustymabe: maybe i am understanding this wrong (this was my first try on multiarch images) but this seems to work fine : https://quay.io/repository/jbtrystram/tang?tab=tags&tag=latest

COSA is very much a kitchen sink today

dustymabe: yeah, probably time to parametrize that job

jbtrystram: how are those images built?

https://github.com/jbtrystram/kola-container-images/tree/main/.github/workflows

GH actions, using the `coreos/actions-lib/build-container` action

using emulation or cross compilation or something? I highly doubt github CI has ppc and s390 builders

it uses qemu-user-static

https://github.com/coreos/actions-lib/blob/6f48d6e84f52dd5d3da0b89510003f21f7313016/build-container/action.yml#L93C33-L93C49

so emulation

interesting.. should we rely on that?

interestingly, the build times don't look terrible: https://github.com/jbtrystram/kola-container-images/actions

honestly I feel a bit uneasy about it

i'm not sure. maybe? i'd say i'd probably want all our multi-arch images to be built the same way. so if we keep build-cosa, i'd rather do it the same way for those too

i.e. this is now load bearing infra and who has insight into the failures.. who maintains it over time?

to play devil's advocate, we could probably figure out build failure notifications. but obviously, that's work vs what we already have set up in the pipeline

i am happy to make that work with Jenkins to have everything aligned. As said, that's just a POC repo to start the discussion

+1 - ok - so what is the open question we're trying to solve here? IF we should do it, or WHERE we should do it?

can we agree the quay.io/coreos-assembler ORG is probably the right place to push to?

IF first, then let's vote the WHERE/HOW condition if needed

+1 from me

+1

+1

anyone opposed to `IF` ?

+1

that was what I was voting on :)

haven't heard any detractors so far

ok so WHERE seems close to agreed on too.. now HOW

one suggestion is to not make this a separate repo but just a folder in the cosa one

I'd argue to keep it consistent and just another jenkins job that uses our multi-arch builders

Jonathan Lebon: define repo.. git repo or quay registry repo ?

that also allows atomically adding tests and supporting Dockerfile. git repo

that also allows atomically adding/modifying tests and supporting Dockerfile. git repo

yeah, maybe that's a good idea, though if the repo is forked from another it might make it harder to keep track/and/or pull in changes

maybe that's TBD pending investigation

not sure i follow

dustymabe: the containerfiles are very simple, and not forked from other repos so I don't think that's an issue. I like jonathan's solution

https://github.com/jbtrystram/targetcli-containers says : `forked from alicefr/targetcli-containers`

anyway let's take this tangent to the FCOS matrix channel after the meeting - i'll add a proposed here now:

!proposed we will build manifest listed container images for specific tests using Jenkins jobs and our multi-arch builders (like we build cosa and fcos-buildroot today) and store them in the quay.io/coreos-assembler org in quay.

gotcha, i follow now yeah, sounds good 👍️

votes?

!undo

anyone else want to vote on the proposed?

should the proposal mentions where we store the containerfiles ?

jbtrystram: I think that's TBD - use your best judgement :)

we can discuss further in the FCOS channel after the meeting

ok

so unfortunately we had to cancel our last set of releases for 2023 because we encountered some upgrade test failures on aarch64

it appears the autopruning code may have some corner case that isn't handled

there was a PR added to OSTree to enhance logging in this area: https://github.com/ostreedev/ostree/pull/3123

planning to look at this today are releases blocked on this currently?

Jonathan Lebon: yes :(

it was hard to progress without understanding the problem more completely

ouch ok

we didn't want to ship out further updates that could exacerbate the problem

it's good that we can reproduce it manually at least

and the week the problem showed itself I was technically away and not able to dedicate appropriate eyes on the investigation

yes, it's rather easy to reproduce - let me know if you need help

the currently built `next` in the compose repo should trigger the problem

the currently built `next` in the compose OSTree repo should trigger the problem

or `testing` too I think

ack, thanks

or maybe not. I can't remember - I think the upgrade tests that were failing were next (remember we switched over `next` to F39 sooner than `testing` so it takes a different upgrade path

or maybe not. I can't remember - I think the upgrade tests that were failing were next (remember we switched over `next` to F39 sooner than `testing` so it takes a different upgrade path).

yeah, grab a debug pod on the aarch64 builder and see if you can repro.. if you can't I'll jump in to help. Thanks so much Jonathan Lebon

yeah, this might just need a slight tweak in the autoprune code. it's a bit sad that it has become load-bearing now on some arches

Jonathan Lebon: at least we know it does work when tests pass, though :)

I would like to accelerate us changing our /boot partition strategy at some point in the future, but $priorities just make it hard

ok there are a few new changes for us to consider

should be transparent to us

seems transparent to us as failures would happen during rpm builds?

well, to FCOS at least. yeah

i think we can safely skip this one

i think we can safely skip this one; we don't ship that package

Jonathan Lebon: some text for the `!info` ?

we don't ship 389-ds-base

Do we know if there are any implications for `rpm-ostree` here and how it handles yum repos?

rpm-ostree already doesn't download filelists by default IIRC

Jonathan Lebon: i.e. you can't just `rpm-ostree install /usr/bin/foo` ?

you can. `/usr/bin` is in the primary metadata

i'll have to double-check. i think we do know to download it if you request e.g. /usr/share/...

but that seems independent of this change since i think the change is happening at the dnf layer and not libdnf

ok but in general you think we shouldn't have any issues because of this? or should we assign an action item/open an issue?

i think so, but i think it's probably worth verifying current rpm-ostree client-side behaviour wrt filelists

```core@apu2:~$ ls /var/cache/rpm-ostree/repomd/fedora-39-x86_64/repodata/ 31c6030540cfa955dd235257a517808be8d6932cade44a1c0f24793362d5d11b-comps-Everything.x86_64.xml.zck 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml.zck bdd852babb8d6f012c9522ad194b37fa7df83d3ad1a9f4b2c2140e19528b9c96-primary.xml.zck repomd.xml```

Like this ````core@apu2:~$ 31c6030540cfa955dd235257a517808be8d6932cade44a1c0f24793362d5d11b-comps-Everything.x86_64.xml.zck 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml.zck bdd852babb8d6f012c9522ad194b37fa7df83d3ad1a9f4b2c2140e19528b9c96-primary.xml.zck repomd.xml``` ````

Like this ``` core@apu2:~$ 31c6030540cfa955dd235257a517808be8d6932cade44a1c0f24793362d5d11b-comps-Everything.x86_64.xml.zck 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml.zck bdd852babb8d6f012c9522ad194b37fa7df83d3ad1a9f4b2c2140e19528b9c96-primary.xml.zck repomd.xml ```

Like this ``` core@apu2:~$ ls /var/cache/rpm-ostree/repomd/fedora-39-x86_64/repodata/ 31c6030540cfa955dd235257a517808be8d6932cade44a1c0f24793362d5d11b-comps-Everything.x86_64.xml.zck 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml.zck bdd852babb8d6f012c9522ad194b37fa7df83d3ad1a9f4b2c2140e19528b9c96-primary.xml.zck repomd.xml ```

hmm, confusingly it looks like some paths do skip it and some don't

that might just be an oversight

ok let's open an issue to investigate further..

i'd peel that off as a separate rpm-ostree issue

yeah

^^ agree?

👍️

anyone with anything for open floor?

jdoss: Adam B ?

I got nothing this week. Happy new year folks. Lets make FCOS even more awesome in 2024!

it's worth noting I ran into this myself this past week: https://github.com/coreos/fedora-coreos-tracker/issues/1641

i've been trying to find and fix bugs associated with adding additional files to an FCOS build

I guess I am still fighting ipxe booting RPi CM 4s with FCOS but that is a swamp of pain and suffering right now.

would love to know the right way to navigate that sort of thing, since it feels like i'm swimming against the current with everything i try

I just use layering for everything now.

yeah, i want it baked in for the live image and to work offline, so hard mode x2

I like your speed Adam. 🙂

Adam B: yeah. I'm not going to lie, `cosa` is very much built for building coreos and deviating from that too much might yield some pain

the rootfs squash image inside the initrd is currently the blocker for me, since cpio has a 4GB max for file entries

"some" :)

honestly cosa has been pretty solid

Adam B: one option may be to write the ISO to partition one of a disk image (i.e. USB key) and write other files to a filesystem on partition 2

then after the install pick those files up and copy them into the right places

yes, i've realized this might be my only option...

still trying to figure out how this impacts updates

and also how to safely modify the disk image without breaking its bootability

you can also do https://matrix.to/#/!YWqcsiUQiCaqimYdQT:fedoraproject.org/$h-bXNIpf0JvGsss02bHdwjpjK4Mb-vPWGYBPajyIQ1c?via=fedoraproject.org&via=fedora.im&via=matrix.org but just write a the raw disk image into partition 2 and install that instead of what is baked in the ISO

since it sounds like you are building the raw disk metal image without issue

yes, raw disk image builds alright

slowly, but alright

Adam B: let's keep chatting over in the FCOS channel after the meeting

anyone with anything else for open floor? if no I'll close out the meeting soon