2024-01-03 16:30:05 <@dustymabe:matrix.org> !startmeeting fedora_coreos_meeting 2024-01-03 16:30:07 <@meetbot:fedora.im> Meeting started at 2024-01-03 16:30:05 UTC 2024-01-03 16:30:07 <@meetbot:fedora.im> The Meeting name is 'fedora_coreos_meeting' 2024-01-03 16:30:16 <@dustymabe:matrix.org> !topic roll call 2024-01-03 16:30:25 <@dustymabe:matrix.org> .hi dustymabe 2024-01-03 16:31:18 <@apiaseck:matrix.org> .hello c4rt0 2024-01-03 16:31:29 <@jdoss:beeper.com> .hello jdoss 2024-01-03 16:31:44 <@dustymabe:matrix.org> !chair apiaseck jdoss 2024-01-03 16:31:59 <@jlebon:fedora.im> .hello jlebon 2024-01-03 16:32:21 <@dustymabe:matrix.org> !chair Jonathan Lebon 2024-01-03 16:33:57 <@jbtrystram:matrix.org> .hello jbtrystram 2024-01-03 16:34:10 <@dustymabe:matrix.org> !chair jbtrystram 2024-01-03 16:34:34 <@dustymabe:matrix.org> welcome all - we'll get started soon. /me pings travier to see if he is around too 2024-01-03 16:34:58 <@dustymabe:matrix.org> also - happy new year to all - we made it 2024-01-03 16:35:44 <@dustymabe:matrix.org> !topic Action items from last meeting 2024-01-03 16:35:47 <@jdoss:beeper.com> \o/ 2024-01-03 16:35:56 <@dustymabe:matrix.org> !info there are no action items from the last meeting. 2024-01-03 16:36:23 <@dustymabe:matrix.org> !topic Create a new repo to host Container images needed by Kola tests 2024-01-03 16:36:28 <@dustymabe:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1639 2024-01-03 16:37:21 <@aaradhak:matrix.org> .hello aaradhak 2024-01-03 16:37:40 <@apiaseck:matrix.org> Oh - that's a good idea 2024-01-03 16:37:42 <@dustymabe:matrix.org> jbtrystram: want to intro this one? 2024-01-03 16:37:57 <@jbtrystram:matrix.org> Sure :) 2024-01-03 16:39:12 <@jbtrystram:matrix.org> Some kola tests depend on external container images. so far there were only one : a `tang` image used for the LUKS tests. With the iSCSI boot stuff I had to add another one : a `target` service to serve an iscsi target. 2024-01-03 16:39:45 <@jbtrystram:matrix.org> now, the images are owned by the authors of the tests, which are fine, as long as they are around 2024-01-03 16:40:10 <@jbtrystram:matrix.org> I propose to create a new repository under github.com/coreos to host, build and publish those images 2024-01-03 16:40:43 <@dustymabe:matrix.org> I had resisted this model (i.e. individual container images for tests) in the past because it's hard to manage/maintain. I didn't realize we had the tang one slip through 2024-01-03 16:40:48 <@jbtrystram:matrix.org> I created a POC repo here : https://github.com/jbtrystram/kola-container-images 2024-01-03 16:40:55 <@dustymabe:matrix.org> In the past I've recommended we just build the container image on the host and then use it 2024-01-03 16:41:20 <@dustymabe:matrix.org> but I agree it can be nicer to have a container image already in a registry somewhere 2024-01-03 16:42:23 <@jlebon:fedora.im> yeah, if the test is relying on internet access anyway, it seems more efficient to use ready made images 2024-01-03 16:43:01 <@jbtrystram:matrix.org> it could be paired with a dedicated org in quay 2024-01-03 16:43:45 <@jlebon:fedora.im> this makes sense to me overall hmm, i don't think it's worth another quay org to manage. we could put it under the existing coreos-assembler one 2024-01-03 16:44:29 <@jbtrystram:matrix.org> Jonathan Lebon: i forgot about this one, the coreos-assembler org is a great fit for this 2024-01-03 16:44:56 <@dustymabe:matrix.org> I think part of the reason I wanted to not do this in the past was we didn't have a good way to build multi-arch container images, but we do have a mechanism for that now (we do it for cosa and for the fcos-buildroot containers) 2024-01-03 16:45:44 <@dustymabe:matrix.org> so we'd need to have new jenkins jobs for the new containers OR we'd need to parameterize the job somehow 2024-01-03 16:45:56 <@jbtrystram:matrix.org> dustymabe: I reused coreos/actions-lib/build-container@main 2024-01-03 16:46:22 <@dustymabe:matrix.org> jbtrystram: that's not going to work for multi-arch 2024-01-03 16:47:07 <@jlebon:fedora.im> an alternative btw when evaluating these is to just ship the tools necessary as part of cosa so we can reuse the cosa image itself. i think for the iSCSI stuff it'd probably add too much, but it might be a good fit for the tang one 2024-01-03 16:47:13 <@dustymabe:matrix.org> we need something like https://github.com/coreos/fedora-coreos-pipeline/blob/main/jobs/build-fcos-buildroot.Jenkinsfile 2024-01-03 16:47:32 <@dustymabe:matrix.org> Jonathan Lebon: true. 2024-01-03 16:48:02 <@jbtrystram:matrix.org> dustymabe: maybe i am understanding this wrong (this was my first try on multiarch images) but this seems to work fine : https://quay.io/repository/jbtrystram/tang?tab=tags&tag=latest 2024-01-03 16:48:02 <@dustymabe:matrix.org> COSA is very much a kitchen sink today 2024-01-03 16:48:12 <@jlebon:fedora.im> dustymabe: yeah, probably time to parametrize that job 2024-01-03 16:48:30 <@dustymabe:matrix.org> jbtrystram: how are those images built? 2024-01-03 16:49:06 <@jbtrystram:matrix.org> https://github.com/jbtrystram/kola-container-images/tree/main/.github/workflows 2024-01-03 16:49:36 <@jbtrystram:matrix.org> GH actions, using the `coreos/actions-lib/build-container` action 2024-01-03 16:49:48 <@dustymabe:matrix.org> using emulation or cross compilation or something? I highly doubt github CI has ppc and s390 builders 2024-01-03 16:49:51 <@jlebon:fedora.im> it uses qemu-user-static 2024-01-03 16:50:07 <@jlebon:fedora.im> https://github.com/coreos/actions-lib/blob/6f48d6e84f52dd5d3da0b89510003f21f7313016/build-container/action.yml#L93C33-L93C49 2024-01-03 16:50:20 <@jlebon:fedora.im> so emulation 2024-01-03 16:50:42 <@dustymabe:matrix.org> interesting.. should we rely on that? 2024-01-03 16:51:25 <@jlebon:fedora.im> interestingly, the build times don't look terrible: https://github.com/jbtrystram/kola-container-images/actions 2024-01-03 16:52:25 <@dustymabe:matrix.org> honestly I feel a bit uneasy about it 2024-01-03 16:52:47 <@jlebon:fedora.im> i'm not sure. maybe? i'd say i'd probably want all our multi-arch images to be built the same way. so if we keep build-cosa, i'd rather do it the same way for those too 2024-01-03 16:52:58 <@dustymabe:matrix.org> i.e. this is now load bearing infra and who has insight into the failures.. who maintains it over time? 2024-01-03 16:53:49 <@jlebon:fedora.im> to play devil's advocate, we could probably figure out build failure notifications. but obviously, that's work vs what we already have set up in the pipeline 2024-01-03 16:54:39 <@jbtrystram:matrix.org> i am happy to make that work with Jenkins to have everything aligned. As said, that's just a POC repo to start the discussion 2024-01-03 16:55:07 <@dustymabe:matrix.org> +1 - ok - so what is the open question we're trying to solve here? IF we should do it, or WHERE we should do it? 2024-01-03 16:55:25 <@dustymabe:matrix.org> can we agree the quay.io/coreos-assembler ORG is probably the right place to push to? 2024-01-03 16:55:51 <@jbtrystram:matrix.org> IF first, then let's vote the WHERE/HOW condition if needed 2024-01-03 16:56:17 <@jlebon:fedora.im> +1 from me 2024-01-03 16:56:27 <@dustymabe:matrix.org> +1 2024-01-03 16:56:31 <@apiaseck:matrix.org> +1 2024-01-03 16:56:36 <@dustymabe:matrix.org> anyone opposed to `IF` ? 2024-01-03 16:56:41 <@jbtrystram:matrix.org> +1 2024-01-03 16:56:57 <@jlebon:fedora.im> that was what I was voting on :) 2024-01-03 16:56:59 <@dustymabe:matrix.org> haven't heard any detractors so far 2024-01-03 16:57:28 <@dustymabe:matrix.org> ok so WHERE seems close to agreed on too.. now HOW 2024-01-03 16:57:44 <@jlebon:fedora.im> one suggestion is to not make this a separate repo but just a folder in the cosa one 2024-01-03 16:57:50 <@dustymabe:matrix.org> I'd argue to keep it consistent and just another jenkins job that uses our multi-arch builders 2024-01-03 16:58:32 <@dustymabe:matrix.org> Jonathan Lebon: define repo.. git repo or quay registry repo ? 2024-01-03 16:58:42 <@jlebon:fedora.im> that also allows atomically adding tests and supporting Dockerfile. git repo 2024-01-03 16:59:08 <@jlebon:fedora.im> that also allows atomically adding/modifying tests and supporting Dockerfile. git repo 2024-01-03 16:59:31 <@dustymabe:matrix.org> yeah, maybe that's a good idea, though if the repo is forked from another it might make it harder to keep track/and/or pull in changes 2024-01-03 16:59:43 <@dustymabe:matrix.org> maybe that's TBD pending investigation 2024-01-03 17:00:29 <@jlebon:fedora.im> not sure i follow 2024-01-03 17:01:23 <@jbtrystram:matrix.org> dustymabe: the containerfiles are very simple, and not forked from other repos so I don't think that's an issue. I like jonathan's solution 2024-01-03 17:02:28 <@dustymabe:matrix.org> https://github.com/jbtrystram/targetcli-containers says : `forked from alicefr/targetcli-containers` 2024-01-03 17:03:19 <@dustymabe:matrix.org> anyway let's take this tangent to the FCOS matrix channel after the meeting - i'll add a proposed here now: 2024-01-03 17:03:28 <@dustymabe:matrix.org> !proposed we will build manifest listed container images for specific tests using Jenkins jobs and our multi-arch builders (like we build cosa and fcos-buildroot today) and store them in the quay.io/coreos-assembler org in quay. 2024-01-03 17:03:31 <@jlebon:fedora.im> gotcha, i follow now yeah, sounds good 👍️ 2024-01-03 17:03:35 <@dustymabe:matrix.org> votes? 2024-01-03 17:03:37 <@apiaseck:matrix.org> !agreed 2024-01-03 17:04:28 <@dustymabe:matrix.org> !undo 2024-01-03 17:04:54 <@dustymabe:matrix.org> anyone else want to vote on the proposed? 2024-01-03 17:05:50 <@jbtrystram:matrix.org> should the proposal mentions where we store the containerfiles ? 2024-01-03 17:06:08 <@dustymabe:matrix.org> !agreed we will build manifest listed container images for specific tests using Jenkins jobs and our multi-arch builders (like we build cosa and fcos-buildroot today) and store them in the quay.io/coreos-assembler org in quay. 2024-01-03 17:06:23 <@dustymabe:matrix.org> jbtrystram: I think that's TBD - use your best judgement :) 2024-01-03 17:06:42 <@dustymabe:matrix.org> we can discuss further in the FCOS channel after the meeting 2024-01-03 17:07:06 <@jbtrystram:matrix.org> ok 2024-01-03 17:07:09 <@dustymabe:matrix.org> !topic aarch64 failing to upgrade with /boot filesystem full 2024-01-03 17:07:14 <@dustymabe:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1637 2024-01-03 17:07:56 <@dustymabe:matrix.org> so unfortunately we had to cancel our last set of releases for 2023 because we encountered some upgrade test failures on aarch64 2024-01-03 17:08:19 <@dustymabe:matrix.org> it appears the autopruning code may have some corner case that isn't handled 2024-01-03 17:08:49 <@dustymabe:matrix.org> there was a PR added to OSTree to enhance logging in this area: https://github.com/ostreedev/ostree/pull/3123 2024-01-03 17:08:55 <@jlebon:fedora.im> planning to look at this today are releases blocked on this currently? 2024-01-03 17:09:03 <@dustymabe:matrix.org> Jonathan Lebon: yes :( 2024-01-03 17:09:19 <@dustymabe:matrix.org> it was hard to progress without understanding the problem more completely 2024-01-03 17:09:20 <@jlebon:fedora.im> ouch ok 2024-01-03 17:09:40 <@dustymabe:matrix.org> we didn't want to ship out further updates that could exacerbate the problem 2024-01-03 17:10:03 <@jlebon:fedora.im> it's good that we can reproduce it manually at least 2024-01-03 17:10:15 <@dustymabe:matrix.org> and the week the problem showed itself I was technically away and not able to dedicate appropriate eyes on the investigation 2024-01-03 17:10:38 <@dustymabe:matrix.org> yes, it's rather easy to reproduce - let me know if you need help 2024-01-03 17:10:51 <@dustymabe:matrix.org> the currently built `next` in the compose repo should trigger the problem 2024-01-03 17:11:03 <@dustymabe:matrix.org> the currently built `next` in the compose OSTree repo should trigger the problem 2024-01-03 17:11:25 <@dustymabe:matrix.org> or `testing` too I think 2024-01-03 17:11:55 <@jlebon:fedora.im> ack, thanks 2024-01-03 17:12:01 <@dustymabe:matrix.org> or maybe not. I can't remember - I think the upgrade tests that were failing were next (remember we switched over `next` to F39 sooner than `testing` so it takes a different upgrade path 2024-01-03 17:12:11 <@dustymabe:matrix.org> or maybe not. I can't remember - I think the upgrade tests that were failing were next (remember we switched over `next` to F39 sooner than `testing` so it takes a different upgrade path). 2024-01-03 17:12:49 <@dustymabe:matrix.org> yeah, grab a debug pod on the aarch64 builder and see if you can repro.. if you can't I'll jump in to help. Thanks so much Jonathan Lebon 2024-01-03 17:13:01 <@jlebon:fedora.im> yeah, this might just need a slight tweak in the autoprune code. it's a bit sad that it has become load-bearing now on some arches 2024-01-03 17:13:11 <@dustymabe:matrix.org> !info we are debugging the issue in https://github.com/coreos/fedora-coreos-tracker/issues/1637 and will try to get out releases as soon as we understand the problem more fully 2024-01-03 17:13:39 <@dustymabe:matrix.org> Jonathan Lebon: at least we know it does work when tests pass, though :) 2024-01-03 17:14:22 <@dustymabe:matrix.org> I would like to accelerate us changing our /boot partition strategy at some point in the future, but $priorities just make it hard 2024-01-03 17:14:38 <@dustymabe:matrix.org> !topic tracker: Fedora 40 changes considerations 2024-01-03 17:14:44 <@dustymabe:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1626 2024-01-03 17:15:06 <@dustymabe:matrix.org> ok there are a few new changes for us to consider 2024-01-03 17:15:21 <@dustymabe:matrix.org> !info 121. Changes/Linker Error On Security Issues 2024-01-03 17:15:29 <@dustymabe:matrix.org> !link https://fedoraproject.org/wiki/Changes/Linker_Error_On_Security_Issues 2024-01-03 17:16:18 <@jlebon:fedora.im> should be transparent to us 2024-01-03 17:16:30 <@dustymabe:matrix.org> seems transparent to us as failures would happen during rpm builds? 2024-01-03 17:16:37 <@jlebon:fedora.im> well, to FCOS at least. yeah 2024-01-03 17:17:00 <@dustymabe:matrix.org> !info 121. Changes/Linker Error On Security Issues should be transparent to us as failures would happen during rpm builds before reaching FCOS builds. 2024-01-03 17:17:10 <@dustymabe:matrix.org> !info 122 389_Directory_Server_3.0.0 2024-01-03 17:17:16 <@dustymabe:matrix.org> !link https://fedoraproject.org/wiki/Changes/389_Directory_Server_3.0.0 2024-01-03 17:17:43 <@jlebon:fedora.im> i think we can safely skip this one 2024-01-03 17:17:54 <@jlebon:fedora.im> i think we can safely skip this one; we don't ship that package 2024-01-03 17:18:01 <@dustymabe:matrix.org> Jonathan Lebon: some text for the `!info` ? 2024-01-03 17:18:25 <@jlebon:fedora.im> we don't ship 389-ds-base 2024-01-03 17:18:42 <@jlebon:fedora.im> !info we don't ship 389-ds-base 2024-01-03 17:18:50 <@dustymabe:matrix.org> !info 122 389_Directory_Server_3.0.0 - Nothing to do, we don't ship 389-ds-base. 2024-01-03 17:19:00 <@dustymabe:matrix.org> !info 123. DNF: Do not download filelists by default 2024-01-03 17:19:08 <@dustymabe:matrix.org> !link https://fedoraproject.org/wiki/Changes/DNFConditionalFilelists 2024-01-03 17:19:55 <@dustymabe:matrix.org> Do we know if there are any implications for `rpm-ostree` here and how it handles yum repos? 2024-01-03 17:20:23 <@jlebon:fedora.im> rpm-ostree already doesn't download filelists by default IIRC 2024-01-03 17:21:01 <@dustymabe:matrix.org> Jonathan Lebon: i.e. you can't just `rpm-ostree install /usr/bin/foo` ? 2024-01-03 17:21:31 <@jlebon:fedora.im> you can. `/usr/bin` is in the primary metadata 2024-01-03 17:21:59 <@jlebon:fedora.im> i'll have to double-check. i think we do know to download it if you request e.g. /usr/share/... 2024-01-03 17:22:23 <@jlebon:fedora.im> but that seems independent of this change since i think the change is happening at the dnf layer and not libdnf 2024-01-03 17:22:32 <@dustymabe:matrix.org> ok but in general you think we shouldn't have any issues because of this? or should we assign an action item/open an issue? 2024-01-03 17:23:20 <@jlebon:fedora.im> i think so, but i think it's probably worth verifying current rpm-ostree client-side behaviour wrt filelists 2024-01-03 17:24:21 <@dustymabe:matrix.org> ```core@apu2:~$ ls /var/cache/rpm-ostree/repomd/fedora-39-x86_64/repodata/ 31c6030540cfa955dd235257a517808be8d6932cade44a1c0f24793362d5d11b-comps-Everything.x86_64.xml.zck 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml.zck bdd852babb8d6f012c9522ad194b37fa7df83d3ad1a9f4b2c2140e19528b9c96-primary.xml.zck repomd.xml``` 2024-01-03 17:24:38 <@dustymabe:matrix.org> Like this ````core@apu2:~$ 31c6030540cfa955dd235257a517808be8d6932cade44a1c0f24793362d5d11b-comps-Everything.x86_64.xml.zck 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml.zck bdd852babb8d6f012c9522ad194b37fa7df83d3ad1a9f4b2c2140e19528b9c96-primary.xml.zck repomd.xml``` ```` 2024-01-03 17:24:51 <@dustymabe:matrix.org> Like this ``` core@apu2:~$ 31c6030540cfa955dd235257a517808be8d6932cade44a1c0f24793362d5d11b-comps-Everything.x86_64.xml.zck 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml.zck bdd852babb8d6f012c9522ad194b37fa7df83d3ad1a9f4b2c2140e19528b9c96-primary.xml.zck repomd.xml ``` 2024-01-03 17:25:02 <@dustymabe:matrix.org> Like this ``` core@apu2:~$ ls /var/cache/rpm-ostree/repomd/fedora-39-x86_64/repodata/ 31c6030540cfa955dd235257a517808be8d6932cade44a1c0f24793362d5d11b-comps-Everything.x86_64.xml.zck 8e09489af54bbd4ab85470d449f0b0afa4a26fc3eb97c1665c741427bbc8f060-filelists.xml.zck bdd852babb8d6f012c9522ad194b37fa7df83d3ad1a9f4b2c2140e19528b9c96-primary.xml.zck repomd.xml ``` 2024-01-03 17:25:34 <@jlebon:fedora.im> hmm, confusingly it looks like some paths do skip it and some don't 2024-01-03 17:25:40 <@jlebon:fedora.im> that might just be an oversight 2024-01-03 17:26:10 <@dustymabe:matrix.org> ok let's open an issue to investigate further.. 2024-01-03 17:26:16 <@jlebon:fedora.im> i'd peel that off as a separate rpm-ostree issue 2024-01-03 17:26:17 <@jlebon:fedora.im> yeah 2024-01-03 17:26:39 <@dustymabe:matrix.org> !info 123. DNF: Do not download filelists by default - We'll open an issue and investigate this further 2024-01-03 17:27:18 <@dustymabe:matrix.org> !info 213. Wget2 as wget !link https://fedoraproject.org/wiki/Changes/Wget2asWget 2024-01-03 17:27:34 <@dustymabe:matrix.org> !info 213. Wget2 as wget 2024-01-03 17:27:39 <@dustymabe:matrix.org> !link https://fedoraproject.org/wiki/Changes/Wget2asWget 2024-01-03 17:27:55 <@dustymabe:matrix.org> !info 213. Wget2 as wget - Nothing to do, we don't ship wget. 2024-01-03 17:28:01 <@dustymabe:matrix.org> ^^ agree? 2024-01-03 17:28:11 <@jlebon:fedora.im> 👍️ 2024-01-03 17:28:34 <@dustymabe:matrix.org> !topic Open Floor 2024-01-03 17:28:44 <@dustymabe:matrix.org> anyone with anything for open floor? 2024-01-03 17:28:55 <@dustymabe:matrix.org> jdoss: Adam B ? 2024-01-03 17:29:25 <@jdoss:beeper.com> I got nothing this week. Happy new year folks. Lets make FCOS even more awesome in 2024! 2024-01-03 17:29:28 <@dustymabe:matrix.org> it's worth noting I ran into this myself this past week: https://github.com/coreos/fedora-coreos-tracker/issues/1641 2024-01-03 17:29:43 <@ajbouh:fedora.im> i've been trying to find and fix bugs associated with adding additional files to an FCOS build 2024-01-03 17:30:15 <@jdoss:beeper.com> I guess I am still fighting ipxe booting RPi CM 4s with FCOS but that is a swamp of pain and suffering right now. 2024-01-03 17:30:16 <@ajbouh:fedora.im> would love to know the right way to navigate that sort of thing, since it feels like i'm swimming against the current with everything i try 2024-01-03 17:30:37 <@jdoss:beeper.com> I just use layering for everything now. 2024-01-03 17:31:01 <@ajbouh:fedora.im> yeah, i want it baked in for the live image and to work offline, so hard mode x2 2024-01-03 17:31:28 <@jdoss:beeper.com> I like your speed Adam. 🙂 2024-01-03 17:31:41 <@dustymabe:matrix.org> Adam B: yeah. I'm not going to lie, `cosa` is very much built for building coreos and deviating from that too much might yield some pain 2024-01-03 17:31:43 <@ajbouh:fedora.im> the rootfs squash image inside the initrd is currently the blocker for me, since cpio has a 4GB max for file entries 2024-01-03 17:32:10 <@ajbouh:fedora.im> "some" :) 2024-01-03 17:32:25 <@ajbouh:fedora.im> honestly cosa has been pretty solid 2024-01-03 17:32:48 <@dustymabe:matrix.org> Adam B: one option may be to write the ISO to partition one of a disk image (i.e. USB key) and write other files to a filesystem on partition 2 2024-01-03 17:33:24 <@dustymabe:matrix.org> then after the install pick those files up and copy them into the right places 2024-01-03 17:33:29 <@ajbouh:fedora.im> yes, i've realized this might be my only option... 2024-01-03 17:33:39 <@ajbouh:fedora.im> still trying to figure out how this impacts updates 2024-01-03 17:34:03 <@ajbouh:fedora.im> and also how to safely modify the disk image without breaking its bootability 2024-01-03 17:34:58 <@dustymabe:matrix.org> you can also do https://matrix.to/#/!YWqcsiUQiCaqimYdQT:fedoraproject.org/$h-bXNIpf0JvGsss02bHdwjpjK4Mb-vPWGYBPajyIQ1c?via=fedoraproject.org&via=fedora.im&via=matrix.org but just write a the raw disk image into partition 2 and install that instead of what is baked in the ISO 2024-01-03 17:35:13 <@dustymabe:matrix.org> since it sounds like you are building the raw disk metal image without issue 2024-01-03 17:35:51 <@ajbouh:fedora.im> yes, raw disk image builds alright 2024-01-03 17:35:55 <@ajbouh:fedora.im> slowly, but alright 2024-01-03 17:36:01 <@dustymabe:matrix.org> Adam B: let's keep chatting over in the FCOS channel after the meeting 2024-01-03 17:36:13 <@dustymabe:matrix.org> anyone with anything else for open floor? if no I'll close out the meeting soon 2024-01-03 17:37:17 <@dustymabe:matrix.org> !endmeeting