2024-02-07 16:30:49 <@jlebon:fedora.im> !startmeeting fedora_coreos_meeting
2024-02-07 16:30:49 <@meetbot:fedora.im> Meeting started at 2024-02-07 16:30:49 UTC
2024-02-07 16:30:49 <@meetbot:fedora.im> The Meeting name is 'fedora_coreos_meeting'
2024-02-07 16:30:54 <@jlebon:fedora.im> !topic roll call
2024-02-07 16:30:59 <@dustymabe:matrix.org> !hi
2024-02-07 16:31:02 <@zodbot:fedora.im> Dusty Mabe (dustymabe) - he / him / his
2024-02-07 16:32:25 <@jmarrero:matrix.org> !hi
2024-02-07 16:32:27 <@zodbot:fedora.im> No Fedora Accounts users have the @jmarrero:matrix.org Matrix Account defined
2024-02-07 16:32:35 <@jmarrero:matrix.org> I think I still need to link this thing. 
2024-02-07 16:33:16 <@spresti:fedora.im> !hi
2024-02-07 16:33:17 <@zodbot:fedora.im> Steven Presti (spresti)
2024-02-07 16:33:29 <@gurssing:matrix.org> !hi gursewak
2024-02-07 16:33:30 <@zodbot:fedora.im> Gursewak Singh (gursewak)
2024-02-07 16:34:00 <@jbtrystram:matrix.org> !hi jbtrystram
2024-02-07 16:34:02 <@zodbot:fedora.im> Jean-Baptiste Trystram (jbtrystram) - he / him / his
2024-02-07 16:34:14 <@jlebon:fedora.im> alrighty, let's start!
2024-02-07 16:34:25 <@jlebon:fedora.im> !topic Action items from last meeting
2024-02-07 16:34:33 <@jlebon:fedora.im> 
travier open an issue to track hardening our units
2024-02-07 16:34:52 <@jlebon:fedora.im>  i don't think i've seen that issue come out
2024-02-07 16:35:07 <@jlebon:fedora.im> oh wait, nvm
!info travier filed https://github.com/coreos/fedora-coreos-tracker/issues/1662
2024-02-07 16:35:17 <@jlebon:fedora.im> !info travier field https://github.com/coreos/fedora-coreos-tracker/issues/1662
2024-02-07 16:35:59 <@jlebon:fedora.im> ok, let's move on to the first 
2024-02-07 16:36:02 <@jlebon:fedora.im> ok, let's move on to the first topic
2024-02-07 16:36:03 <@jlebon:fedora.im> !topic Consider zstd for compression of shipped artifacts
2024-02-07 16:36:09 <@jlebon:fedora.im> dustymabe: want to intro this one?
2024-02-07 16:36:42 <@spresti:fedora.im> !link https://github.com/coreos/fedora-coreos-tracker/issues/1660
2024-02-07 16:37:30 <@dustymabe:matrix.org> yep. this came up recently in a conversation with @baude - he was noticing it was taking a long time to decompress the xz applehv image and wondering on how to improve it. We switched to compress/decompress with zstd and I was thoroughly impressed by the performance (at least on his M2 mac)
2024-02-07 16:37:43 <@dustymabe:matrix.org> so I decided to dig in a little bit further and this ticket is the result
2024-02-07 16:38:23 <@dustymabe:matrix.org> one pain point we have (on the infra/releng side) is how long our pipelines take to run. Compression os a big part of that so I was interested to know what some real world comparison would look like
2024-02-07 16:38:42 <@dustymabe:matrix.org> There are some results in the ticket in a table
2024-02-07 16:39:39 <@dustymabe:matrix.org> It looks like level 10 zstd compression may give us a reasonable speedup versus extra storage cost ratio
2024-02-07 16:40:30 <@spresti:fedora.im> So 90% faster but 6% more space?
2024-02-07 16:40:33 <@dustymabe:matrix.org> One other thing I was proposing was to switch over our rawhide stream to use zstd compression as a way to flesh out the investigation a little more: https://github.com/coreos/fedora-coreos-config/pull/2840
2024-02-07 16:40:33 <@jlebon:fedora.im> wait, i'm confused. the impetus for this was to make decompression for users faster, but in testing the difference doesn't seem drastic
2024-02-07 16:40:59 <@dustymabe:matrix.org> which should give us some more data on actual speedup versus size 
2024-02-07 16:41:26 <@jlebon:fedora.im> e.g. in https://github.com/coreos/fedora-coreos-tracker/issues/1660#issue-2109301823 we have 13s vs 9s. and the different levels doesn't seem to affect that much
2024-02-07 16:41:34 <@jlebon:fedora.im> how come baude saw a drastic differnce?
2024-02-07 16:41:36 <@jlebon:fedora.im> how come baude saw a drastic difference?
2024-02-07 16:41:59 <@jbtrystram:matrix.org> maybe he have some hardware zstd acceleration ? 
2024-02-07 16:42:17 <@dustymabe:matrix.org> Jonathan Lebon: correct. That's on my one system. My hard drives are pretty fast. and it's also when I was doing the compress and decompress back to back (maybe local caching) helped? 
2024-02-07 16:42:48 <@dustymabe:matrix.org> jbtrystram: maybe?
2024-02-07 16:43:00 <@jlebon:fedora.im> jbtrystram: possibly. though that's not something we should assume most people have
2024-02-07 16:43:27 <@jlebon:fedora.im> so i think the discussion mostly is now about the pipeline speedup aspect
2024-02-07 16:43:33 <@jlebon:fedora.im> so i think the discussion mostly is now about the pipeline speedup aspect IIUC
2024-02-07 16:43:42 <@dustymabe:matrix.org> I'm interested in others experience/input (i.e. try it out on your system and report performance results)
2024-02-07 16:43:46 <@ravanelli:matrix.org> .hi
2024-02-07 16:44:16 <@jlebon:fedora.im> dustymabe: makes sense. i can give it a try too
2024-02-07 16:44:59 <@jbtrystram:matrix.org> I'll try and report back as well
2024-02-07 16:45:04 <@dustymabe:matrix.org> Jonathan Lebon: that's a big part of it, yes. If we could speed up our pipelines compression by a significant factor it would help us out. The `rawhide` test should at least give us some data
2024-02-07 16:45:47 <@dustymabe:matrix.org> jbtrystram: Jonathan Lebon if you want to play around with the level of compression you can just modify cmd-compress like I'm doing here: https://github.com/coreos/coreos-assembler/pull/3721
2024-02-07 16:46:34 <@dustymabe:matrix.org> one other thing we have to consider in this is compatibility - i.e. is it reasonable to assume that zstd is as ubiquitous as xz now? Also, does coreos-installer support it?
2024-02-07 16:49:29 <@dustymabe:matrix.org> That's mostly it from my side
2024-02-07 16:50:41 <@jlebon:fedora.im> i feel conflicted about this :)
2024-02-07 16:52:31 <@dustymabe:matrix.org> That's fair :) - I think we're still just information gathering
2024-02-07 16:52:39 <@jlebon:fedora.im> i think 15% smaller images is worth spending 30 mins rather than e.g. 5 mins in the pipeline. most of the time, we don't actually care how long pipeline builds take. except during prod builds where we have to retry on e.g. test flakes. it'd be great to have the pipeline go faster, but normally by the time we get to archive step, we're just about to upload to S3 and any failure after that are e.g. cloud tests and upgrade tests which can be retried.
2024-02-07 16:53:32 <@jlebon:fedora.im> from a user's perspective going from 661M to 757M for the qemu image for example is not great optics
2024-02-07 16:54:11 <@jlebon:fedora.im> if decompression was drastically different, then I could see an argument
2024-02-07 16:54:16 <@jlebon:fedora.im> if decompression was drastically different, then I could see an argument for bettering UX
2024-02-07 16:54:25 <@dustymabe:matrix.org> but the uncompressed size is the same? that's just the bandwidth that gets used (which yeah, not great)
2024-02-07 16:55:16 <@dustymabe:matrix.org> right. I'm interested in others experience with the decompression speed. I also need to try it on an image that I didn't just compress on my same system and see if it makes a difference
2024-02-07 16:56:33 <@jlebon:fedora.im> makes sense, yeah. would be good to have more info on that
2024-02-07 16:57:54 <@jbtrystram:matrix.org> looking around, it appears zstd offer faster decompression speed overall (for similar compression ratio), but I don't know enough about our storage costs to weight in :) 
2024-02-07 16:58:06 <@jbtrystram:matrix.org> worth noting : since Fedora 33, the filesystem is compressed by default with zstd.
2024-02-07 16:58:41 <@dustymabe:matrix.org> jbtrystram: you mean on BTRFS
2024-02-07 16:59:56 <@jlebon:fedora.im> !info we would like to gather more info on decompression speed and invite people to try out the zstd vs xz paths on their systems and report results.
2024-02-07 17:00:03 <@jlebon:fedora.im> dustymabe:  is that accurate? ^
2024-02-07 17:00:18 <@jbtrystram:matrix.org> looks like RPMs where switched to zstd as well in F31 : https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression
2024-02-07 17:02:06 <@dustymabe:matrix.org> Jonathan Lebon: yeah. seems accurate. Maybe we can agree to try it out in rawhide to give people real world artifacts to pull and decompress and also flesh out any potential bugs (like coreos-installer support)
2024-02-07 17:02:30 <@jlebon:fedora.im> your over concern about compatibility is a good one. i'm not sure how to evaluate it.
maybe we could check how far back are zstd tools available -- e.g. is it in CentOS 7?
2024-02-07 17:02:45 <@jlebon:fedora.im> (not meant to be answered here)
2024-02-07 17:03:30 <@jlebon:fedora.im> coreos-installer has zstd support for the initramfs at least
2024-02-07 17:03:57 <@jlebon:fedora.im> for `coreos-installer download`, it wouldn't be hard to test locally. you point it at the image using `--image-url ...` and then pass `--decompress`
2024-02-07 17:04:18 <@dustymabe:matrix.org> +1
2024-02-07 17:04:50 <@jlebon:fedora.im> briefly looking at the code, it doesn't look like it
2024-02-07 17:05:12 <@jlebon:fedora.im> https://github.com/coreos/coreos-installer/blob/ed985e44d3ce343d9301ffa00fea7e590547c962/src/download.rs#L114
2024-02-07 17:06:04 <@jlebon:fedora.im> !info this would require adding zstd image decompression support to coreos-installer
2024-02-07 17:06:12 <@dustymabe:matrix.org> it's OK we can fix it if we decide it's useful enough
2024-02-07 17:06:42 <@mnguyen:fedora.im> I vaguely remember two things baude mentioned in our meeting
2024-02-07 17:06:47 <@jlebon:fedora.im> right, let's evaluate first if it's worth it, and then we can figure out the missing pieces
2024-02-07 17:07:03 <@mnguyen:fedora.im> one is xz isn't included by default on mac?
2024-02-07 17:07:35 <@music:fedora.im> the standard `zstd` libraries and tools also support multi-threaded compression, while `xz` libraries don’t
2024-02-07 17:08:26 <@dustymabe:matrix.org> I think we can move on to another topic
2024-02-07 17:08:45 <@jlebon:fedora.im> music: xz supports multithreaded compression, no? we pass `-T<N>` in our pipelines
2024-02-07 17:08:50 <@jlebon:fedora.im> dustymabe: ack, sounds good
2024-02-07 17:09:06 <@jlebon:fedora.im> !topic tracker: Fedora 40 changes considerations 
2024-02-07 17:09:12 <@jlebon:fedora.im> !link https://github.com/coreos/fedora-coreos-tracker/issues/1626
2024-02-07 17:09:55 <@jlebon:fedora.im> looks like there's a few more that were added
2024-02-07 17:10:35 <@music:fedora.im> Hmm, you’re right. Added in 2014, apparently, and I just never noticed.
2024-02-07 17:10:40 <@jlebon:fedora.im> !topic Update the system JDK in Fedora from java-17-openjdk to java-21-openjdk.
2024-02-07 17:10:43 <@jlebon:fedora.im> !link https://fedoraproject.org/wiki/Changes/Java21
2024-02-07 17:10:51 <@jlebon:fedora.im> we don't ship java
2024-02-07 17:11:12 <@jlebon:fedora.im> !topic ROCm 6 Release
2024-02-07 17:11:16 <@jlebon:fedora.im> !link https://fedoraproject.org/wiki/Changes/ROCm6Release
2024-02-07 17:12:11 <@jlebon:fedora.im> cool, TIL.
we don't ship this.
2024-02-07 17:12:22 <@jlebon:fedora.im> !topic PyTorch Release
2024-02-07 17:12:25 <@jlebon:fedora.im> !link https://fedoraproject.org/wiki/Changes/PyTorchRelese
2024-02-07 17:12:58 <@jlebon:fedora.im> we don't ship PyTorch
2024-02-07 17:13:04 <@jlebon:fedora.im> and similarly for the last one (iotop)
2024-02-07 17:13:32 <@spresti:fedora.im> Should we info it?
2024-02-07 17:13:56 <@jlebon:fedora.im> !info none of the new changes should affect FCOS directly. we don't ship any of these packages by default.
2024-02-07 17:14:21 <@spresti:fedora.im> Thank you! (sorry was not sure if we should)
2024-02-07 17:14:43 <@jlebon:fedora.im> np! thanks for reminding me. presumably i should've done it for each one
anyway, shall we move on to the last topic?
2024-02-07 17:15:47 <@jlebon:fedora.im> !topic highlights from FCOS 2023
2024-02-07 17:15:52 <@jlebon:fedora.im> no links for this one
2024-02-07 17:16:41 <@dustymabe:matrix.org> yeah. mainly one of our team members wants to do a presentation and include some highlights from FCOS for 2023
2024-02-07 17:16:55 <@dustymabe:matrix.org> I figure maybe we can sift through https://fedoraproject.org/coreos/release-notes?arch=x86_64&stream=stable ?
2024-02-07 17:17:05 <@dustymabe:matrix.org> or source inspiration from somewhere else
2024-02-07 17:17:16 <@dustymabe:matrix.org> I can probably try to pull and look at the countme stats again
2024-02-07 17:18:23 <@dustymabe:matrix.org> I feel like most of what we have been up to is maintenance, new platform support, and "behind the scenes" type stuff like OSBuild and the Container Native stuff
2024-02-07 17:18:55 <@dustymabe:matrix.org> From the release notes here's what jumped out to me:

```
Added Apple Hypervisor support
Support root on iSCSI (non-HBA)
Platform Request: Microsoft Hyper-V
Added support for ppc64le
GCP added aarch64 image
Kubevirt
Included podman quadlet functionality
```
2024-02-07 17:18:59 <@jlebon:fedora.im> yeah definitely. which could be worth highlighting too depending on the crowd
2024-02-07 17:19:10 <@dustymabe:matrix.org> not sure if all of those are worthy or not
2024-02-07 17:19:42 <@jlebon:fedora.im> maybe 7 is too small?
2024-02-07 17:21:17 <@jlebon:fedora.im> (going over the list too)
2024-02-07 17:22:47 <@jlebon:fedora.im> nmstate enablement
Fedora 38 rebase
ISO from RAM work maybe? 
2024-02-07 17:23:02 <@dustymabe:matrix.org> WFM - depends on the audience probably
2024-02-07 17:23:16 <@jlebon:fedora.im> i'd add the osbuild work too honestly and let them decide if it makes sense to talk about that
2024-02-07 17:23:24 <@dustymabe:matrix.org> yeah
2024-02-07 17:24:09 <@jlebon:fedora.im> seems like we have a good list :)
2024-02-07 17:24:29 <@jlebon:fedora.im> should move to open floor?
2024-02-07 17:25:00 <@jlebon:fedora.im> !topic Open Floor
2024-02-07 17:25:28 <@jlebon:fedora.im> anything anyone wants to bring up?
2024-02-07 17:25:43 <@dustymabe:matrix.org> nothing from me today other than I might not make meetings for a few weeks later this month/early next month
2024-02-07 17:26:29 <@spresti:fedora.im> Been spending a bit of time working on the fcos action, the action-items are not getting matched right still. 
Soon should be fixed https://github.com/coreos/fcos-meeting-action/pull/69


2024-02-07 17:26:42 <@dustymabe:matrix.org> should we remind the next person in the rotation?
2024-02-07 17:27:04 <@jbtrystram:matrix.org> that's me i think
2024-02-07 17:27:28 <@spresti:fedora.im> I hope to have it fixed by then ;'(
2024-02-07 17:27:45 <@jlebon:fedora.im> jbtrystram: consider yourself reminded :)
2024-02-07 17:27:49 <@jmarrero:matrix.org> on https://github.com/coreos/fedora-coreos-tracker/issues/1647 We still think this is related to bootc somehow? I am not sure from the conversation how is this not just a kernel issue.
2024-02-07 17:28:59 <@dustymabe:matrix.org> jmarrero: not bootc, but something ostree/bootc related maybe? most other variants of Fedora aren't reporting issues only us (bootc and coreos)
2024-02-07 17:29:26 <@dustymabe:matrix.org> I'd love for someone to dig in further
2024-02-07 17:29:55 <@jmarrero:matrix.org> OK I will bring it to my meetings with the team and Colin and see what we find. 
2024-02-07 17:30:28 <@dustymabe:matrix.org> jmarrero: i reserve the right to be wrong about the problem being bootc/ostree related :) 
2024-02-07 17:30:33 <@jlebon:fedora.im> probably not bootc related if we're hitting it
2024-02-07 17:30:42 <@dustymabe:matrix.org> but from the current information that's where it leads
2024-02-07 17:30:57 <@jlebon:fedora.im> this looks like some grub config issue maybe?
2024-02-07 17:31:25 <@dustymabe:matrix.org> Jonathan Lebon: yeah. i'm mostly pointing at "something in our  stack" 
2024-02-07 17:31:25 <@jmarrero:matrix.org> It's weird because it's only aarch64 right?
2024-02-07 17:31:39 <@dustymabe:matrix.org> correct
2024-02-07 17:32:17 <@dustymabe:matrix.org> IIRC this started happening before we enabled OSBuild so it shouldn't be OSBuild related
2024-02-07 17:32:25 <@jmarrero:matrix.org> I am not sure how different arm and 86 grub configs are.
2024-02-07 17:32:30 <@dustymabe:matrix.org> IIRC this started happening before we enabled OSBuild in rawhide so it shouldn't be OSBuild related
2024-02-07 17:32:42 <@jmarrero:matrix.org> I am not sure how different arm and x86 grub configs are.
2024-02-07 17:33:18 <@dustymabe:matrix.org> jmarrero: maybe we should just start with "can you reproduce the problem" and go from there?
2024-02-07 17:33:42 <@jmarrero:matrix.org> Yeah, I'll dig.
2024-02-07 17:33:46 <@jlebon:fedora.im> Peter talks about changes that were done in qemu to work around this. would be good to dig into that too
2024-02-07 17:34:31 <@dustymabe:matrix.org> yeah I don't really understand that point from him. if that were true wouldn't the other variants be having trouble in their testing?
2024-02-07 17:34:34 <@dustymabe:matrix.org> i.e. openQA?
2024-02-07 17:34:42 <@dustymabe:matrix.org> which I assume uses qemu
2024-02-07 17:34:59 <@dustymabe:matrix.org> anyway we can probably close out the meeting and take this to the CoreOS channel
2024-02-07 17:35:02 <@jlebon:fedora.im> i would assume so. except our grub configs are very unique to us currently
2024-02-07 17:35:21 <@jlebon:fedora.im> so might be a combination of bug in QEMU + specific grub serial config or something
2024-02-07 17:35:25 <@jlebon:fedora.im> anyway, agreed
2024-02-07 17:35:39 <@jlebon:fedora.im> anything else for open floor?
2024-02-07 17:36:07 <@jbtrystram:matrix.org> I have some findings to share about the kdump SSH issue but not sure if this is the right place ?
2024-02-07 17:36:24 <@jlebon:fedora.im> jbtrystram: let's take that to the FCOS channel too :)
2024-02-07 17:36:49 <@jlebon:fedora.im> closing the meeting in 30 seconds if quiet
2024-02-07 17:37:21 <@jlebon:fedora.im> !endmeeting