Meeting started at 2024-02-07 16:30:49 UTC

The Meeting name is 'fedora_coreos_meeting'

!hi

Dusty Mabe (dustymabe) - he / him / his

!hi

No Fedora Accounts users have the @jmarrero:matrix.org Matrix Account defined

I think I still need to link this thing.

!hi

Steven Presti (spresti)

!hi gursewak

Gursewak Singh (gursewak)

!hi jbtrystram

Jean-Baptiste Trystram (jbtrystram) - he / him / his

alrighty, let's start!

travier open an issue to track hardening our units

i don't think i've seen that issue come out

oh wait, nvm !info travier filed https://github.com/coreos/fedora-coreos-tracker/issues/1662

ok, let's move on to the first

ok, let's move on to the first topic

dustymabe: want to intro this one?

yep. this came up recently in a conversation with @baude - he was noticing it was taking a long time to decompress the xz applehv image and wondering on how to improve it. We switched to compress/decompress with zstd and I was thoroughly impressed by the performance (at least on his M2 mac)

so I decided to dig in a little bit further and this ticket is the result

one pain point we have (on the infra/releng side) is how long our pipelines take to run. Compression os a big part of that so I was interested to know what some real world comparison would look like

There are some results in the ticket in a table

It looks like level 10 zstd compression may give us a reasonable speedup versus extra storage cost ratio

So 90% faster but 6% more space?

One other thing I was proposing was to switch over our rawhide stream to use zstd compression as a way to flesh out the investigation a little more: https://github.com/coreos/fedora-coreos-config/pull/2840

wait, i'm confused. the impetus for this was to make decompression for users faster, but in testing the difference doesn't seem drastic

which should give us some more data on actual speedup versus size

e.g. in https://github.com/coreos/fedora-coreos-tracker/issues/1660#issue-2109301823 we have 13s vs 9s. and the different levels doesn't seem to affect that much

how come baude saw a drastic differnce?

how come baude saw a drastic difference?

maybe he have some hardware zstd acceleration ?

Jonathan Lebon: correct. That's on my one system. My hard drives are pretty fast. and it's also when I was doing the compress and decompress back to back (maybe local caching) helped?

jbtrystram: maybe?

jbtrystram: possibly. though that's not something we should assume most people have

so i think the discussion mostly is now about the pipeline speedup aspect

so i think the discussion mostly is now about the pipeline speedup aspect IIUC

I'm interested in others experience/input (i.e. try it out on your system and report performance results)

.hi

dustymabe: makes sense. i can give it a try too

I'll try and report back as well

Jonathan Lebon: that's a big part of it, yes. If we could speed up our pipelines compression by a significant factor it would help us out. The `rawhide` test should at least give us some data

jbtrystram: Jonathan Lebon if you want to play around with the level of compression you can just modify cmd-compress like I'm doing here: https://github.com/coreos/coreos-assembler/pull/3721

one other thing we have to consider in this is compatibility - i.e. is it reasonable to assume that zstd is as ubiquitous as xz now? Also, does coreos-installer support it?

That's mostly it from my side

i feel conflicted about this :)

That's fair :) - I think we're still just information gathering

i think 15% smaller images is worth spending 30 mins rather than e.g. 5 mins in the pipeline. most of the time, we don't actually care how long pipeline builds take. except during prod builds where we have to retry on e.g. test flakes. it'd be great to have the pipeline go faster, but normally by the time we get to archive step, we're just about to upload to S3 and any failure after that are e.g. cloud tests and upgrade tests which can be retried.

from a user's perspective going from 661M to 757M for the qemu image for example is not great optics

if decompression was drastically different, then I could see an argument

if decompression was drastically different, then I could see an argument for bettering UX

but the uncompressed size is the same? that's just the bandwidth that gets used (which yeah, not great)

right. I'm interested in others experience with the decompression speed. I also need to try it on an image that I didn't just compress on my same system and see if it makes a difference

makes sense, yeah. would be good to have more info on that

looking around, it appears zstd offer faster decompression speed overall (for similar compression ratio), but I don't know enough about our storage costs to weight in :)

worth noting : since Fedora 33, the filesystem is compressed by default with zstd.

jbtrystram: you mean on BTRFS

dustymabe: is that accurate? ^

looks like RPMs where switched to zstd as well in F31 : https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression

Jonathan Lebon: yeah. seems accurate. Maybe we can agree to try it out in rawhide to give people real world artifacts to pull and decompress and also flesh out any potential bugs (like coreos-installer support)

your over concern about compatibility is a good one. i'm not sure how to evaluate it. maybe we could check how far back are zstd tools available -- e.g. is it in CentOS 7?

(not meant to be answered here)

coreos-installer has zstd support for the initramfs at least

for `coreos-installer download`, it wouldn't be hard to test locally. you point it at the image using `--image-url ...` and then pass `--decompress`

+1

briefly looking at the code, it doesn't look like it

https://github.com/coreos/coreos-installer/blob/ed985e44d3ce343d9301ffa00fea7e590547c962/src/download.rs#L114

it's OK we can fix it if we decide it's useful enough

I vaguely remember two things baude mentioned in our meeting

right, let's evaluate first if it's worth it, and then we can figure out the missing pieces

one is xz isn't included by default on mac?

the standard `zstd` libraries and tools also support multi-threaded compression, while `xz` libraries don’t

I think we can move on to another topic

music: xz supports multithreaded compression, no? we pass `-T<N>` in our pipelines

dustymabe: ack, sounds good

looks like there's a few more that were added

Hmm, you’re right. Added in 2014, apparently, and I just never noticed.

we don't ship java

cool, TIL. we don't ship this.

we don't ship PyTorch

and similarly for the last one (iotop)

Should we info it?

Thank you! (sorry was not sure if we should)

np! thanks for reminding me. presumably i should've done it for each one anyway, shall we move on to the last topic?

no links for this one

yeah. mainly one of our team members wants to do a presentation and include some highlights from FCOS for 2023

I figure maybe we can sift through https://fedoraproject.org/coreos/release-notes?arch=x86_64&stream=stable ?

or source inspiration from somewhere else

I can probably try to pull and look at the countme stats again

I feel like most of what we have been up to is maintenance, new platform support, and "behind the scenes" type stuff like OSBuild and the Container Native stuff

From the release notes here's what jumped out to me: ``` Added Apple Hypervisor support Support root on iSCSI (non-HBA) Platform Request: Microsoft Hyper-V Added support for ppc64le GCP added aarch64 image Kubevirt Included podman quadlet functionality ```

yeah definitely. which could be worth highlighting too depending on the crowd

not sure if all of those are worthy or not

maybe 7 is too small?

(going over the list too)

nmstate enablement Fedora 38 rebase ISO from RAM work maybe?

WFM - depends on the audience probably

i'd add the osbuild work too honestly and let them decide if it makes sense to talk about that

yeah

seems like we have a good list :)

should move to open floor?

anything anyone wants to bring up?

nothing from me today other than I might not make meetings for a few weeks later this month/early next month

Been spending a bit of time working on the fcos action, the action-items are not getting matched right still. Soon should be fixed https://github.com/coreos/fcos-meeting-action/pull/69

should we remind the next person in the rotation?

that's me i think

I hope to have it fixed by then ;'(

jbtrystram: consider yourself reminded :)

on https://github.com/coreos/fedora-coreos-tracker/issues/1647 We still think this is related to bootc somehow? I am not sure from the conversation how is this not just a kernel issue.

jmarrero: not bootc, but something ostree/bootc related maybe? most other variants of Fedora aren't reporting issues only us (bootc and coreos)

I'd love for someone to dig in further

OK I will bring it to my meetings with the team and Colin and see what we find.

jmarrero: i reserve the right to be wrong about the problem being bootc/ostree related :)

probably not bootc related if we're hitting it

but from the current information that's where it leads

this looks like some grub config issue maybe?

Jonathan Lebon: yeah. i'm mostly pointing at "something in our stack"

It's weird because it's only aarch64 right?

correct

IIRC this started happening before we enabled OSBuild so it shouldn't be OSBuild related

I am not sure how different arm and 86 grub configs are.

IIRC this started happening before we enabled OSBuild in rawhide so it shouldn't be OSBuild related

I am not sure how different arm and x86 grub configs are.

jmarrero: maybe we should just start with "can you reproduce the problem" and go from there?

Yeah, I'll dig.

Peter talks about changes that were done in qemu to work around this. would be good to dig into that too

yeah I don't really understand that point from him. if that were true wouldn't the other variants be having trouble in their testing?

i.e. openQA?

which I assume uses qemu

anyway we can probably close out the meeting and take this to the CoreOS channel

i would assume so. except our grub configs are very unique to us currently

so might be a combination of bug in QEMU + specific grub serial config or something

anyway, agreed

anything else for open floor?

I have some findings to share about the kdump SSH issue but not sure if this is the right place ?

jbtrystram: let's take that to the FCOS channel too :)

closing the meeting in 30 seconds if quiet