Meeting started at 2024-03-27 16:30:00 UTC

The Meeting name is 'fedora_coreos_meeting'

!hi

!hi

Dusty Mabe (dustymabe) - he / him / his

Jean-Baptiste Trystram (jbtrystram) - he / him / his

!hi

No Fedora Accounts users have the @fifofonix:matrix.org Matrix Account defined

!hi

Timothée Ravier (siosm) - he / him / his

Hi all, let's start in about a minute...

!hi

Joseph Marrero (jmarrero)

.hi

As far as I can see there is one action item from the last meeting: !topic fifofonix to bring up a 1.28 cluster with zswap

so i did that and updated the issue.

as expected kubelet did not start

someone has requested i pass in --flag that allegedly allows. kubelet with zram and i'm going to get to that in the next week or so to see whether it works.

!hi

None (jlebon)

!hi

Daniel Rudolf (phrozenbyte)

Thank you for this fifofonix ❤️

no problem. sorry for the delay. too much skiing/snowboarding :o)

.hi

There's never too much of skiing :)

!hi aaradhak

Aashish Radhakrishnan (aaradhak)

!hi

Michael Armijo (marmijo)

Should I schedule another action item here, to check on progress next week?

Sure.

That is, if there is consensus, that checking the --flag (whatever it is - need to go to notes) is useful/relevant.

Let's move on to the topics for this week.

fifofonix: thanks for testing that! yeah, i think that'd be helpful. we'll need to devise a strategy there at some point, and the more information the better.

Sorry Jonathan Lebon - didn't mean to interupt

presumably we also want to see more recent k8s versions at some point too. will take a while but i do want to bump forward on that too soon.

Any volunteers to introduce this one?

I added the `meeting` label here. IIRC we typically do for new platform requests to discuss them?

Anything specific you want to talk about? They are adding support themselves, which is great

we should have invited that contributor to the community meeting x)

I think in general we need to voice any concern if we have any.. and approve the platform ID

I missed that they were planning to do the work to add the support, is that the case? i.e. we just mentor and do code reviews ?

that's the first time I heard about this Linode

proposed info: let's ask @ nesv to join the community meeting to discuss the request next week

oh I see. there are existing Ignition PR open

+1 for the akamai platform ID as it matches the one that has been used in cloud-init and I don't see reasons to diverge

one thing probably worth talking about is whether to make it an emerging platform or a full one. but it sounds like they'd like to have FCOS be a usable uploaded image, which would require more the latter.

just from the context in the ticket it almost seems like they want to use it for their own needs, and offering it to customers too is a win as well

Jonathan Lebon: it sounded like they'd have to do the image upload anyway, so it could be emerging and they do the build and upload IIUC

will we just add support for it and let them test and report issues for it?

i think i'd prefer we did the upload ourselves if it'll have high visibility

that info ends up in the stream metadata too, which is updated during releases

We should be the ones doing all the builds as we sign them

Well, the ostree commits

I'd say we're not yet at the stage where this question matters so we can revisit this discussion once we're there

would be good to dig a little more in the networking bits

it sounds like we can just assume DHCP?

but it's not clear what we lose out from doing that

(Note for Adam, let's talk about https://github.com/coreos/fedora-coreos-tracker/issues/1690 and then https://github.com/coreos/fedora-coreos-tracker/issues/1575 after)

Can we move on in that case to the next topic?

Anyone interested in investigating more about Linode/Akamai?

we probably also need to decide if we want to support regions that don't support the metadata service

How would we support those regions? I don't think we should

yeah, it'd be good to get more info on that bit too. how's that supposed to work overall?

yeah, it'd be good to get more info on that bit too. i.e. how's that supposed to work overall?

Given that it's a "new" service, it might just be that it's not yet available everywhere but planned

proposed: Reserve the `akamai` platform ID for the Akamai Connected Cloud (Linode) platform

travier: let's add some more color

right, but presumably people can still have their nodes do useful things somehow in those regions. is it e.g. all done via SSH postprovisioning ?

proposed: We think this is a good idea. We will reserve the `akamai` platform ID for the Akamai Connected Cloud (Linode) platform and support the contributors from Akamai on adding support for it.

does that help ^^

+1

+1

(sorry, have to drop for another meeting, but +1 overall from me on this!)

+1

> For Linode-provided instance images, there is a "helper" application that manages the SSH keys on an instance. All Linode-provided images have this helper pre-installed.

Maybe we can add support for that in afterburn

Ah - I probably should wait with the above a minute longer. Sorry folks

You got 4/5 +1 so I'd say it's fine. Let's move to the zip one

travier: can you lead this one please?

travier: I can outline it quickly if you want to, the issue's title is a bit misleading

Go head :)

Go ahead :)

I'd like to talk about FCOS' "ship only strictly required packages" principle. Starting point was that I was a little startled about FCOS not shipping `zip` (because I've never experienced FCOS to not ship such basic utility before). The discussion is no longer about adding `zip` as single package, but a change of principles.

IMHO FCOS should ship all utilities that are widely assumed to be "the basic tools". I know how this might sound, but I don't want to clutter FCOS ;-) The foremost question is: What's a "basic tool"? I don't want to reinvent the wheel, so being inspired by @travier I thought: why not use Fedora's default toolbox as baseline? So, by comparing the packages FCOS already ships and the packages the toolbox ships, the difference really is just marginally. FCOS already ships next to anything one would except, but nine packages with a total installed size of 3.58 MB.

Due to that I feel like that the current principles aren't very comprehensible from an user's perspective: FCOS ships everything, but nine supposedly arbitrary packages. So I wanna suggest changing FCOS' package principle to also include these basline packages.

Ideally, we include only what's strictly needed, so each included package is properly justified.

toolbox itself can be much looser defined and include more things as it's a container and not the base system

toolbox itself can be much loosely defined and include more things as it's a container and not the base system

I think the point of toolbox isn't really what default packages are included in toolbox versus FCOS. it's more that with toolbox you can then install whatever you want to investigate issues

dustymabe: Exactly, that's how I always perceived toolbox as well. For such basic things like extracting a .zip (truly, it's not about this single package because FCOS can indeed already extract .zip archives with bsdtar) using toolbox just feels a little exaggerated to me.

to me the most compelling reason to ship something like zip would be so that Ignition could pull and extract them

i.e. https://github.com/coreos/ignition/issues/867

then again - not sure if Ignition would do it with `unzip` the binary or a go library

Just for completeness, the nine packages I mentioned are the following: mtr, tcpdump, traceroute, whois, bc, tree, time, zip, and unzip.

Ignition would probably use a pure Go library if available

for `mtr` someone already requested it: https://github.com/coreos/fedora-coreos-tracker/issues/1644

and `whois` in https://github.com/coreos/fedora-coreos-tracker/issues/432

Any reason to not request each package independently and evaluate their usefulness? Each package has to have a clear goal.

Yeah, personally I rather want to talk about changing the principle, because from an user's perspective (can only speak for myself though) the current principle isn't very comprehensible having everything but nine packages.

Aside: apologies, need to drop. just want to say my dev fleet which uses a whole host of open source products (and our own products) is on f40 with zero incidents with all logging/monitoring clean. a smooth upgrade for a major (so far). only glitch in the matrix is i can't build NVIDIA's gpu driver for f40 but that is there issue and not atypical when we're leaning into updates. Big THANK YOU as usual.

Aside: apologies, need to drop. just want to say my dev fleet which uses a whole host of open source products (and our own products) is on f40 with zero incidents with all logging/monitoring clean. a smooth upgrade for a major (so far). only glitch in the matrix is i can't build NVIDIA's gpu driver for f40 but that is their issue and not atypical when we're leaning into updates. Big THANK YOU as usual.

PhrozenByte: I think the problem here is you are arbitrarily identifying 9 packages that you think should be included

whereas someone else could identify 15 packages they think should be included

and then we'd have that same conversation over again with them (after we added your 9) :)

100%

what we possibly should do is ID when packages (like zip/unzip) have an alternative and then put those in the docs

That's why I wanna talk about the principle ;-) I didn't make this list of 9 packages up, it just happens to be the only packages missing from Fedora's toolbox. Personally I don't need any of those packages.

for example in the `mtr` discussion we identified the alternative of `tracepath` does exist

The principle is there to minimize the image to run a stable host for container workflows. Changing that, will mean a lot of people will find essential a lot of different packages that are not part of the main goal of FCOS. If the functionality can be achieved by layering or using toolbox I don't see why we would include it.

The question is: What are the rules toolbox uses to add these packages? Why are they included in toolbox, but not in FCOS?

fedora toolbox packages list is a moving target

PhrozenByte: you'd have to ask the toolbox maintainers?

it's a separate project

The toolbox maintainers add tools there as they please as they are not constrained in the same way that we are

toolbox is a different project, with very different goal that what FCOS is

toolbox isn't really concerned by attack surface, image size, and so on..

I think it's quite an interesting discussion. In advance I'd like to thank for your considerations PhrozenByte

PhrozenByte++

dustymabe gave a cookie to phrozenbyte. They now have 1 cookie, 1 of which was obtained in the Fedora 39 release cycle

PhrozenByte: ++

c4rt0 gave a cookie to phrozenbyte. They now have 2 cookies, 2 of which were obtained in the Fedora 39 release cycle

Thanks

Yet they obviously don't include arbitrary packages and the list of packages happens to be extremely similar. Just for the record, I'm don't really stick on using toolbox as reference. I can only say that not having `zip` was very unexpected, even though I'm using FCOS for years now.

Yet they obviously don't include arbitrary packages and the list of packages happens to be extremely similar. Just for the record, I don't really stick on using toolbox as reference. I can only say that not having `zip` was very unexpected, even though I'm using FCOS for years now.

we do definitely add things to the base but we try to contemplate it heavily before we do

Just a reminder - we only have a window of about 6 minutes left until the official end of our today's discussion.

apiaseck wow, time passed quickly!

If everyone agrees, let's move to what seem to be the last topic for today.

apiaseck: let's propose and agree on it first?

but maybe.. does anyone else want to advocate specifically for zip or unzip ?

Propose that we document how to use bsdtar to unzip files

+1

Sounds like a reasonable suggestion for zip, yes 👍️

+1

TIL about bsdtar

Any other votes on the proposed?

(which is not in toolbox :) )

dustymabe also mentioned tracepath as alternative for mtr, should that be documented as well in terms of a more general "document alternatives" proposal?

good idea too

I will move on with agreed to the proposed above

!agree We will document how to use bsdtar to unzip files

Should we move on to last topic? Do you guys still have tim?

Should we move on to last topic? Do you guys still have time?

the `F39 drops some firmware packages that provide wifi/BT firmware` might be good to cover

but we're out of time :/

for two reasons

ok real quick on the first reason

The window for F40 changes is closing soon

we decided last time to implement some CLHM thing that would let users know that their firmware might be dropped for f40 when we planned to remove the firmware packages but then we never did it

so I propose we push that to f41 and actually do the work this time

I'm for doing less work and not doing it :)

'not doing it' == what?

if we do nothing right now those firmwares just stay in

adding support to CLHM and waiting for F41

I think we should not go out of our way to support things that are not there by default

so if it's more work, we don't do it and we drop those packages

but that's the thing. they were in the base by default

we can add a note to the release notes

but not wifi support

then they got removed because they got broke out into a subpackage

yeah, but the failure case here is bad

your system that was working (yes, you layered on networkmanager-wifi) now you can't connect to at all

I know, but there is only so much we can do. If no one did it in 6 months, why wait 6 more?

no one knew they needed to do it??????

oh wait, which part

sure, it's "only" in the tracker

no one did the CLHM helper or no one manually layered their wifi packages

so users don't see it

yeah, that's why we need to do something to tell them what they need to do

I'm for sending communication alongside the rebase

ok let me quickly get into why I wanted to bring this up today (time sensitive)

there are a few more firmwares that were broken out into subpackages: https://github.com/coreos/fedora-coreos-tracker/issues/1700#issuecomment-2021454488 two of them are wifi/BT firmwares (which fit in this same category)

I think whatever we do here, we should probably apply to those as well

We already don't include intel wifi firmwares (iwlwifi) which are also likely very common wifi firmwares

I guess someone could be running fedora coreos on a edge device using wifi... suddenly they upgrade to f40 and lose network?

correct

that seems bad

travier: looks like those were broken out into subpackages after?

https://src.fedoraproject.org/rpms/linux-firmware/commits/rawhide

Have we ever done something like that in the past? Where we took out support for hardware with an upgrade?

We expect users to read coreos-status mails and act on it. This is one those cases

To me, this is an edge case (pun not intended)

yes, this could break someone

but podman v5 is a bigger change

and it's not "avoidable" in comparison

here it's layering another package

but we did a CLM helper for that too

all I'm saying is let's just do what we planned to do last time

but I'm OK with getting outvoted :)

But overall, I'm OK keeping those. I'm OK with someone else doing the CLHM work. I don't think it's worth our time.

should we add the context to the ticket and see if anyone else wants to weigh in and vote on it next week?

(sorry for pushing the meeting long)

for https://github.com/coreos/fedora-coreos-tracker/issues/1700#issuecomment-2021454488 as those are audio firmware, I'd say it's much safer to drop them

nxpwireless-firmware: ah, maybe not :/

nxpwireless-firmware tiwilink-firmware aren't audio FW IIUC yeah, we can drop the audio ones

Proposed: We'll push back the removal to F41. If no-one does the CLHM message work before F41, we'll drop them

+1

Could be phrased more diplomatically

travier: that works for me.. do we want to take a stance on the two new FW removals?

ins't the `testing` stream for that ? we drop them, users notice 1% of machines down, fixes them before it's too late?

Let's include them for F40?

travier: OK

not everyone runs testing but yes, that's the goal

jbtrystram: yes, ideally

Let's re-discuss that next week?

travier: sounds good

i.e. re-discuss the two new FW removals?

I'd say the plan for the whole thing

proposed action: Let's re-discuss the plan of action for this issue next week.

+1

+1

I'll add context to the ticket

!action: Let's re-discuss the plan of action for this issue next week.

Thanks Adam

Just to keep the consistency...

Thanks for the feedback!

Thanks all for attending!

ash: is organizing a test day

Thanks Adam for running

@info F40 test day is next week: https://github.com/coreos/fedora-coreos-tracker/issues/1685

Yes we are looking for volunteers for the test week, so any advice (here or through other streams) is more than welcome!

Yes we are looking for volunteers for the test week, so any advice on how to increase number of participants (here or through other streams) is more than welcome!

Also, I came across this list of platform owners - platform owners - https://gitlab.cee.redhat.com/coreos/team-operations/-/blob/main/TESTING_INFRA.md?ref_type=heads

Also, I came across this list of platform owners - https://gitlab.cee.redhat.com/coreos/team-operations/-/blob/main/TESTING\_INFRA.md?ref\_type=heads

ash: This is an internal link :)

Also, I came across this list of platform owners -

Would be nice to update the list and ask people to test :)

yes, I am planning to reach out to everyone in the team and update the list

thanks everyone for coming today

With the FCOS test week mentioned I can now !endmeeting

Well, that didn't wok :o)