Meeting started at 2026-08-25 16:01:13 UTC

The Meeting name is 'ELN SIG 25 Aug '26'

The Meeting Name is now eln

!hi

Stephen Gallagher: Stephen Gallagher (sgallagh) - he / him / his

!hi

Simon de Vlieger: Simon de Vlieger (supakeen) - he / him / his

!hi

James Troy: James Troy (bhoy-troy)

Hi; so we can definitely do this, I can add toolbox images to image-builder and then they can be enabled but this is all very mixed across RHEL/CentOS/Fedora.

RHEL builds with Konflux (had to find out), CentOS builds its container image with image-builder but doesn't *have* a toolbox image, and then ELN is now to pick.

Personally I'd say, let's add toolbox images for ELN and CentOS to image-builder and enable them, done. But it depends how much you want that parity.

(as far as there is no parity)

There's no CS toolbox? Really?

[Troy Dawson](https://matrix.to/#/@tdawson:fedora.im) how come no CS toolbox image?

Just for the reason of there being no toolbox image in CentOS I'd say, let's do it in image-builder and then enable it for both ELN (which is already building one currently) *and* CentOS :)

But we'd need to ask Troy indeed.

it would sure be nice to have one, although that probably means adding CS to toolbox itself

Wouldn't we want to do all three in the same environment if possible?

doubtful RHEL would move *off* of konflux

We would, but it's split at the moment, RHEL isn't going to go to image-builder for the containers since they're on Konflux. for it.

yselkowitz: Yes, I was suggesting that we should consider using Konflux rather than image-builder for all three

(as in they use something else to build the containers *in* konflux)

!hi

But I have no idea of the state of Konflux in Fedora at the moment for doing these things.

If we're adding ELN and CS, why diverge?

Conan Kudo 😌: Neal Gompa (ngompa) - he / him / his

and afaik right now we can't call out to konflux as part of a compose, iiuc that's being worked on

@sgallagh:fedora.im Because Konflux isn't adopted yet in Fedora and CentOS.

in fedora that is

(for image builds)

the reason there's no CS toolbox image is that the toolbox people didn't want to support centos

it's been a long-standing problem

Be it containers, disks, or ISOs.

"didn't want to"??

yes

It's also perfectly good (and maybe sane?) to keep the container/toolbox builds for ELN on Kiwi until Konflux is viable for that parity instead.

Didn't want upstream to provide support for it, or didn't want toolbox to work on CS at all?

"Support" is overloaded

https://github.com/containers/toolbox/issues/275

didn't want to support CS based toolbox environments at all

Well, CentOS 7 I can absolutely see, since the kernel is ancient

CentOS being totally unwanted upstream is why there is no Hyperscale toolbox image either

The ticket also describes 'there are no official toolbox images for centos, only community' I think?

if CS provides official images, adding the support to toolbox itself is minimal

yes, but they don't want to, which is why no images exist

making the images is easy

e.g. for ELN

they just don't want them

https://github.com/containers/toolbox/pull/1814

*who* doesn't want them?

toolbx upstream

*Them*, obviously ;-)

each attempt to discuss it gets pushed to "use UBI instead"

!hi

afaics it's that toolbx upstream doesn't want to be *making* toolbx images

sorry i'm late kid dr appt

but CS should be making them, and toolbx just use them

https://github.com/containers/toolbox/issues/1019#issuecomment-1589091890

can't see them objecting to that?

toolbx won't take code to autoselect images they don't support

you'd basically have to manually select centos every time

Right but it seems most of that is 3 years old, perhaps it could be revisited or help them to support CentOS images if it's a CI problem?

(we're steering away from ELN a bit here I think)

That second link says they're running CS images in their CI, which doesn't sound hostile to CS to me...

last time I talked to them about it was devconf.cz last year

they were not open to it then

from everything I've seen, if CS builds their own toolbx images which are compliant, adding CLI support for CS should be a no-brainer

this sounded familiar to me, and relates to a lot of stuff I've worked on over the years. whats the ask for us here?

I think the juice isn't worth the squeeze

not anymore anyway

not anymore because...?

So the options for ELN's container (and toolbox) builds are: 1. migrate them to `image-builder` 2. stick with kiwi; in both cases they need to be migrated to Konflux in $future and in both cases they won't be 'built the same as RHEL' until the Konflux bit happens.

because toolbx upstream just strings us along

it's not worth wasting time on it

!hi

Troy Dawson: Troy Dawson (tdawson)

distrobox handles CS fine, why are we worried about toolbx?

Sorry I'm late. And to answer the question. Nobody has asked for a CS toolbox. No issue at all.

distrobox is fine, but unfortunately Red Hat makes us care about toolbx

do we?

Do we have any idea what a timeline looks like for Konflux in Fedora? (Read: "Should we just delay this until it's available?")

I have been repeatedly told I need to use toolbx over distrobox in Fedora because of it... so yes?

could we patch the toolbox package in CentOS Stream to use the community stream image?

*I* don't have any idea sadly I wish I did.

that's why all the Fedora deliverables ship toolbx instead of distrobox despite users begging us to switch away

could we ship both?

I'm sorry you were told that, I prefer you use the tool you want... because I respect your time 😊

we ship both moby-engine and podman in CoreOS, for example, I believe

yes, I appreciate you :)

alas you aren't the problem :(

I don't see any reason we couldn't ship both toolbx and distrobox images, but I've (personally) not heard anyone ask us to do so.

fair

is there such a thing as a "distrobox image"? haven't heard that term before

distrobox doesn't require anything different from toolbx

it just has more features than toolbx

distrobox has the ability to bootstrap an environment from standard images

not really, you just specify an image

toolbx lost that feature a long time ago, afaik

(distrobox and toolbx forked from the same original toolbox from Container Linux)

Anecdote != data, but when I last played with distrobox a couple years ago, it was entirely broken wrt SELinux.

the suse people made fixes for that, but they don't exist in fedora

I remember Eric Curain fixing some of that up a while back, not sure what current state is

I remember Eric Curtin fixing some of that up a while back, not sure what current state is

I never bothered to port any of it back to fedora since I am not supposed to ship it on Kinoite

OK, I think we're far off topic now, though. Sorry.

approaching the half-hour point here

sorry sorry

what was the original aska gain?

what was the original ask gain?

> So the options for ELN's container (and toolbox) builds are: 1. migrate them to image-builder 2. stick with kiwi; in both cases they need to be migrated to Konflux in $future and in both cases they won't be 'built the same as RHEL' until the Konflux bit happens.

what to do with ELN container and toolbox images

anyway, Konflux is unimportant, for an ELN pov, I'm not sure what we gain from a toolbox image, but if people care enough to make toolbx accept it, then we should have one

we already have an image, just a question of how we build it

With the added note that we *could* align ELN and CentOS on the container/toolbox images until Konflux happens in those by switching them over to `image-builder` but idk the value of that.

any portion of the bootc image help with that? it contains a kernel, but it still uses the host kernel, iiuc

not really

toolbox images are userspace only

the Fedora toolbox image is built as a separate rootfs with Kiwi. can we do the same for ELN while waiting on Konflux?

could we do a separate build with the konflux tenant we already have for just the usersapce build?

I think we need further investigation here, so maybe let's hold for now while we figure this out

@gotmax23:fedora.im That's the current state, the question was if we want to swap it over to `image-builder` to align ELN with CentOS or to wait until Konflux is viable in CentOS and Fedora and then switch to that instead :)

fair, happy to review any investigations

I'll update the ticket with the above options as well and then we can move on.

let's follow up in https://github.com/fedora-eln/eln/issues/597

I think we already build ELN toolbox images with kiwi

yes we do

not sure if there's benefit to switching to image-builder now just to switch again to Konflux?

we're not making any decisions about this today, and we're already past :30 so let's follow up in ticket

It aligns ELN for its container build with CentOS; that's a very minor benefit.

But I'll put all of that in the ticket.

every time at fedora branching, there are signing-related issues in ELN, because we have been using the rawhide key since the beginning

however, we do not ship any rawhide content in our repos, so at least theoretically we could have our own key and rotate less frequently (e.g. every 3 years with RHEL branchings) to avoid this

any thoughts?

I think it'd save a lot of work.

yeah, I don't see any significant downsides

it was annoying when all my ELN Packit builds broke after branching :)

it's always painful, and the more ELN is used, the more painful it gets

as long as it's possible for rawhide and eln content to mix in side tags, then I can't think of a downside either

For some quick context, in the early iterations of ELN, it was backed by Rawhide content, so sharing the key made sense.

the eln builds aren't signed at that point anyway, so afaik it *shouldn't* matter

We changed that some years ago, but the use of the Rawhide key just sort of continued via inertia

so my proposal is to work with releng to move to a new ELN key starting with F46/EL11 branching, so one more time, and use that for ~3 years until F52/EL12 branching

I like that proposal.

any other thoughts on this?

> Koji generates mock configs with ``gpgcheck=0`` in the yum/dnf configuration,

because packages in Koji's buildroot repos are unsigned.

so I think the signing issue shouldn't be a problem regardless

nothing else from me

And my axe! errr, my agreement.

there seems to be consensus here, so

[Stephen Gallagher](https://matrix.to/#/@sgallagh:fedora.im) ?

https://youtu.be/4wWb2XIoN4E?si=9oVtzhbYqQjFHxXU&t=56

it'd be nice if we could have it generate signed stuff, though...

Sorry, was distracted looking for that clip :)

since rpm has gradually been turning the screws on unsigned packages

This topic is a big one and I'd prefer to just ask everyone to read it and come to #eln:fedoraproject.org in the next day or so with feedback and thoughts.

I think it's too deep in the weeds to get into in 15 minutes.

Could we skip to https://github.com/fedora-eln/eln/issues/586 instead, today?

sure

This one is a little more straightforward. The amount of time we wait for builds to reach the `eln` tag once they're submitted to Bodhi is definitely not in the sweet spot, but we're not really sure what the "sweet spot" actually looks like.,

We want wait for the builds to reach the tag so that the next batch to fire off will have whatever changes the previous batch included.

But we also don't want (or maybe we do?) it to wait forever if the Bodhi update gets completely stuck and never pushes stable.

Most recently, the mass-rebuild had issues because the signing was taking well over an hour (the current timeout value) and so further batches were starting without the mass-rebuild builds in the buildroot, which caused some pain that yselkowitz had to clean up

Yaakov and I agree that it's better to wait as long as reasonable for things to get signed, but we have each suggested different approaches:

My suggestion was to have the timeout based on the size of the update; set a minimum timeout of half an hour and allow it to grow to up to 48 hours in the case of a huge number of builds.

Yaakov's was basically to just always set a very long timeout and add a feature allowing us to cancel the wait manually if we have external knowledge that waiting won't gain us anything.

Both ideas have their merits, and there may be other ideas out there, so please chime in :-)

/walloftext

to clarify, by default, EBS waits for one batch's Bodhi update to go stable before moving on to the next, but this timeout causes it stop waiting and move on to the next one anyways?

or am I misunderstanding how this works?

Correct

hmm, so what's the benefit to making the timeout shorter? isn't it better to wait longer then building things out of order and potentially breaking the buildroot?

hmm, so what's the benefit to making the timeout shorter? isn't it better to wait longer than building things out of order and potentially breaking the buildroot?

Assuming you mean the 30 minute floor in my suggestion: I was proposing making it shorter for very small batches where we could reasonably expect that if it was waiting past 30 minutes, it probably indicated an infra problem. Why wait an hour if we can reasonably guess that it's not going to help.

Assuming you mean the 30 minute floor in my suggestion: I was proposing making it shorter for very small batches where we could reasonably expect that if it was waiting past 30 minutes, it probably indicated an infra problem. Why wait an hour if we can reasonably guess that it's not going to help?

even if it's a small batch, signing could still be backed up due to unrelated updates and slow things down, no?

my point exactly

(Yaakov's suggestion sounds more reasonable to me, but you all obviously have more context than I do :)

gotmax23: Certainly possible

There's a fair question to be asked around "how long is too long to wait?", though. If signing is so backed up that it takes a two-package batch 15 hours to get through, that's also fifteen hours of backup before the next batch fires.

Yes, we queue them all and will kick them off together when the timeout happens, so they aren't lost.

At some point, being too far behind can start to impact consumers of ELN (such as Anaconda awaiting a fix to unbreak ELN in their CI)

it would be nice if there was a way to partially base the timeout number on current conditions/signing queue load. or at least, we know that, e.g., the python mass rebuild is happening, and can adjust it manually accordingly

unfortunately we are at time

We don't really have any mechanism to acquire that information, though

(Automatically, I mean)

right. I think there are signing queue metrics somewhere, but not sure how easy they are to access

And yeah, we're at time. If you would like to continue this discussion, I'll be in #eln:fedoraproject.org though!

anyway, thanks everyone!

thank you all, please follow up in tickets or in channel, and either see you there, or back here next week