16:29:39 <dustymabe> #startmeeting fedora_coreos_meeting 16:29:39 <zodbot> Meeting started Wed Nov 23 16:29:39 2022 UTC. 16:29:39 <zodbot> This meeting is logged and archived in a public location. 16:29:39 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 16:29:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:29:39 <zodbot> The meeting name has been set to 'fedora_coreos_meeting' 16:29:43 <travier> .hello siosm 16:29:44 <zodbot> travier: siosm 'TimothΓ©e Ravier' <travier@redhat.com> 16:29:45 <dustymabe> #topic roll call 16:29:51 <dustymabe> .hi 16:29:52 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com> 16:30:14 <fifofonix> .hi 16:30:15 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com> 16:30:18 <c4rt0> .hi 16:30:18 <zodbot> c4rt0: c4rt0 'Adam Piasecki' <c4rt0gr4ph3r@gmail.com> 16:30:23 <walters> .hi 16:30:24 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com> 16:30:41 <dustymabe> #chair travier fifofonix c4rt0 walters 16:30:41 <zodbot> Current chairs: c4rt0 dustymabe fifofonix travier walters 16:30:46 <mnguyen> .hello mnguyen 16:30:47 <zodbot> mnguyen: mnguyen 'Michael Nguyen' <mnguyen@redhat.com> 16:30:56 <dustymabe> #chair ravanelli mnguyen 16:30:56 <zodbot> Current chairs: c4rt0 dustymabe fifofonix mnguyen ravanelli travier walters 16:31:02 <lorbus> .hi 16:31:03 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com> 16:31:20 <dustymabe> #chair lorbus 16:31:20 <zodbot> Current chairs: c4rt0 dustymabe fifofonix lorbus mnguyen ravanelli travier walters 16:31:51 <jlebon> .hello2 16:31:52 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com> 16:32:03 <anthr76[m]> .hello 16:32:03 <zodbot> anthr76[m]: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1". 16:32:14 <anthr76[m]> .hello anthr76 16:32:15 <zodbot> anthr76[m]: anthr76 'Anthony Rabbito' <hello@anthonyrabbito.com> 16:32:18 <dustymabe> #chair jlebon anthr76[m] 16:32:18 <zodbot> Current chairs: anthr76[m] c4rt0 dustymabe fifofonix jlebon lorbus mnguyen ravanelli travier walters 16:32:21 <dustymabe> welcome all 16:32:34 <dustymabe> #topic Action items from last meeting 16:32:47 <dustymabe> Looks like there were no action items from last meeting, so we can move on 16:32:55 <dustymabe> #topic FOSDEM 2023: Image Based Linux dev room 16:32:59 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1348 16:33:05 * dustymabe passes floor to travier 16:33:11 <travier> Heya 16:33:19 <c4rt0> Hi all 16:33:37 <dustymabe> π 16:33:41 <travier> The UAPI group / image based summit crowd is organizing a dev room at FOSDEM 16:34:09 <travier> Feel free to submit a talk about FCOS (or anything else related to the topic) there if you intend to be there 16:34:13 <travier> that's it :) 16:34:26 <travier> (I likely won't be at FOSDEM this year) 16:35:11 <dustymabe> yeah, since devconf moved to the summer it makes going to FOSDEM for non-Europe based harder 16:35:47 <ravanelli> .hi 16:35:49 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com> 16:35:54 <dustymabe> thanks for sharing travier 16:36:04 <travier> π 16:36:07 <dustymabe> #chair ravanelli 16:36:07 <zodbot> Current chairs: anthr76[m] c4rt0 dustymabe fifofonix jlebon lorbus mnguyen ravanelli travier walters 16:36:24 <dustymabe> #topic amd-gpu-firmware no longer being installed between stable to testing 16:36:29 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1345 16:36:58 <dustymabe> For this one the linux-firmware package broke out the GPU firmware it was previously shipping in the main package into subpackages 16:37:17 <dustymabe> in f36 the subpackages were requires, in f37+ they are recommends 16:37:25 <dustymabe> so when we moved to f37 we dropped them out 16:37:47 <dustymabe> the question is: should we add them back? 16:37:57 <dustymabe> I think the answer is probably yes 16:38:02 <jlebon> how many new subpackages fall in this category? 16:38:14 <dustymabe> but I think it's worth us discussing how much we think they are used 16:38:48 <dustymabe> jlebon: amd-gpu-firmware intel-gpu-firmware nvidia-gpu-firmware 16:39:01 <travier> $ rpm -qi amd-gpu-firmware-20221109-144.fc37.noarch | grep Size 16:39:01 <travier> Size : 22564513 16:39:47 <dustymabe> is that ~ 22MiB ? 16:40:05 <travier> $ rpm -qi intel-gpu-firmware-20221109-144.fc37.noarch| grep Size 16:40:05 <travier> Size : 8329555 16:40:05 <travier> $ rpm -qi nvidia-gpu-firmware-20221109-144.fc37.noarch| grep Size 16:40:05 <travier> Size : 1262337 16:40:26 <jlebon> dustymabe: looks like it 16:40:35 <travier> 22 + 8 +1 16:41:07 <dustymabe> definitely not insignificant 16:41:52 <travier> I think we can probably skip intel & nvidia 16:42:05 <travier> it's unlikely someone will use them right now 16:42:25 <jlebon> travier: can you expand on that? 16:42:34 <anthr76[m]> travier: What is your reasoning? 16:42:52 <dustymabe> ehh, I think we probably shouldn't split hairs - probably should do all or nothing 16:43:07 <jlebon> yeah, agreed 16:44:17 <jlebon> this is technically a regression, so i'm inclined to add them back before it hits stable even if we decide later on to remove them with some window 16:44:28 <travier> this is for GPU/OpenCL workloads 16:44:39 <travier> NVIDIA will require the binary drivers 16:44:57 <dustymabe> jlebon: yeah, that's a good point 16:45:00 <travier> I don't think there is Intel cards for that although we have the ARC ones now 16:45:24 <travier> so I really think we don't need the NVIDIA ones 16:45:35 <travier> intel maybe, amd yes 16:46:06 <jlebon> travier: couldn't some users be pulling in the drivers themselves though? 16:46:20 <travier> what do you mean? 16:46:24 <anthr76[m]> FWIW I know quite a few folks using iGPUs with Intel way more then AMD in fact. https://github.com/intel/intel-device-plugins-for-kubernetes 16:46:55 <anthr76[m]> The real question (to me) is do all GPUs require direct firmware loading? If not which ones don't 16:47:21 <jlebon> travier: are the nvidia drivers easily available in a way that we could expect some users to layer them? 16:47:57 <travier> jlebon: you get them from RPM Fusion and usually you need to freeze the kernel as it moves too fast in Fedora for NVIDIA 16:48:19 <anthr76[m]> Also, IIRC these really aren't drivers more or less the layer before the driver. mesa (a driver for example) can't start if the firmware isn't initialized. 16:48:25 <travier> I really don't see users running NVIDIA workloads on nouveau 16:48:34 <travier> that would be a complete waste of maney 16:48:36 <travier> money* 16:49:20 <travier> (but if you want to add all firmware I won't mind, just trying to reduce the size of what we add) 16:49:37 <jlebon> travier: ack ok, so it's possible they do pull it in, but unlikely 16:49:52 <dustymabe> yeah even so i think nvidia is working on making the open source side of things better and I'd prefer not to track progress there. I think I'd rather just include it especially since it's the smallest of the 3 16:49:52 <jlebon> yeah, the 1M cost doesn't seem worth being inconsistent 16:50:02 <travier> the binary NVIDIA drivers do not use those firmwares files 16:50:11 <travier> they use their own 16:50:18 * dustymabe wishes all of the drivers were just open source 16:50:27 <travier> soon! 16:50:29 <travier> :) 16:50:38 <anthr76[m]> β’οΈ 16:50:49 <fifofonix> fifofonix: actually experimenting with nvidia's open kernel for the kernel interface piece presently. :o) 16:50:53 <dustymabe> so OK - maybe we should open a new issue to really discuss what we think we should do in the future here 16:51:23 <dustymabe> but for now maybe we should do what jlebon suggests and get them back in before we ship the next `stable` 16:51:31 <travier> +1 16:52:13 <jlebon> +1 16:52:28 <travier> do we have numbers for image size between f36 & f37? 16:52:47 <anthr76[m]> +1 16:53:05 <dustymabe> #proposed we will add the amd-gpu-firmware intel-gpu-firmware nvidia-gpu-firmware packages back to FCOS to restore the previous functionality we had and open a new ticket to discuss the merits of these packages and if we ever want to remove them in the future. 16:53:26 <dustymabe> travier: we could compare the meta.json size values 16:54:01 <jlebon> 1.55G vs 1.57G for the uncompressed qemu qcow2 it looks like 16:54:25 <jlebon> ack to proposal 16:54:33 <travier> ack 16:55:49 <dustymabe> any other inputs? 16:55:55 <dustymabe> vote 16:56:52 * dustymabe will +1 16:57:03 <jlebon> anthr76[m]: nice work noticing this before it gets to stable! 16:57:09 <dustymabe> #agreed we will add the amd-gpu-firmware intel-gpu-firmware nvidia-gpu-firmware packages back to FCOS to restore the previous functionality we had and open a new ticket to discuss the merits of these packages and if we ever want to remove them in the future. 16:57:15 <dustymabe> yes thank you anthr76[m] 16:57:24 <anthr76[m]> Happy to help and be here :) 16:58:01 <dustymabe> we generally do a thorough analysis of added/removed packages (see https://github.com/coreos/fedora-coreos-tracker/issues/1221) but I just didn't have time to do it this cycle 16:58:27 <dustymabe> #topic open floor 16:58:37 <travier> this one is tricky as it's not removed, it's "not added" :) 16:59:04 <dustymabe> travier: right, but rpm-ostree db diff would report on it anyway 16:59:04 <travier> as it's a new sub package 16:59:11 <dustymabe> so I would have caught it 16:59:17 <travier> would it? 16:59:21 <travier> I don't think so 16:59:26 <dustymabe> absolutely it would 16:59:26 <jlebon> travier: the subpackages are in f36 16:59:39 <dustymabe> this ^^ 16:59:40 <travier> oh, my bad then 16:59:47 <travier> π 16:59:50 <dustymabe> travier: sorry, you would have been right otherwise 16:59:57 <travier> π 17:00:36 <travier> https://fedoraproject.org/wiki/Changes/Linux_Firmware_Minimization 17:00:37 <travier> ? 17:00:46 <dustymabe> note that soon we'll try to start evaluating the change proposals for f38 17:01:11 <dustymabe> travier: interesting.. maybe we overlooked that 17:01:32 <travier> https://koji.fedoraproject.org/koji/buildinfo?buildID=2089044 > I can see them here indeed 17:01:39 <dustymabe> weird. I don't see it in https://github.com/coreos/fedora-coreos-tracker/issues/1222 17:01:56 <anthr76[m]> I saw the breakout on my desktop, but wasn't aware of the change proposal. So I didn't think much of it on my FCOS nodes until GPUs stop initializing :) 17:02:28 <dustymabe> hmm. it's not in the list of 37 changes: https://fedoraproject.org/wiki/Releases/37/ChangeSet#Fedora_Linux_37_Accepted_System-Wide_Changes 17:02:29 <travier> https://src.fedoraproject.org/rpms/linux-firmware/blob/rawhide/f/linux-firmware.spec#_23 17:02:42 <travier> It's a F37 change :) 17:03:12 <dustymabe> travier: isn't there a difference in a proposed change and an accepted change ? 17:03:29 <travier> it should be indeed 17:03:32 <jlebon> looks like it was withdrawn based on bcotton's edit here: https://fedoraproject.org/w/index.php?title=Changes/Linux_Firmware_Minimization&diff=649159&oldid=649137 17:03:32 <travier> it's really weird 17:04:05 <travier> in the spec the change is there and it impacts us as we don't ship recommends 17:04:17 <travier> so they did not do the full change, just a part of it 17:05:01 <travier> so it's indeed both not a F37 change and a F37 change :D 17:05:02 <dustymabe> yeah the change proposal talks about using a `Supplements metadata` to automatically install appropriate firmware based on the hardware of the machine 17:05:09 <travier> well, I'll see myself out :) 17:05:34 <dustymabe> i imagine that the all of the change wasn't done, but some pieces they thought were harmless were 17:05:44 <travier> π 17:05:59 <dustymabe> i.e. changing to recommends is pretty harmless for most users, but we're obviously doing something that most users aren't 17:06:28 <dustymabe> anyway :) - we're on top of it now 17:06:44 <dustymabe> and our "stream" model is working great at giving us clues about what's to come 17:07:05 <travier> +1 17:07:25 <dustymabe> for example: I would bet that not a lot of people use this firmware given it took til this late in the cycle 17:07:26 <jlebon> dustymabe: weird we didn't notice it when we moved testing-devel to f37 17:07:33 <dustymabe> though I guess that could just mean not a lot of people try `next` 17:07:56 <jlebon> it is in the git diff indeed of https://github.com/coreos/fedora-coreos-config/commit/08ace84d0104661970047642dc49c91bc764943e 17:08:10 <dustymabe> we didn't move testing devel over to f37 until after `testing` had shipped 17:08:55 <dustymabe> https://github.com/coreos/fedora-coreos-config/pull/2083#issuecomment-1312052842 17:09:20 <dustymabe> summary: we'll do better next time :) 17:09:42 <dustymabe> hopefully we aren't migrating our internal build systems to a whole new model next release too 17:10:29 <dustymabe> anything else for open floor? 17:10:37 <jlebon> dustymabe: right, not arguing that we should've caught this for testing or next. it probably came in through bump-lockfiles which we don't inspect really. for testing-devel, we could've had CI that detected this in that PR. 17:11:00 <jlebon> though really even for bump-lockfile, one thing I've wanted to do was defer to PR the result if the pkgset changed 17:11:20 <jlebon> i think there's an issue somewhere for that 17:11:29 <dustymabe> yeah that would be nice 17:11:39 <anthr76[m]> I have something about native containers.. 17:11:41 <dustymabe> but I think where we really want to catch it is in rawhide 17:12:00 <jlebon> dustymabe: +1 indeed 17:12:13 <dustymabe> but I think that would imply using lockfiles for rawhide too, which we could do 17:12:16 <dustymabe> anthr76[m]: go for it! 17:12:28 <jlebon> we could compare against the previous rawhide build 17:13:48 <anthr76[m]> I just wanted a sanity check here before I open an issue (I suppose against ignition?)... (full message at <https://libera.ems.host/_matrix/media/v3/download/libera.chat/1881daf39cfb1145c7a770cd90ae27e12810fffc>) 17:14:14 * dustymabe will copy the rest of that here so the meeting log will have it 17:14:21 <dustymabe> I'm building a derivation like so: https://github.com/anthr76/infra/blob/de2d532938eb168be6672ef035d58a9ccec611ab/armature/prod/fcos-layers/k8s-node/Dockerfile#L16-L19 17:14:26 <dustymabe> Basically applying this big butane file. https://github.com/anthr76/infra/blob/main/armature/prod/fcos-layers/k8s-node/config.bu.yaml 17:14:31 <dustymabe> Though, when rebasing units that are enabled aren't actually getting enabled. For example: https://github.com/anthr76/infra/blob/de2d532938eb168be6672ef035d58a9ccec611ab/armature/prod/fcos-layers/k8s-node/config.bu.yaml#L159 17:14:35 <anthr76[m]> Sorry :) how does that show on IRC? 17:14:35 <dustymabe> I assume the symlink is getting lost somewhere.. 17:14:50 <dustymabe> anthr76[m]: matrix posts a link to the full message 17:14:57 <anthr76[m]> Is it more preferred to attend meetings on IRC? 17:15:08 <dustymabe> not really, i'm happy either way 17:15:23 <jlebon> anthr76[m]: yeah, i would file an issue 17:15:23 <dustymabe> i just wanted the full message to be in the meeting log 17:15:46 <jlebon> might be some interaction between layering and systemd's firstboot preset logic 17:16:20 <jlebon> or an issue with ignition-liveapply 17:17:43 <anthr76[m]> ack'd will do. It's simple to reproduce in a vm, and you can probably even reproduce it in a container with enough fiddling 17:18:34 <dustymabe> any other topics for open floor? 17:18:46 <jlebon> short-term hack, you can `systemctl enable` the units directly in your Dontainerfile 17:18:51 <jlebon> whoops 17:18:54 <jlebon> Dockerfile* 17:19:12 <jlebon> had started to type Containerfile and then saw it was a Dockerfile 17:19:51 <anthr76[m]> I do believe I tried that at some point and it was still no good. I like to try to actually make the symlink to multi-user.target 17:19:55 <dustymabe> and thus Dontainerfile was born π 17:20:18 <anthr76[m]> I'm all for de-dockering everything but it's really hard :P 17:20:40 <dustymabe> any other topics here? 17:20:48 <anthr76[m]> I gave up on Containerfile when I had to hack renovatebot around it all the time :/ and instructing LSPs and what not 17:20:54 <anthr76[m]> Good on my side 17:21:06 * dustymabe closes out the meeting in 60s 17:21:22 <fifofonix> thanks 17:21:23 <jlebon> anthr76[m]: interesting. add that bit to the issue too. `systemctl enable` is used in some of the the coreos-layering-examples 17:22:04 <anthr76[m]> Sure. I will double check before hand and post it up in about a hour 17:22:11 <dustymabe> #endmeeting