2024-05-22 16:31:46 <@jlebon:fedora.im> !startmeeting fedora_coreos_meeting 2024-05-22 16:31:49 <@meetbot:fedora.im> Meeting started at 2024-05-22 16:31:46 UTC 2024-05-22 16:31:49 <@meetbot:fedora.im> The Meeting name is 'fedora_coreos_meeting' 2024-05-22 16:31:52 <@siosm:matrix.org> !hi 2024-05-22 16:31:54 <@jlebon:fedora.im> !topic roll call 2024-05-22 16:31:56 <@zodbot:fedora.im> Timothรฉe Ravier (siosm) - he / him / his 2024-05-22 16:32:02 <@marmijo:fedora.im> !hi 2024-05-22 16:32:06 <@zodbot:fedora.im> Michael Armijo (marmijo) 2024-05-22 16:32:22 <@hricky:fedora.im> !hi 2024-05-22 16:32:24 <@zodbot:fedora.im> Hristo Marinov (hricky) - he / him / his 2024-05-22 16:32:27 <@ravanelli:matrix.org> !hi ravanelli 2024-05-22 16:32:31 <@zodbot:fedora.im> Renata Ravanelli (ravanelli) 2024-05-22 16:32:37 <@dustymabe:matrix.org> !hi 2024-05-22 16:32:40 <@zodbot:fedora.im> Dusty Mabe (dustymabe) - he / him / his 2024-05-22 16:33:25 <@jbrooks:matrix.org> !hi jasonbrooks 2024-05-22 16:33:29 <@zodbot:fedora.im> Jason Brooks (jasonbrooks) - he / him / his 2024-05-22 16:33:39 <@jbtrystram:matrix.org> !hi 2024-05-22 16:33:43 <@zodbot:fedora.im> Jean-Baptiste Trystram (jbtrystram) - he / him / his 2024-05-22 16:34:04 <@jlebon:fedora.im> let's wait another minute 2024-05-22 16:35:54 <@jlebon:fedora.im> ok, let's get started! 2024-05-22 16:35:55 <@jlebon:fedora.im> !topic Action items from last meeting 2024-05-22 16:36:25 <@jlebon:fedora.im> ACTION: All to think about the python discussion ACTION: ravanelli to contact the libteam maintainers to figure out if they intend to drop the package from fedora 2024-05-22 16:36:25 <@aaradhak:matrix.org> !hi aaradhak 2024-05-22 16:36:56 <@blackwell:fedora.im> !hi 2024-05-22 16:37:02 <@jlebon:fedora.im> i think a few of us have done the former. Renata Ravanelli did you do the second one yet? 2024-05-22 16:37:16 <@zodbot:fedora.im> Aashish Radhakrishnan (aaradhak) 2024-05-22 16:37:17 <@zodbot:fedora.im> Jason Blackwell (blackwell) 2024-05-22 16:37:41 <@dogphilosopher:fedora.im> !hi 2024-05-22 16:37:45 <@zodbot:fedora.im> Noel Miller (dogphilosopher) 2024-05-22 16:37:57 <@jlebon:fedora.im> dogphilosopher: ๐ 2024-05-22 16:38:55 <@ravanelli:matrix.org> Jonathan Lebon: sorry, not yet. I will 2024-05-22 16:39:15 <@jlebon:fedora.im> Renata Ravanelli: ack, sounds good! i'll re-action it then 2024-05-22 16:39:22 <@jlebon:fedora.im> !action ravanelli to contact the libteam maintainers to figure out if they intend to drop the package from fedora 2024-05-22 16:39:31 <@jlebon:fedora.im> ok, let's move to topics 2024-05-22 16:39:40 <@jlebon:fedora.im> !topic Consider specializing mount units on first boot, after partitioning 2024-05-22 16:39:50 <@jlebon:fedora.im> !link https://github.com/coreos/fedora-coreos-tracker/issues/1732 2024-05-22 16:40:00 <@jlebon:fedora.im> travier: want to introduce this one? 2024-05-22 16:40:21 <@siosm:matrix.org> Shouldn't we do the Python discussion first? 2024-05-22 16:40:31 <@siosm:matrix.org> But I can speak about this one 2024-05-22 16:40:43 <@siosm:matrix.org> But I can also speak about this one 2024-05-22 16:42:03 <@siosm:matrix.org> The idea is that it's non-intuitive for users to use stable names for disks for partionning using Ignition and for a good reason: as soon as you do it, your config is not generic anymore 2024-05-22 16:42:35 <@siosm:matrix.org> moreover, you usually don't know that you need it until you ran into issues and it's usually too late then 2024-05-22 16:43:03 <@jlebon:fedora.im> i'm fine with swapping if you'd like. that's the order it was in in the checklist :) 2024-05-22 16:43:26 <@siosm:matrix.org> So instead of forcing everyone to have node specific configs, I think we should have something, maybe in Ignition, "specialize" the mount units to reference stable names 2024-05-22 16:44:09 <@jlebon:fedora.im> so i think there are two separate issues at play here 2024-05-22 16:44:47 <@siosm:matrix.org> we could run that logic on existing nodes as well as that would make the setup more robust to future name changes 2024-05-22 16:45:06 <@jlebon:fedora.im> the first is that it's hard to reference block devices in a reliable way in the storage.disks and storage.filesystems sections. the second is that your mount unit can be written in a way that breaks between boots/in an upgrade. 2024-05-22 16:46:11 <@jlebon:fedora.im> the second issue I would argue is actually mostly a documentation problem 2024-05-22 16:46:38 <@siosm:matrix.org> to me it's all interconnected 2024-05-22 16:46:48 <@jlebon:fedora.im> not just of course. e.g. i think with_mount_units could be made smarter 2024-05-22 16:47:00 <@siosm:matrix.org> you are not going to write the mount units manually; they are written by ignition 2024-05-22 16:47:40 <@jlebon:fedora.im> right. so i think one improvement we could make here is to add a lint for this in butane 2024-05-22 16:48:42 <@dustymabe:matrix.org> today I think with_mount_units is butane sugar, right? 2024-05-22 16:48:54 <@jlebon:fedora.im> first, if you use `with_mount_unit: true`, the generated unit should use by-label/ instead, and warn if a label is not provided. second, if you define a mount unit manually, we could check the What 2024-05-22 16:49:03 <@jlebon:fedora.im> the second could live in Ignition actually 2024-05-22 16:49:03 <@siosm:matrix.org> I think so, which is why this would have to live elsewhere 2024-05-22 16:50:07 <@siosm:matrix.org> The label helps, but it does not resolves things for existing nodes 2024-05-22 16:50:23 <@siosm:matrix.org> I'm +1 for adding docs about using labels 2024-05-22 16:50:24 <@dustymabe:matrix.org> what is the problem for existing nodes? 2024-05-22 16:50:55 <@siosm:matrix.org> Say you do a major unamed OS update from 8 to 9 and the disks are not discovered in the same order anymore 2024-05-22 16:51:57 <@dustymabe:matrix.org> and this is because people used unstable names in their units? 2024-05-22 16:52:05 <@jlebon:fedora.im> note this isn't just a CoreOS problem though. we should lean on leapp there where I could see this "specialization" live 2024-05-22 16:52:22 <@dustymabe:matrix.org> i.e. `/dev/sda6` versus `/dev/disk/by-label/my-data` ? 2024-05-22 16:52:33 <@siosm:matrix.org> unfortunately the names are stable until they aren't 2024-05-22 16:52:37 <@jlebon:fedora.im> dustymabe: yeah, exactly 2024-05-22 16:52:37 <@siosm:matrix.org> yes 2024-05-22 16:53:04 <@siosm:matrix.org> If you use `/dev/disk/by-label/my-data` then you immediately have a per-node config 2024-05-22 16:53:18 <@siosm:matrix.org> for the partitionning part 2024-05-22 16:53:27 <@jlebon:fedora.im> travier: how so? labels are generic. e.g. 'etcd-data' 2024-05-22 16:53:36 <@jlebon:fedora.im> the device backing them may not be 2024-05-22 16:53:36 <@dustymabe:matrix.org> hmm. part of me feels like we should fix it - hmm or rather, maybe we should do CLHM and tell the user how to fix and maybe even give them a migration script 2024-05-22 16:54:13 <@dustymabe:matrix.org> i'm not sure how I feel about automagically changing peoples mount units - when there might be some complicated units (i.e. we can't test everything and I don't think we should try) 2024-05-22 16:54:46 <@siosm:matrix.org> agree this is risky 2024-05-22 16:55:03 <@dustymabe:matrix.org> note I feel like this warning for the upcoming wifi removal has worked out well: ``` [dustymabe@media ~]$ ssh core@192.168.100.1 Fedora CoreOS 40.20240504.2.0 ########################################################################## WARNING: NetworkManager-wifi is a requested layered package on this system, but no Wi-Fi drivers are requested. The Wi-Fi drivers will no longer be included by default in the future. More context and remediation steps are available in the following FAQ entry: https://docs.fedoraproject.org/en-US/fedora-coreos/faq/#wifi-fw To disable this warning, use: sudo systemctl disable coreos-check-wireless-firmwares.service ########################################################################## Tracker: https://github.com/coreos/fedora-coreos-tracker Discuss: https://discussion.fedoraproject.org/tag/coreos ``` 2024-05-22 16:55:32 <@siosm:matrix.org> but the idea would be to provide an option for those that want to "specialize" existing installations and automatically specialize new installations 2024-05-22 16:55:37 <@dustymabe:matrix.org> so basically we can just let people know when they have units that might be unstable and then they can either take action or disable the warning 2024-05-22 16:55:54 <@dustymabe:matrix.org> right, we can give them instructions for how to improve their situation 2024-05-22 16:57:02 <@siosm:matrix.org> I also agree that working with leapp folks would be good, but this is RHEL only afaik 2024-05-22 16:57:05 <@jlebon:fedora.im> MOTD seems like it'd be a good fit here indeed 2024-05-22 16:57:11 <@apiaseck:matrix.org> Not sure if I understand this correctly but could we not mount the volumes in etc/fstab based on the uuid's? 2024-05-22 16:57:50 <@siosm:matrix.org> apiaseck: yes, that's the idea, but we need to make it easier for users to set this up 2024-05-22 16:57:57 <@dustymabe:matrix.org> travier: Jonathan Lebon, but I agree for newly provisioned systems it would be good to create mount units with more correct names if we can too 2024-05-22 16:59:26 <@siosm:matrix.org> (I don't think we should solve this / take a decision in this meeting but wanted to start the discussion) 2024-05-22 17:00:02 <@jlebon:fedora.im> what do folks think about an MOTD for this at least? 2024-05-22 17:00:48 <@jlebon:fedora.im> i can file an issue against butane also to see how we can improve things there 2024-05-22 17:01:17 <@siosm:matrix.org> I'm OK with that. We would also need actionable docs 2024-05-22 17:01:57 <@jlebon:fedora.im> yeah, we basically should audit all our docs for this IMO 2024-05-22 17:02:28 <@jlebon:fedora.im> and e.g. a section or at least a NOTE box on the storage page 2024-05-22 17:03:02 <@jlebon:fedora.im> travier: want to file a docs issue for this? 2024-05-22 17:03:16 <@siosm:matrix.org> Sure, will do 2024-05-22 17:03:24 <@jlebon:fedora.im> !action jlebon to file a butane issue to discuss improvements around device names 2024-05-22 17:03:48 <@jlebon:fedora.im> !action travier to file a docs issue to audit storage docs and add some details re. device naming 2024-05-22 17:04:56 <@jlebon:fedora.im> i've thought about something similar to your idea about selectors before, but using udev rules to create more symlinks 2024-05-22 17:05:14 <@jlebon:fedora.im> there might be something there worth exploring too 2024-05-22 17:05:30 <@siosm:matrix.org> ๐ 2024-05-22 17:05:40 <@dustymabe:matrix.org> i'm not sure how fancy we should get :) 2024-05-22 17:05:48 <@jlebon:fedora.im> ok, move on to the next topic? 2024-05-22 17:06:15 <@jlebon:fedora.im> !topic revisit python discussion 2024-05-22 17:06:21 <@jlebon:fedora.im> !link https://github.com/coreos/fedora-coreos-tracker/issues/1730 2024-05-22 17:06:59 <@walters:fedora.im> (tangentially related to this one) was the bootc one on the agenda for today? 2024-05-22 17:07:27 <@jlebon:fedora.im> Colin Walters: it's next in the list :) 2024-05-22 17:07:33 <@jlebon:fedora.im> (https://github.com/coreos/fcos-meeting-action/issues/88#issue-2310854304) 2024-05-22 17:07:53 <@jlebon:fedora.im> do folks want to discuss that one first instead? 2024-05-22 17:09:42 <@siosm:matrix.org> If folks are there for the bootc one we could move to it? 2024-05-22 17:10:06 <@jlebon:fedora.im> WFM 2024-05-22 17:10:17 <@jlebon:fedora.im> !topic Roadmap to Fedora Bootable Containers 2024-05-22 17:10:21 <@jlebon:fedora.im> !link https://github.com/coreos/fedora-coreos-tracker/issues/1726 2024-05-22 17:10:48 <@walters:fedora.im> I'm happy to talk about this one if it's ok, though travier wrote the issue 2024-05-22 17:11:01 <@siosm:matrix.org> go ahead! 2024-05-22 17:11:38 <@walters:fedora.im> OK so I think as most but not all know, I used to be on this team working on coreos and have for a while now been working on doing something different, but related that landed in RHEL in https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/using_image_mode_for_rhel_to_build_deploy_and_manage_operating_systems/index 2024-05-22 17:12:23 <@walters:fedora.im> There's a lot to say here but I think all the prior discussion around device naming etc. is a good example of a mature project discussion. However I also think CoreOS is well-defined and well maintained it is also a kind of "technological peak" 2024-05-22 17:12:44 <@walters:fedora.im> All the opinions the project has also constrain where it can go and what it can do 2024-05-22 17:13:20 <@walters:fedora.im> The fedora bootc project is all about basically saying "anywhere you can dnf, you can bootc" and I hope it feels literally "more fedora/rhel" in that aspect 2024-05-22 17:13:39 <@walters:fedora.im> And it's all about going all in on containers as the technical heart of things, the center of gravity 2024-05-22 17:13:58 <@walters:fedora.im> So this issue is about rebasing FCOS on bootc technologies; there's a lot of sub-details there 2024-05-22 17:14:48 <@walters:fedora.im> A lot of ways to slice this; one for example would be that we could cease maintaining FCOS rawhide separately (things like https://github.com/coreos/fedora-coreos-config/pull/3004 ) and always maintain bootc rawhide, and just base fcos on that 2024-05-22 17:15:03 <@walters:fedora.im> Getting to using container images on the wire for updates is a huge one 2024-05-22 17:15:38 <@walters:fedora.im> So I hope we can figure out how to make incremental progress on these things 2024-05-22 17:16:45 <@walters:fedora.im> It's important to note there are *very* fundamental differences in the fedora-bootc approach vs current FCOS - especially things like the "evergreen stream" model on top of fedora, that drives things like the python bits, etc. And the general assumption users start by deriving their own container images. 2024-05-22 17:17:14 <@walters:fedora.im> I stubbed out a table here https://docs.fedoraproject.org/en-US/bootc/fedora-coreos/ that could definitely use some expansion 2024-05-22 17:18:40 <@jlebon:fedora.im> right. i think another way to say that latter point is that FCOS is intended to be usable as is for the most part, while fedora-bootc very likely requires derivation. and that drives so much of the UX. 2024-05-22 17:18:41 <@walters:fedora.im> Overall I hope to finally achieve what I've been trying to do for years in having a shared "base image" between all the (current ostree, hopefully future container-based) fedora variants 2024-05-22 17:19:00 <@dustymabe:matrix.org> > And the general assumption users start by deriving their own container images. Yeah. I think this is where we differ 2024-05-22 17:20:29 <@walters:fedora.im> btw one interesting view into this is that I started by forking the FCOS docs as the bootc docs, and that was *very* enlightening as to the differences. Compare https://docs.fedoraproject.org/en-US/bootc/sysconfig-network-configuration/ with https://docs.fedoraproject.org/en-US/fedora-coreos/sysconfig-network-configuration/ 2024-05-22 17:21:13 <@walters:fedora.im> I mentioned this to dusty but I thought it was cool when I realized we can actually just document `podman run fedora-bootc nm-generator` as a way to run it (which we can with FCOS too! But the fcos docs don't think about fcos-as-container) 2024-05-22 17:21:37 <@walters:fedora.im> And another side thing here is a lot of the FCOS docs are "encode these files in butane" when it's WAY more ergonomic to just document a dockerfile `COPY` instruction 2024-05-22 17:21:46 <@jbrooks:matrix.org> I'm interested to know what benefits this group sees in the bootc approach, for instance, is the prospect of collaborating w/ other image-based Fedora variants attractive? Will it save FCOS time, etc 2024-05-22 17:22:29 <@jlebon:fedora.im> it'd be good to flesh out the CI story around the base images. i.e. how exactly do we share and contribute to versioned base images/image definitions. FCOS uses lockfiles right now which... definitely is a massive detail wrt the rest of Fedora both implementation-wise and conceptually 2024-05-22 17:24:19 <@jlebon:fedora.im> Jason Brooks: to answer your question more directly, i think i see a lot of potential benefits in sharing the workload to maintain stability 2024-05-22 17:24:24 <@siosm:matrix.org> Ideally we would integrate bootc into Fedora CI and release process to be able to gate package update on bootc image working. 2024-05-22 17:25:12 <@dustymabe:matrix.org> yes, travier that's the real value I see 2024-05-22 17:25:12 <@walters:fedora.im> one side note: a toplevel goal I have is that it's actually the fedora/centos bootc project, and we try aggressively to avoid letting docs, build system etc diverge between the two 2024-05-22 17:25:19 <@siosm:matrix.org> that would already be a massive benefit to FCOS that would help us move forward with getting closer to bootc 2024-05-22 17:25:46 <@dustymabe:matrix.org> can you elaborate on this? 2024-05-22 17:26:22 <@jlebon:fedora.im> travier: indeed. what i'd say is ideally we don't have lockfiles for the base images and sprint towards that gating CI instead 2024-05-22 17:26:38 <@walters:fedora.im> right now the docs describe *both* c9s and fedora:40, and we are sharing the image definitions pretty closely 2024-05-22 17:27:20 <@dustymabe:matrix.org> ok so you were just referring to c9s and fedora:40 bootc and not referring to FCOS there? 2024-05-22 17:27:39 <@walters:fedora.im> right 2024-05-22 17:28:28 <@conan_kudo:matrix.org> hey folks 2024-05-22 17:28:29 <@conan_kudo:matrix.org> !hi 2024-05-22 17:28:35 <@zodbot:fedora.im> Neal Gompa (ngompa) - he / him / his 2024-05-22 17:28:49 <@dustymabe:matrix.org> ๐ 2024-05-22 17:28:56 <@conan_kudo:matrix.org> if y'all don't mind, I've got a few questions here 2024-05-22 17:29:11 <@siosm:matrix.org> (Note we have 2 minutes left in this meeting) 2024-05-22 17:29:17 <@siosm:matrix.org> (Note: we have 2 minutes left in this meeting) 2024-05-22 17:29:20 <@conan_kudo:matrix.org> okay, then I'll make it quick 2024-05-22 17:30:24 <@walters:fedora.im> bigger picture we're obviously not going to change many things overnight; but we are spinning up more resources around this and so this will likely just become an ongoing merger/collaboration hopefully 2024-05-22 17:30:53 <@conan_kudo:matrix.org> 1. with the new "image mode"/bootc/etc. stack, from the Kinoite perspective, does this mean we need a new backend to manage system updates? 2. how compatible with existing image build tools is this new stack? 3. does it _have_ to result in read-only environments? 2024-05-22 17:31:03 <@jlebon:fedora.im> also note for everyone that there are fedora-bootc community meetings on Tuesdays and Thursdays: https://gitlab.com/fedora/bootc/tracker#meetings 2024-05-22 17:31:28 <@jlebon:fedora.im> Conan Kudo: those sound like questions more for ^ :) 2024-05-22 17:31:56 <@conan_kudo:matrix.org> sure, I can take it there 2024-05-22 17:32:15 <@siosm:matrix.org> yeah, probably better for tomorrow's meeting 2024-05-22 17:32:35 <@walters:fedora.im> agreed, but briefly 1) no, rpm-ostree will continue to work. 2) compat, but note a big goal of bootc (vs ostree) is to own installation; there's `bootc install to-filesystem` and we are building things up there 3) yes but you can make a transient writable overlay if you want 2024-05-22 17:33:13 <@siosm:matrix.org> 1) it will need one in the end 2024-05-22 17:33:26 <@siosm:matrix.org> 1. it will need one in the end (once we remove rpm-ostree) 2024-05-22 17:33:47 <@jlebon:fedora.im> ok, we're over time now. let's do a quick open floor before closing 2024-05-22 17:33:56 <@jlebon:fedora.im> !topic Open Floor 2024-05-22 17:34:05 <@jlebon:fedora.im> anything anyone wants to bring up for open floor? 2024-05-22 17:34:33 <@siosm:matrix.org> We'll talk about Python next week ๐ 2024-05-22 17:34:35 <@dustymabe:matrix.org> !info dusty is mostly AFK for the next few months while on leave (taking care of kids mostly) 2024-05-22 17:35:36 <@jlebon:fedora.im> alrighty, closing in 60s 2024-05-22 17:36:14 <@jbtrystram:matrix.org> Thanks Jonathan Lebon for running the meeting :) 2024-05-22 17:36:37 <@jlebon:fedora.im> !endmeeting