16:30:41 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:41 <zodbot> Meeting started Wed Oct 20 16:30:41 2021 UTC.
16:30:41 <zodbot> This meeting is logged and archived in a public location.
16:30:41 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:41 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:41 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:45 <dustymabe> #topic roll call
16:30:48 <dustymabe> .hi
16:30:49 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:05 <skunkerk> .hello sohank2602
16:31:06 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:31:10 <jlebon> .hello2
16:31:11 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:20 <fifofonix> .hello2
16:31:21 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com>
16:31:30 <ueno> .hi
16:31:33 <zodbot> ueno: ueno 'Daiki Ueno' <dueno@redhat.com>
16:31:48 <scorreia> .hi
16:31:49 <zodbot> scorreia: scorreia 'Sergio Correia' <scorreia@redhat.com>
16:31:51 <jaimelm> .hello2
16:31:52 <zodbot> jaimelm: jaimelm 'Jaime Magiera' <jaimelm@umich.edu>
16:31:59 <travier> .hello siosm
16:32:00 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:32:06 <ravanelli> .hi
16:32:07 <zodbot> ravanelli: ravanelli 'Renata Andrade Matos Ravanelii' <renata.ravanelli@gmail.com>
16:32:13 <walters> .hi
16:32:14 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:32:40 <jlebon> #chair fifofonix ueno scorreia jaimelm travier ravanelli walters
16:33:03 <lucab> .hi
16:33:04 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com>
16:33:05 <bgilbert> .hi
16:33:06 <jlebon> .ping
16:33:08 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:33:11 <zodbot> pong
16:33:22 <jlebon> #chair lucab bgilbert
16:33:23 <miabbott> .hello miabbott
16:33:24 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:33:41 <jlebon> hmm, weird not sure why it's not responding to the #chair's
16:33:57 <jlebon> #chair miabbott
16:34:12 <jlebon> ohhh. dustymabe do you have to #chair me first?
16:34:15 <dustymabe> #chair dustymabe
16:34:15 <zodbot> Current chairs: dustymabe
16:34:34 <dustymabe> .chair jlebon
16:34:35 <zodbot> jlebon is seated in a chair with a nice view of a placid lake, unsuspecting that another chair is about to be slammed into them.
16:34:48 <dustymabe> #chair jlebon
16:34:48 <zodbot> Current chairs: dustymabe jlebon
16:34:53 <jlebon> #chair fifofonix ueno scorreia jaimelm travier ravanelli walters
16:34:53 <zodbot> Current chairs: dustymabe fifofonix jaimelm jlebon ravanelli scorreia travier ueno walters
16:34:55 <jlebon> #chair lucab bgilbert
16:34:55 <zodbot> Current chairs: bgilbert dustymabe fifofonix jaimelm jlebon lucab ravanelli scorreia travier ueno walters
16:34:58 <jlebon> #chair miabbott
16:34:58 <zodbot> Current chairs: bgilbert dustymabe fifofonix jaimelm jlebon lucab miabbott ravanelli scorreia travier ueno walters
16:35:14 <miabbott> 🎉
16:35:20 <mnguyen> .hello mnguyen
16:35:20 <zodbot> mnguyen: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:35:30 <miabbott> #chair mnguyen
16:35:30 <zodbot> Current chairs: bgilbert dustymabe fifofonix jaimelm jlebon lucab miabbott mnguyen ravanelli scorreia travier ueno walters
16:35:35 <jaimelm> Are these stackable, metal chairs?
16:35:44 <dustymabe> nice turnout
16:35:47 * jmarrero watches in suspense
16:36:01 <dustymabe> jaimelm: i'm imagining the metal chairs from WWF
16:36:13 <jaimelm> right, exactly
16:36:25 <lucab> last meeting we ran out of time before getting to https://github.com/coreos/fedora-coreos-tracker/issues/982, so let's try to have it near the top of the stack today
16:36:28 <saqali> .hi
16:36:29 <zodbot> saqali: saqali 'Saqib Ali' <saqali@redhat.com>
16:36:39 <dustymabe> lucab: yep - it's at the top of the agenda
16:36:55 <lucab> danke :)
16:37:02 <jlebon> #topic Action items from last meeting
16:37:23 <jlebon> * dustymabe to write docs for rpi4 including updating eeprom
16:37:24 <jlebon> * bgilbert to write docs for switching to FAT32
16:37:47 <dustymabe> re-action for me :)
16:37:56 <jlebon> #action dustymabe to write docs for rpi4 including updating eeprom
16:38:07 <davdunc> .hello2
16:38:08 <zodbot> davdunc: davdunc 'David Duncan' <davdunc@amazon.com>
16:38:17 <dustymabe> luckily they released a new version of the firmware so I can finish and publish the docs now
16:38:36 <jlebon> #info bgilbert filed https://github.com/coreos/fedora-coreos-docs/issues/326 for switching to FAT32
16:38:41 <miabbott> #chair davdunc
16:38:41 <zodbot> Current chairs: bgilbert davdunc dustymabe fifofonix jaimelm jlebon lucab miabbott mnguyen ravanelli scorreia travier ueno walters
16:38:54 <travier> +1 lucab, let's start with the Keylime discussion
16:39:20 <jlebon> bgilbert: were you planning to write up the docs too?
16:39:27 <bgilbert> yup, it's on my list
16:39:38 <jlebon> ok cool. since there's an issue I won't re-action it
16:39:43 <jlebon> ok, let's move on!
16:39:44 <bgilbert> +1
16:39:55 <jlebon> #topic New Package Request: rust-keylime_agent
16:39:59 <jlebon> #link https://github.com/coreos/fedora-coreos-tracker/issues/982
16:40:25 <jlebon> who wants to introduce this?
16:40:55 <ueno> I can: it's an attester program of the keylime stack, which verifies that the running system in a good state, against remote verifier
16:42:07 <ueno> using TPM and IMA - I guess it would be particularly useful to monitor IoT devices
16:44:18 <jlebon> i forgot a bit the context around keylime+OSTree, is the idea that only files in /usr will be measured?
16:45:39 <travier> It's a bit complex because there are two use cases at play here
16:46:11 <travier> The first one is remote attestation at boot time. The second one is integrity monitoring happening later
16:46:14 <travier> https://keylime-docs.readthedocs.io/en/latest/user_guide.html
16:46:47 <travier> AFAIK, there is nothing preventing keylime from running from a privileged container with a lot of access to the filesystem and devices, etc.
16:46:56 <lucab> so reasonably this works only after 1) the system is normally booted (i.e. outside of initramfs), and 2) with the network fully up, correct?
16:47:48 <travier> the main question is more about how early do we want to be able to perform an attestation against a remote server. If the answer is before we pull any container image, then we need this agent on the node, otherwise we can use a container image
16:48:17 <travier> AFAIK yes, this is post initramfs and network up
16:49:38 <jlebon> worth noting things might've already been fetched over the network already (notably Ignition config and remote resources within the config)
16:50:47 <travier> Let me rephrase/clarify that as it's not about fetching but potential authorization: a setup could require all container images to be fetched from a given registry that would only allow attested node to fetch images
16:51:21 <travier> So you could have a node be able to fetch images from the cluster registry only after it has been attested as good
16:52:04 <travier> Of course the direct workaround here is to host this specific container in a public registry
16:52:51 <travier> The ignition config hash could also be included in the attestation too
16:53:19 <jlebon> travier: good point
16:53:38 <travier> ueno, scorreia I am making sense or is this not how keylime works right now?
16:54:06 <ueno> travier, that makes sense to me
16:55:30 <walters> i don't quite know why someone would want to gate something as basic as fetching container images on attestation though
16:56:11 * jlebon has to go to another meeting, but bgilbert will take over
16:56:47 <travier> Well, we could have a feature that would prevents a node from doing anything in a cluster before it attest that it is in a good state
16:57:20 <lucab> as it doesn't seem to be involved in bootstrapping the OS, it sounds like it could be easily layered in possibly?
16:57:23 <travier> This was a feature in CoreOS Tectonic
16:57:25 <jaimelm> interesting
16:59:08 <dustymabe> lucab: yeah that's what I'm wondering
16:59:28 <lucab> well but it wasn't gating pulling all containers
16:59:28 <walters> right, Tectonic replaced the kubelet node join with attestation, but at least OpenShift today hard requires pulling container images before kubelet runs today, so any attempt to replicate that would require the ability to pull at least a subset of images
17:00:17 <walters> anyways I think my bottom line here is I'm not hard opposed but I haven't seen any attempt to try to containerize, and I think that should be the default first path
17:00:44 <lucab> also because each container image has different authZ policies, so the base infra/bootstrapping ones are more freely pull-able
17:02:17 <travier> true, we should try container first and include it only if we find a case where this does not work
17:02:38 <bgilbert> anyone want to do a #proposed?
17:03:04 <travier> and yes, the only case I can come up where we need this on the host is easily worked around
17:04:00 <travier> I might be missing some use cases)
17:04:02 <travier> (*
17:06:13 <travier> #proposed: We do not include the keylime agent in the host for now and will investigate if a container based deployment could satisfy the use cases.
17:06:40 <dustymabe> who is going to investigate?
17:06:44 <bgilbert> dustymabe: +1
17:07:01 <dustymabe> we know where those unassigned action items go
17:07:02 <jaimelm> travier: ack
17:07:23 <bgilbert> in particular, are we planning to handle that in the FCOS WG?
17:07:36 <travier> Well, it's on my todo list to work on Keylime but I have not been able to get to it for now
17:08:13 * jdoss waves
17:08:50 <dustymabe> is it something we could ask the requestors to investigate?
17:09:00 <dustymabe> maybe in collaboration with travier
17:09:03 <travier> sorry ueno and scorreia as I had implied that we could include it before and we are going forward with another option now
17:09:24 <scorreia> dustymabe: yeah, we could investigate in collaboration with travier
17:09:24 <ueno> travier, np, ok, it seems worth trying :-)
17:09:26 <lucab> personally, I'd say "we believe this to fit better in a container. If that path does not get explored upstream, users can still layer in the RPM"
17:09:42 <travier> +1 lucab
17:10:01 <dustymabe> want to do a new #proposed ?
17:10:10 <lucab> *by upstream/community
17:10:38 <lucab> #proposed we believe this to fit better in a container. If that path does not get explored by upstream/community, users can still layer in the RPM
17:11:53 <dustymabe> ack - would be nice to understand this use case a bit more (maybe even add some docs?)
17:11:59 <lucab> (to be clear, we may be wrong and this may be impossible to containerize)
17:12:00 <jaimelm> ack
17:12:03 <bgilbert> +1
17:12:39 <dustymabe> ueno: scorreia: would you be able to investigate containerization as an option (along with travier) - package layering as a fallback - and maybe add some docs
17:12:53 <dustymabe> would be nice to have our users use it, but they won't if they don't know it exists or how
17:13:22 <ueno> dustymabe, sure
17:13:53 <bgilbert> walters, +/-1?
17:14:00 <travier> +1
17:14:03 <miabbott> +1
17:14:29 <travier> We can revisit if it turns out some things can not work from inside a container
17:14:45 <bgilbert> #agreed We believe this to fit better in a container. If that path does not get explored by upstream/community, users can still layer in the RPM.
17:14:57 <walters> (sorry I got context switched)  but skimming back +1
17:15:07 <bgilbert> looking for a topic that'll fit in 10 minutes
17:15:15 <bgilbert> dustymabe, NIC renaming udev rules maybe?
17:15:51 <dustymabe> WFM
17:16:01 <dustymabe> might be good to get an update on the final F35 change
17:16:02 <bgilbert> #topic forwarding NIC renaming udev rules from the initramfs
17:16:07 <bgilbert> #undo
17:16:07 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x7f25703809b0>
17:16:14 <bgilbert> #topic F35: CHANGE: More flexible use of SSSD fast cache for local users
17:16:20 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/875
17:16:45 <bgilbert> travier, looks like you're the assignee
17:17:27 <travier> Still working on that one. Hopefully will close it this week. Nothing blocking us so far
17:17:30 <bgilbert> +1
17:18:09 <bgilbert> #topic forwarding NIC renaming udev rules from the initramfs
17:18:14 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/553
17:18:26 <bgilbert> dustymabe?
17:18:38 <dustymabe> yep
17:18:45 <dustymabe> OK for this one, we had a partner engineer working on OpenShift that hit this. They are seeing very inconsistent NIC naming with some of the new 5G ethernet cards they are using so they want to rename NICs based on MAC address and they are hitting on this issue.
17:19:27 <dustymabe> right now they are going to have to create an ignition config per node to get the NIC renaming to work, but they think it's worth it because of how bad the NIC naming consistency is for them
17:20:05 <dustymabe> the one trip up is if doing an install via PXE using the kargs then they won't have anything in that install boot that puts in place the NIC renaming in the real root
17:20:22 <dustymabe> so currently:
17:20:24 <dustymabe> 1. set kargs
17:20:39 <dustymabe> 2. install boot ignition config (does this mean they can't use kargs to do the install too)?
17:20:54 <dustymabe> 3. first boot igition config
17:21:05 <bgilbert> it sounds like the root cause is that stable NIC naming is broken for their case?
17:21:30 <dustymabe> bgilbert: yes, it's the cause for them wanting to use NIC renaming
17:21:43 <dustymabe> but you could have other reasons for wanting NIC renaming
17:22:34 <bgilbert> well, sure, but do we?
17:22:42 <bgilbert> it looks like the last decision was "don't pursue for now"
17:22:43 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/553#issuecomment-658949868
17:23:08 <dustymabe> "do we"? I think that's up to the user if they want to rename their NICs or not.
17:23:47 <bgilbert> asking because, before this case came up, the bug was closed with "we'll reopen if we get a lot of user input"
17:23:48 <bgilbert> https://github.com/coreos/fedora-coreos-tracker/issues/553#issuecomment-662554147
17:24:11 <bgilbert> and absent that, I'm wondering whether we should just fix the bug
17:24:36 <dustymabe> right, here's user input - and also one more data point I hadn't considered before, which is the PXE install extra ignition config needed
17:24:52 <lucab> it sounds like a systemd/udev RFE where there should be some generator inspecting the cmdline and generating the udev rules?
17:24:55 <dustymabe> bgilbert: yeah, I don't doubt we should fix the bug, not ure it's well defined, we could ask
17:25:32 <dustymabe> lucab: it's kind of a lost feature in the transition from legacy network stack to NM IIRC
17:26:20 <dustymabe> let me see if I can find the link
17:26:53 <dustymabe> I think it was https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/479
17:27:17 <dustymabe> but also that's approximate, maybe not quite the exact same thing
17:27:45 <dustymabe> so let me summarize
17:28:05 <dustymabe> I think the new user input does change things slightly, mostly because the PXE install case was not considered before
17:28:29 <dustymabe> and I also believe that one could have reason to rename their NICs outside of unreliable NIC naming
17:28:33 <dustymabe> EOM
17:29:43 <bgilbert> dustymabe, what do you think we should do?
17:29:45 <bgilbert> also time check
17:30:16 <dustymabe> bgilbert: it would be really nice if there was something at another level taking care of the renaming, but absent that we could copy forward the generated udev rule from the initramfs into the real root
17:30:33 <dustymabe> so it would be part of the "propagate initramfs networking" logic
17:30:58 <bgilbert> right
17:31:07 <dustymabe> which is fragile, but it's the best we've got
17:31:26 <bgilbert> should we try to drive toward a conclusion in the next few minutes, or defer?
17:32:09 <bgilbert> dustymabe ^
17:32:13 <dustymabe> probably not much else to discuss I don't think, unless people have questions
17:32:18 <bgilbert> +1
17:32:25 <bgilbert> discussion?
17:32:49 <travier> I'm wondering if we could have a persist naming karg or option in Ignition
17:33:16 <dustymabe> i could put my proposal in the ticket and allow for silent acceptance over the next week?
17:33:26 <dustymabe> and invite more discussion there
17:33:27 <bgilbert> dustymabe, wfm
17:33:46 <dustymabe> travier: preferrably not in Ignition itself.. these people are trying to drive networking via kargs
17:33:53 <lucab> https://www.freedesktop.org/software/systemd/man/systemd-network-generator.service.html#Kernel%20command%20line%20options
17:34:02 <dustymabe> if we had what you propose we'd still need the install boot ignition config
17:34:30 <dustymabe> lucab: interesting
17:34:53 <dustymabe> wonder if we could add a feature to tell that network generator pieces to ignore
17:34:56 <lucab> just found right now, I've no idea how link units play with NM
17:35:16 <dustymabe> link units work great - it's an alternative that's documented in our docs
17:35:23 <dustymabe> ok i'll add a summary to the ticket
17:35:30 <dustymabe> thanks bgilbert
17:35:31 <bgilbert> +1
17:35:36 <travier> dustymabe: thanks
17:35:38 <bgilbert> #topic open floor
17:35:43 <bgilbert> #info devconf.cz CFP closes this week
17:35:48 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/988
17:36:01 <bgilbert> #info we really need some volunteers for CI flakes. One example: https://github.com/coreos/fedora-coreos-tracker/issues/942
17:36:10 <dustymabe> please submit a talk if you're interested to devconf.cz
17:36:13 <lucab> unless it errors out on $something, I think it will just produce some unused stuff in /run
17:37:05 <bgilbert> will close in 30s
17:37:36 <bgilbert> #endmeeting