16:30:24 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:24 <zodbot> Meeting started Wed Apr 27 16:30:24 2022 UTC.
16:30:24 <zodbot> This meeting is logged and archived in a public location.
16:30:24 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:24 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:24 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:29 <dustymabe> #topic roll call
16:30:42 <dustymabe> .hi
16:30:43 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:49 <jbrooks> .hello jasonbrooks
16:30:50 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:30:53 <jlebon> .hello2
16:30:54 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:00 <walters> .hello2
16:31:01 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:31:10 <gursewak> .hi
16:31:11 <zodbot> gursewak: gursewak 'Gursewak Singh' <gurssing@redhat.com>
16:31:32 <miabbott> .hello miabbott
16:31:33 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:33:25 <dustymabe> #chair jbrooks jlebon walters gursewak miabbott
16:33:25 <zodbot> Current chairs: dustymabe gursewak jbrooks jlebon miabbott walters
16:33:50 * ffmancera says hello o/
16:34:59 <dustymabe> #chair mnguyen
16:34:59 <zodbot> Current chairs: dustymabe gursewak jbrooks jlebon miabbott mnguyen walters
16:35:07 <ffmancera> .hi
16:35:07 <zodbot> ffmancera: ffmancera 'Fernando F. Mancera' <ferferna@redhat.com>
16:35:13 <dustymabe> #chair ffmancera thaller
16:35:13 <zodbot> Current chairs: dustymabe ffmancera gursewak jbrooks jlebon miabbott mnguyen thaller walters
16:35:29 <thaller> .hi
16:35:30 <zodbot> thaller: thaller 'None' <thaller@redhat.com>
16:35:40 <saqali> .hi
16:35:41 <zodbot> saqali: saqali 'Saqib Ali' <saqali@redhat.com>
16:35:50 <aaradhak> .hi
16:35:51 <zodbot> aaradhak: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com>
16:36:15 <dustymabe> #chair saqali aaradhak
16:36:15 <zodbot> Current chairs: aaradhak dustymabe ffmancera gursewak jbrooks jlebon miabbott mnguyen saqali thaller walters
16:36:18 <dustymabe> ok let's get started
16:36:24 <dustymabe> #topic Action items from last meeting
16:36:33 <dustymabe> * jlebon to reach out to nmstate team to make sure they can attend a future meeting to discuss this ticket
16:36:45 <jaimelm> .hello2
16:36:45 <dustymabe> #info jlebon reached out and we have reps from the nmstate here with us today
16:36:46 <zodbot> jaimelm: jaimelm 'Jaime Magiera' <jaimelm@umich.edu>
16:36:52 <dustymabe> #chair jaimelm
16:36:52 <zodbot> Current chairs: aaradhak dustymabe ffmancera gursewak jaimelm jbrooks jlebon miabbott mnguyen saqali thaller walters
16:37:07 <dustymabe> the next action item was:
16:37:09 <dustymabe> * jaimelm and dustymabe to meet with the releng/infra team to talk about containers and where everything fits
16:37:13 <dustymabe> I have a few updates for that one
16:37:21 <dustymabe> #info jaimelm and dustymabe met with Fedora releng/infra to discuss forward strategy for Fedora container hosting. The goal is to get off of their own infra and onto quay.io (https://pagure.io/fedora-infrastructure/issue/10386), thought they have some open questions about if flatpaks can be hosted there.
16:37:38 <dustymabe> #info We are going to all work together over the next week to try to chase down any open questions and then move forward with putting our FCOS containers in quay.io if no blockers are found.
16:37:49 <dustymabe> #info This will lead Fedora's transition to quay.io by some months, but the releng/infra team feels it's best if we start there for new containers rather than go to the old infra right before a transition.
16:38:20 <lucab> .hi
16:38:22 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com>
16:38:24 <dustymabe> thank you jaimelm for kicking off that coordination
16:38:28 <dustymabe> #chair lucab
16:38:28 <zodbot> Current chairs: aaradhak dustymabe ffmancera gursewak jaimelm jbrooks jlebon lucab miabbott mnguyen saqali thaller walters
16:39:06 <walters> cool
16:39:11 <jlebon> jaimelm++
16:39:11 <zodbot> jlebon: Karma for jaimelm changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:39:58 <dustymabe> #topic New Package Request: nmstate-libs and nmstate
16:40:03 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1175
16:40:29 <dustymabe> welcome thaller and ffmancera - thank you for joining us for this topic
16:41:18 <dustymabe> we had some open questions from last week about how nmstate integrates with the host and how it intersects with existing NM configuration (i.e. NM keyfiles)
16:41:37 <walters> Offhand I am a weak +1
16:41:38 <ffmancera> Sure, we can answer these questions
16:41:53 <dustymabe> for example.. with FCOS today we try to configure the host up front with Ignition (runs in the initramfs)
16:42:15 <ravanelli> .hi
16:42:16 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:42:32 <dustymabe> In this scenario, people usually create files on the host using Ignition and then the machine comes up and applies the configuration
16:42:33 <thaller> hi all. nmstate has no own configuration, it talks to NetworkManager via the D-Bus API. For NM it's the same all all clients (w.r.t. keyfiles, etc)
16:43:04 <miabbott> #chair ravanelli
16:43:04 <zodbot> Current chairs: aaradhak dustymabe ffmancera gursewak jaimelm jbrooks jlebon lucab miabbott mnguyen ravanelli saqali thaller walters
16:43:22 <thaller> nmstate gets configured via a YAML file, but that is the input. It doesn't store that anyway, just translates that into NetworkManager terms.
16:44:10 <jlebon> thaller: do you mean there's no e.g. /etc/nmstate where the canonical YAML files are kept?
16:44:12 <thaller> There should be no conflict with configuration via files... aside, that nmstate could configure something different than what was before.  And races should be avoided, when two tools try to change something at the same time.
16:44:35 <walters> nmstate has no state? 😄
16:44:41 <thaller> jlebon, correct.   Well, the users of nmstate probably will get that YAML from somewhere.
16:44:44 <jlebon> nmstateless
16:45:29 <walters> this is really oriented towards live updates, right?
16:45:39 <dustymabe> hmm. so nmstate is only used for dynamic changes to an already running system?
16:47:13 <walters> the daemonset is just going to...basically copy the data from the CRD to the host and run the binary?
16:47:15 <thaller> dustymabe, nmstate a configuration API and translates to NM profiles (those get persisted). What makes it a bit different, is the API focuses more on devices (unlike NM's profiles), and that you configure several/all devices at once (in one YAML)
16:47:49 <jlebon> are there any plans to have nmstate ship a systemd service that does e.g. `nmstate apply /etc/path/to/dir.d/`?
16:48:26 <ffmancera> jlebon: No, there are not plans to do that
16:49:07 <jlebon> what's the usage model for this? e.g. users manually call `nmstate apply` from a git checkout?
16:49:11 <dustymabe> I think I'm a bit confused. According to the issue: "This request is for enabling day 1 and day 0 network configurations."
16:49:33 <dustymabe> in the context of OpenShift/Kubernetes I assume
16:49:47 <ffmancera> Yes, because Nmstate is able to generate nmconnection files. These files can distributed to the nodes so they are configured.
16:49:47 <thaller> nmstate is a one-shot command/library-call, that users will call whenever they want to (re)configure networking. there is no running service
16:49:49 <walters> so the MCO basically copies its own binary out to `/run` on the host; I assume it would also work just fine for this use case to have the daemonset image contain the nmstate binary and run it in the host context, particularly now that it's Rust
16:51:14 <dustymabe> walters: indeed
16:51:35 <jlebon> is the YAML -> nmconnection conversion completely stateless (i.e. could it be run upfront before provisioning the node) ?
16:51:48 <jlebon> s/stateless/independent of host state/
16:52:04 <ffmancera> Right now, OpenShift Assisted Installer and OpenShift agent based installation with ephemeral ISO use nmstate to convert the nmstate state YAML to NetworkManager keyfiles. They both have to use the container with Python, run a nmstatectl command and then distribute the keyfiles to the different nodes.
16:52:12 <ffmancera> jlebon: Yes! exactly.
16:52:36 <walters> seems like this could actually be run as part of a butane-like flow
16:52:57 <jlebon> ok interesting. so we don't actually have to ship this to leverage nmstate in the provisioning flow
16:53:02 <ffmancera> With the inclusion of nmstate into fcos, OCP or OKD would be able to have the nmstate config provided in ignition and the use nmstatectl to configure the node
16:53:26 <dustymabe> so is this kind of like an NM keyfile generator then?
16:53:36 <jlebon> a lot of similarities with the quadlet discussions here and tradeoffs
16:53:48 <dustymabe> ffmancera: that exact thing is what we were just asking about
16:54:11 <walters> ffmancera: hmmm...but wait, where would ignition write the files?  into some random place in `/etc` and then there's a systemd unit that runs it...on boot?
16:54:11 <dustymabe> but currently there are no plans to make a systemd service from nmstate team that applies an nmstate configuration
16:54:15 <jlebon> ffmancera: but it'd be up to us to ship that service that applies the YAML configs, right?
16:55:23 <ffmancera> dustymabe: it has two modes, it can generate keyfiles and it can communicate directly with NetworkManager without generating the keyfile
16:55:50 <dustymabe> ffmancera: good to know - can it generate the keyfiles even on a system not running NM at all
16:56:05 <ffmancera> dustymabe: Yes, it can
16:56:08 <dustymabe> cool
16:56:24 <lucab> this somehow sounds worse than having a keyfile in an Ignition config, because it would re-configure the network in the middle of the bootup process, no?
16:56:57 <dustymabe> lucab: depends on when it ran I guess
16:57:10 <ffmancera> walters: probably under /etc NetworkManager directory and the user will use NM to use that generated keyfiles
16:57:17 <walters> I think the `README.md` in https://github.com/nmstate/nmstate should be reworked to clarify some of this
16:57:57 <ffmancera> walters: Probably, the README can be improved a lot.
16:59:02 <thaller> lucab, you can do that, if the configuration is static (known at first deployment). nmstate would be used, if you want to reconfigure the system afterwards.
16:59:39 <ffmancera> There is also another use case, it would simplify a lot the kubernetes-nmstate daemon, assisted service and openshift-installer stack for FCOS because it would not be necessary to bundle python in the container.
16:59:44 <dustymabe> ffmancera: just making this clear.. you remain opposed to providing any service on boot that would take and apply (either by generating keyfiles or talking to NM via DBUS directly) an nmstate config?
17:00:08 <walters> wait where is python involved?  i thought nmstate is rust now?
17:00:18 <dustymabe> walters: i had that same Q in my head
17:00:51 <ffmancera> walters: Yes, now but not in the past.. so they need to use python, if we include it won't be necessary anymore
17:01:06 <ffmancera> dustymabe: Yes, we won't provide any systemd service
17:01:29 <jlebon> maybe let's forget about implementation details for now and focus on the UX: how do we feel about natively supporting nmstate YAML files in FCOS/RHCOS? i.e. as a host API
17:01:51 <dustymabe> regarding python, I still don't see the reason why including nmstate in the host would help remove python from a container
17:01:56 <dustymabe> jlebon: +1
17:02:10 <jlebon> it'd mean users would now have the choice of configuring the network either via NM keyfiles, or nmstate YAMLs, which can be a bit confusing
17:02:12 <dustymabe> jlebon: define "as a host API" ?
17:02:26 <jlebon> something that you write in your Ignition config and we apply it
17:02:42 <dustymabe> I don't feel very good about writing our own systemd service to do that and maintaining it
17:02:50 <lucab> (additionally, the goal is to thin the OS by moving stuff to containers, not viceversa)
17:03:13 <thaller> jlebon, usually the YAML files are not written by a human administrator, but are generated by other software, which uses nmstate as API.
17:03:32 <thaller> but the YAML is also a configuration format too. So it could also be used that way...
17:03:54 <dustymabe> thaller: if it's software talking to software why not just talk DBUS?
17:04:14 <dustymabe> I thought the "win" was the configuration format (i.e. being perceived as more user friendly than existing NM keyfiles)
17:04:25 <lucab> I think it hasn't been brought up so far, but how do we fit this with initrd network bringup?
17:04:49 <dustymabe> lucab: we don't I don't think
17:05:26 <dustymabe> the rules there don't change - need special sauce in initrd network? use kargs (for the most part)
17:05:27 <thaller> the win for the software is that it can express the entire host configuration in one YAML, and nmstate makes it happen. Sure, it all translates to D-Bus (or keyfiles), but doing all the steps is unfortunately not entirely trivial.
17:05:34 <dustymabe> well I guess there is the embedding of keyfiles
17:06:01 <thaller> the software "is" talking D-Bus, except that this software is libnmstate.
17:06:38 <dustymabe> dumb question (super dumb question) couldn't the software that wants to control NM just link against the library?
17:06:54 <dustymabe> or compile with it bundled (static?)
17:07:30 <jlebon> not as straightforward for Rust
17:07:39 <dustymabe> another way to ask that - is the client shelling out to `nmstatectl` ?
17:07:56 <thaller> I guess, we want to have a separate shared library (or command line tool), so that it's not vendored in by the software.
17:08:25 <ffmancera> Yes, there are different clients. Some uses nmstatectl and some use the API (libnmstate)
17:09:23 <walters> libnmstate is still Python right?
17:09:48 <ffmancera> No, we have bindings to multiple languages but the implementation is Rust
17:10:06 <thaller> (in the past it was all python)
17:10:19 <ffmancera> We plan to ship Rust library and the C bindings
17:10:30 <ffmancera> And nmstatectl too
17:10:44 <walters> ah, ok
17:11:54 <walters> ah wow I see https://github.com/nmstate/nmstate/blob/base/rust/src/clib/Cargo.toml
17:12:05 <dustymabe> OK I'm going to start making some statements
17:12:11 <dustymabe> please tell me if they aren't true
17:12:24 <dustymabe> A - nmstate is useful for dynamically configuring a running system OR generating NM keyfiles with no running system needed.
17:12:36 <dustymabe> B - It accepts a yaml formatted file as input and this file represents the desired state of the entire system.
17:12:41 <dustymabe> C - There are no plans to write a systemd unit that applies nmstate configuration on boot.
17:13:34 <dustymabe> D - This would primarily be useful for:
17:13:47 <dustymabe> D1 - applications that want to dynamically configure the Network state on the machine that don't want to do it via existing methods that already exist to do that (nmcli and/or dbus)
17:14:15 <dustymabe> D2 - generating NM keyfiles to use for systems that are yet to be deployed
17:15:11 <dustymabe> anything I'm missing?
17:15:34 <ffmancera> I think that is correct. All statements are true
17:15:34 <thaller> sound all correct to me. Minor addendum: (B: entire system OR parts); (C: the other software that is using nmstate would be that service); (D: I think the primary usecase is D1, over D2)
17:16:07 <jlebon> if we want to support nmstate, i'm leaning towards doing this somehow at the Butane level instead I think
17:16:17 <walters> I think my takeaway so far is it would make a lot of sense to document how to use nmstate offline to generate content that goes into ignition (probably in concert with butane, and I guess actually in theory butane could even use the Go bindings)
17:16:41 <jlebon> we've been pretty heavily investing in NM keyfiles so far as our network configuration format. we have sugar a bit everywhere in the installation/provisioning flow that reflects that. natively supporting another format could make the story and implementation messier
17:17:05 <walters> I would also say that kubernetes-nmstate should at least try the path of embedding the binary in their daemonset and e.g. binding in the dbus socket, I suspect that would Just Work
17:17:13 <dustymabe> walters: our docs page actually documents something very similar to this today (using nm-initrd-generator) https://docs.fedoraproject.org/en-US/fedora-coreos/sysconfig-network-configuration/#_generating_networkmanager_keyfiles_using_nm_initrd_generator
17:17:52 <walters> ohh I had missed that, that is nice
17:17:56 <dustymabe> jlebon: yeah - I agree. I was thinking maybe this would be essentially a new backend format (similar to  how you could choose keyfiles of ifcfg files in the past)
17:18:39 <walters> But I see the goals of the project and I'm not opposed to adding to the host by default if that makes some things much easier, it's just not clear to me that it does?
17:19:49 <dustymabe> Do we want to try to record a decision today?
17:20:14 <dustymabe> Are there some things that (if they were different) would change our minds about value of inclusion directly on the host?
17:21:22 <jlebon> i think if it declared some directory in /etc as canonical and shipped a systemd unit that automatically applied them, it'd be more appealing
17:22:00 <dustymabe> yeah, basically the "provisioning" story
17:22:31 <dustymabe> ffmancera: thaller: has it ever been considered to use the yaml format as a possible defacto config (i.e. instead of nm keyfiles) ?
17:23:20 <lucab> the whole flow is quite fuzzy though, and it seems at odd with Ignition where we try to configure all real-rootfs services before pivoting out of initrd
17:24:20 <jlebon> lucab: presumably the systemd service would be implemented so that it's well-ordered wrt NM/networking
17:24:32 <thaller> considered: yes. It also would be easily possible,and in many ways it already is a defined config format that can be used for that. I like however the idea that there is only one persistent storage of the configuration (keyfiles), while the nmstate YAML is ephemeral
17:25:01 <walters> NM already has pluggable backends, and given this has a C library it seems like a whole lot of layers would be shortcut using that
17:25:33 <walters> but anyways we're not redesigning things here in the immediate term
17:25:55 <thaller> walters, what backends do you mean? The formats for profiles (ifcfg,keyfile)? We would rather move towards one format, and not more...
17:27:35 <jlebon> thaller: i see how you got there. it makes sense wanting the YAML to be ephemeral since the source of truth is the NM keyfiles
17:28:05 <dustymabe> #proposed nmstate is useful for dynamically configuring a running system OR generating NM keyfiles with no running system needed. The nmstate YAML config itself can't be used on system boot without an accompanying service to apply the config. In order to use it for provisioning a user would need to write a systemd unit themselves to apply the config they wrote using Ignition. At this time
17:28:07 <dustymabe> we would like to continue not including nmstate in the host and experiment with applications that want to configure networking via nmstate bundling it.
17:28:24 <lucab> jlebon: I do not disagree, but that rules out DBus. So we are back to generating keyfiles, which can done offline.
17:28:33 <lucab> *can be done
17:28:50 <jlebon> lucab: yeah agreed, this is similar to the quadlet convo
17:28:57 <lucab> indeed
17:29:34 <dustymabe> thoughts on #proposed - does that reflect accurately the discussion?
17:30:10 <jlebon> dustymabe: ack
17:31:21 <dustymabe> any other votes?
17:31:36 <dustymabe> +1 from my side
17:32:12 <dustymabe> walters: lucab: ?
17:32:28 <walters> +1
17:33:05 <lucab> ack, but I think we aren't much helping the nmstate folks moving forward
17:33:05 <dustymabe> #agreed nmstate is useful for dynamically configuring a running system OR generating NM keyfiles with no running system needed. The nmstate YAML config itself can't be used on system boot without an accompanying service to apply the config. In order to use it for provisioning a user would need to write a systemd unit themselves to apply the config they wrote using Ignition. At this time
17:33:05 <walters> (but from my PoV, without any prejudice, if us not shipping this is a real problem or barrier I'd like to know and figure out if we do need to change course)
17:33:07 <dustymabe> we would like to continue not including nmstate in the host and experiment with applications that want to configure networking via nmstate bundling it.
17:33:52 <dustymabe> lucab: walters: agreed. thaller ffmancera - let's work together to understand a little more the perspectives on both sides and find the best working solution to the problem.
17:34:20 <dustymabe> I don't think the decision is set in stone, we just need more work/discussion to get there
17:34:25 <lucab> it feels like this is missing a full-flow design discussion, more than a package inclusion one
17:34:31 <dustymabe> lucab: I agree
17:34:35 <jlebon> +1
17:35:19 <dustymabe> thaller: ffmancera: would you be willing to talk more on the subject (maybe deeper dive and maybe video meeting) in the future?
17:35:30 <lucab> the "hook into Butane" hint and the "replace nm-initrd-generator docs usage" direction seem more promising
17:35:58 <dustymabe> #topic open floor
17:36:03 <dustymabe> anyone with anything for open floor?
17:36:17 <ffmancera> dustymabe: Thank you! We will work with client applications of Nmstate to experiment with the solution you proposed (bundling it). Of course, we are willing to discuss this deeper with demos/video meetings.
17:36:34 <aaradhak[m]> Hey all, an update for the FCOS meet next week, we will be having our FCOS community video meeting next Wednesday (May 4th, 12.30pm - 1.30pm) . One of our Red Hat Engineers Brian Tomilson would be presenting a demo on his work on Automated F35 for Raspberry Pi.
17:36:34 <aaradhak[m]> https://discussion.fedoraproject.org/t/automated-fedora-coreos-35-for-raspberry-pi-4-b-400/38359
17:36:34 <aaradhak[m]> Please let us know if there are any specific topics which need to be added in the discussion .
17:36:34 <aaradhak[m]> Meeting link - https://meet.google.com/meo-enbb-rur?hs=224
17:36:36 <jlebon> ffmancera++
17:36:37 <zodbot> jlebon: Karma for ffmancera changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:36:42 <thaller> agree. Thanks everybody for your time.
17:36:48 <ffmancera> dustymabe: Thanks for your time and we will reach you in the future with more details :-)
17:36:51 <dustymabe> we are having a video meeting next week ^^
17:37:16 <jlebon> aaradhak[m]: cool! looking forward to it
17:37:22 <ffmancera> dustymabe: Do you mind if we join to discuss this or is it too son?
17:37:25 <dustymabe> we have one topic already, but there is room for more - thaller ffmancera if we have time between now and then - maybe we could present something there and discuss more
17:37:27 <ffmancera> s/son/soon
17:37:58 <dustymabe> ffmancera: I don't think it's too soon - we might rehash some of what we already said today, but that's OK
17:38:10 <ffmancera> dustymabe: Yes please, we would be glad to do it.
17:38:47 <dustymabe> ffmancera: cool - i'll remind you next week
17:38:54 <ffmancera> dustymabe: Thank you!
17:39:18 <dustymabe> #info we will be having our FCOS community video meeting next Wednesday at https://meet.google.com/meo-enbb-rur?hs=224
17:39:42 <dustymabe> also FYI everyone - we are hoping FEdora 36 GA is next tuesday - which means our `testing` stream will switch over then
17:40:01 <dustymabe> any other topics for open floor? if not I'll close out the meeting in 60s
17:41:03 <dustymabe> #endmeeting