16:30:17 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:17 <zodbot> Meeting started Wed Mar 29 16:30:17 2023 UTC.
16:30:17 <zodbot> This meeting is logged and archived in a public location.
16:30:17 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:17 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:21 <dustymabe> #topic roll call
16:30:24 <dustymabe> .hi
16:30:25 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:57 <jbrooks> .hello jasonbrooks
16:30:58 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:31:09 <marmijo[m]> .hello marmijo
16:31:10 <apiaseck> .hi c4rt0
16:31:10 <zodbot> marmijo[m]: marmijo 'Michael Armijo' <marmijo@redhat.com>
16:31:13 <zodbot> apiaseck: Sorry, but user 'apiaseck' does not exist
16:31:22 <apiaseck> .hello c4rt0
16:31:23 <zodbot> apiaseck: c4rt0 'Adam Piasecki' <c4rt0gr4ph3r@gmail.com>
16:31:49 <gursewak> .hi
16:31:50 <zodbot> gursewak: gursewak 'Gursewak Singh' <gurssing@redhat.com>
16:32:26 <ravanelli> .hi
16:32:26 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:32:30 <jlebon> .hello2
16:32:31 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:47 <dustymabe> #chair jbrooks marmijo[m] apiaseck gursewak jlebon ravanelli
16:32:47 <zodbot> Current chairs: apiaseck dustymabe gursewak jbrooks jlebon marmijo[m] ravanelli
16:32:52 <dustymabe> did I miss anyone?
16:33:52 <dustymabe> ok cool - let's get started
16:33:54 <dustymabe> #topic Action items from last meeting
16:34:14 <dustymabe> #info there are no action items from last meeting!
16:35:00 <jlebon> woohoo!
16:35:07 * dustymabe was hoping to get fabian and maybe walters here today for some discussion on tickets
16:35:35 <dustymabe> oh well - we'll move to tickets and then see who is around
16:35:42 <dustymabe> #topic tracker: Fedora 38 Test Week on March 28
16:35:47 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1440
16:36:36 <dustymabe> #info it's not too late to participate in the Fedora CoreOS test week! There are available test cases to run. Check the test day results page: https://testdays.fedoraproject.org/events/153
16:36:43 <apiaseck> I can say one thing here - from my perspective it was an awesome learning experience.
16:36:49 <dustymabe> apiaseck+_
16:36:52 <dustymabe> apiaseck++
16:36:53 <zodbot> dustymabe: Karma for c4rt0 changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:37:44 <dustymabe> thanks to everyone who's been involved so far and to ravanelli and SumantroMukherje for organizing the test day!
16:38:24 * dustymabe moves on to the next ticket
16:38:24 <ravanelli> dustymabe: thanks you for always helping in the process ;)
16:38:32 <dustymabe> :)
16:38:46 <jlebon> ravanelli++ SumantroMukherje++
16:38:48 <apiaseck> Thanks ravanelli and SumantroMukherje!
16:38:53 <dustymabe> #topic Should the QEMU image use ptp_kvm where available by default?
16:39:02 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1433
16:39:15 <dustymabe> jlebon: want to intro this one?
16:39:33 <jlebon> dustymabe: sure
16:39:46 <jlebon> we talked about this last week (or maybe the week before that)
16:40:09 <jlebon> the TL;DR is: on QEMU we'll start using the ptp_kvm as a clock source for chrony
16:40:28 <jlebon> though there's new information
16:41:29 <jlebon> while working on the PR, one thing that became obvious (via a test breaking) is that we will stop caring about DHCP-provided NTP servers by default on QEMU (when ptp_kvm is available)
16:42:12 <jlebon> since QEMU as a platform can be used in many different ways, i can certainly imagine people today using DHCP to provide their NTP configs
16:42:26 <jlebon> with this change, that config would no longer apply by default
16:42:37 <jlebon> (see last paragraph in https://github.com/coreos/fedora-coreos-config/pull/2263#issuecomment-1481943298)
16:43:14 <jlebon> so i think there's a risk there we could be breaking existing workflows
16:43:44 <jlebon> one potential approach is on QEMU, we *don't* neuter PEERNTP
16:44:20 <jlebon> so then both ptp_kvm and DHCP-provided servers would be used. but it differs from what we do on other chrony-enabled platforms (where we always shutoff PEERNTP)
16:44:47 <jlebon> thoughts?
16:45:02 <dustymabe> what exactly does PEERNTP=yes/no do again?
16:45:15 <jlebon> whether the DHCP NTP option is picked up
16:46:14 <dustymabe> yeah.. that could be a good compromise,
16:46:59 <dustymabe> i.e. add ptp_kvm to the mix but don't disable the DHCP provided NTP
16:47:12 <jlebon> right
16:47:32 <dustymabe> in which case we probably wouldn't need to send an announcement and give people instructions?
16:48:03 <jlebon> yeah, it shouldn't break anything
16:48:24 <dustymabe> but if we left PEERNTP=no we would need to send that announcement?
16:49:02 <jlebon> maybe?  depends on how much we think users might be using DHCP-provided NTP on QEMU
16:49:58 <dustymabe> I think maybe we start with that
16:50:16 <dustymabe> another thing here.. we should really try to hoist all of this "platform-chrony" stuff somewhere else
16:50:35 <jlebon> "that" = leave PEERNTP on?
16:50:40 <dustymabe> jlebon: yes
16:51:23 <dustymabe> i mean, maybe it's not the right thing if you were going greenfield, but it at least still considers the DHCP ntp servers
16:52:04 <jlebon> chrony would probably still prefer the kvm clock, but the DHCP NTP servers could be used to crosscheck it at least
16:52:21 <dustymabe> any other opinions here?
16:52:39 <travier> .hello siosm
16:52:39 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:52:52 <jlebon> anyway, WFM. it does make the code slightly more complex since it'd be inconsistent across platforms, but nothing major
16:53:46 <dustymabe> jlebon: want to do a proposed?
16:53:52 <jlebon> sure
16:54:27 <jlebon> #proposed we will keep PEERNTP=yes on QEMU so that DHCP-provided NTP servers are still picked up even if ptp_kvm is available
16:54:56 <dustymabe> maybe s/are still picked up/are still considered/
16:55:01 <dustymabe> +1 from me
16:55:12 <ravanelli> +1
16:55:27 <jlebon> #agreed we will keep PEERNTP=yes on QEMU so that DHCP-provided NTP servers are still considered even if ptp_kvm is available
16:56:04 <dustymabe> jlebon: what do you think about my statement earlier about trying to get someone else to maintain this code in the future?
16:56:20 <dustymabe> i.e. the chrony rpm itself (or even upstream)
16:56:26 <dustymabe> is there anything here that is coreos specific?
16:56:31 <jlebon> dustymabe: that'd be great, yeah
16:56:43 <jlebon> or maybe nm-cloud-setup ?
16:57:01 <jlebon> not really. you'd want this in traditional Fedora too I think.
16:57:28 <dustymabe> no idea - but we could definitely use some help identifying pieces of code we have now that would be better somewhere else
16:57:39 <dustymabe> either way, sorry about sidetracking this discussion
16:57:49 <jlebon> yeah, agree
16:58:17 <dustymabe> #topic Platform Request: kubevirt
16:58:24 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1126
16:58:43 <dustymabe> I invited fabian to the meeting today, but looks like he wasn't able to make it
16:59:10 <dustymabe> maybe let's just target a quay org we own and can push directly to for now
16:59:33 <dustymabe> (this is re: https://github.com/coreos/fedora-coreos-tracker/issues/1126#issuecomment-1479996473 )
16:59:37 <jlebon> yeah, i wouldn't block on this to get the artifact building and testing
16:59:59 <dustymabe> quay.io/fedora/fedora-coreos-kubevirt ?
17:00:17 <dustymabe> or should we use containerdisk instead of kubevirt ?
17:00:49 <jlebon> containerdisk feels a bit too generic
17:01:11 <dustymabe> yeah, but I think the idea is that these "images" can be run in other orchestraters
17:01:19 <dustymabe> but the platform ID we've chosen is kubevirt
17:01:31 <jlebon> but would the other stuff like ignition and afterburn be the same?
17:01:32 <dustymabe> so we should probably stick with it in the container registry org name
17:01:41 <dustymabe> jlebon: good Q
17:01:43 <dustymabe> :)
17:02:15 <dustymabe> `fedora-coreos-kubevirt` sound good?
17:02:27 <jlebon> yeah, let's stick with that for now
17:02:37 <jlebon> we can always change it later if needed
17:02:59 <jlebon> we could even push to both quay.io/fedora and quay.io/containerdisks
17:03:03 <jlebon> (eventually)
17:03:30 <dustymabe> #proposed We'll target `quay.io/fedora/fedora-coreos-kubevirt` as the location for our kubevirt images for now. We may consider quay.io/containerdisks in the future if more information about that repo comes to light.
17:03:37 <dustymabe> cc gursewak ^^
17:03:50 <jlebon> ack
17:03:54 <gursewak> ack
17:03:56 <marmijo[m]> ack
17:04:29 <dustymabe> #agreed We'll target `quay.io/fedora/fedora-coreos-kubevirt` as the location for our kubevirt images for now. We may consider quay.io/containerdisks in the future if more information about that repo comes to light.
17:05:02 <dustymabe> #topic Tracker for bootc integration
17:05:09 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1446
17:05:50 <dustymabe> Was hoping walters would be here, but I'll try to infer what the desire behind this ticket is/was.
17:06:10 <walters> hey sorry, I am now
17:06:18 <dustymabe> heya walters :)
17:06:28 <dustymabe> want to intro this?
17:07:12 <walters> I edited that issue to clarify that a major point is supporting bootc install which is motivated by https://github.com/coreos/fedora-coreos-tracker/issues/1151
17:08:08 <dustymabe> walters: done?
17:08:09 <walters> The other rationale is that the project rpm-ostree is just confusingly named when OS updates come from container images, so bootc is a more streamlined tool for that.   I also think the zincati/rpm-ostree split has been painful and hope to rectify that with bootc, etc.
17:08:14 <walters> yeah
17:08:38 <dustymabe> so let me re-hash the summary from my head
17:09:06 <dustymabe> 1. bootc is a new piece of tech that could help us achieve some short term and long term goals
17:09:37 <dustymabe> 2. supporting the bootc "flow" would require a significant change in the way we do things today (some positive and some negative pieces of that)
17:09:41 <walters> right, though it uses the same ostree-container library that rpm-ostree uses; that part isn't new
17:10:02 <dustymabe> 3. this current ticket proposes that we start experimenting with bootc more
17:10:11 <walters> (I mentioned this in my Container Days talk https://fedorapeople.org/~walters/2023-containerplumbing-bootc.html - the original project with just bootc upgrade and bootc switch was just 500 LoC)
17:10:36 <walters> yep agree on 2 and 3
17:10:55 <walters> although the value of "significant" depends
17:11:45 <bgilbert> .hi
17:11:46 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
17:12:01 <dustymabe> i guess there is a 3b, which is stating clearly that it would be experimentation and not necessarily hard commiting to it. I think experimentation is good and healthy, but we should use th results of that experimentation to drive the ultimate decision on where we want to go with our production streams
17:12:08 <dustymabe> #chair walters bgilbert
17:12:08 <zodbot> Current chairs: apiaseck bgilbert dustymabe gursewak jbrooks jlebon marmijo[m] ravanelli walters
17:13:12 <jlebon> i like the idea of a stream that's just containers
17:13:29 <dustymabe> walters: so the idea here is that we would create a new development stream that includes bootc and just build the container?
17:13:34 <jlebon> we'd probably still build QEMU at least though for testing
17:14:06 <walters> right
17:14:51 <walters> as far as having a qemu image, not opposed but I also think we want cosa run --cimage quay.io/fedora/fedora-coreos:bootc-devel that defaults to doing an in-place update and switch
17:15:06 <walters> which is a lot more efficient now that https://github.com/coreos/coreos-assembler/pull/3359 landeed
17:16:27 <dustymabe> so bootc itself isn't in Fedora, right?
17:16:33 <dustymabe> so we'd be pulling from the copr when we do builds?
17:16:54 <walters> right, obviously will package eventually but the whole manual process ruins fast feedback cycles
17:17:29 <jlebon> sounds like we'd revive https://github.com/coreos/fedora-coreos-tracker/issues/910 for this :)
17:17:48 <walters> yeah, bootc is a new motivation for that
17:17:56 <jlebon> (well, 'revive' implies it was once alive)
17:18:19 <dustymabe> jlebon: ehh. I'd propose we just call this stream bootc and have bootc be the only difference
17:18:56 <dustymabe> i.e. I don't see value in making it differ significantly.
17:19:13 <walters> well, at a current level we do need git main of bootupd too
17:19:26 <jlebon> dustymabe: maybe? it depends how many COPR components we pull in
17:19:34 <dustymabe> can you do a rpm build of bootupd?
17:20:35 <walters> can yes
17:21:48 <dustymabe> I mean it's probably OK if there were a few COPRs we were pulling from. I think what I'm trying to say is I see https://github.com/coreos/fedora-coreos-tracker/issues/910 as a distinct thing from this (and one that we should discuss merits of independently)
17:23:01 <dustymabe> one thing we should also discuss is how to handle user questions about it
17:23:54 <dustymabe> obviously anyone can pull things from the build browser (or from a quay registry), but it should probably be clear up front that this is for experimentation
17:25:05 <jlebon> i wonder if making a "bootc-specific" stream, we should have the stream be called e.g. `experimental` where bootc happens to be what we want to experiment with right now
17:25:13 <jlebon> if instead of making*
17:25:38 <dustymabe> jlebon: yeah. adding a new stream is non-zero work, so maybe we should make something that could be re-used in the future
17:25:50 <bgilbert> +1
17:25:54 <jlebon> and the name would make it clear it's experimental :)
17:26:09 <walters> definitely agree on having an experimental
17:26:09 <dustymabe> `next` is kind of supposed to be for some experimental things - but I think this is a higher level of experimentation than we planned for there
17:26:44 <dustymabe> ✔️ for `experimental` then
17:26:55 <walters> that said I also think there's a lot less overhead to being container-native in build processes; this is an interesting sub-topic.  The pipeline features desired here are 1) multi-arch  2) integration with kola and build status reporting
17:27:00 <dustymabe> this will probably take $some pipeline work too
17:27:33 <dustymabe> walters +1
17:27:33 <walters> we could even of course have the images here do rpm-ostree install bootc in a Dockerfile, deriving from an existing image
17:27:58 <dustymabe> walters: that is interesting
17:28:20 <dustymabe> if we did that then maybe we don't even have to re-run the tests?
17:28:34 <dustymabe> or maybe we'd want to run at least bootc specific tests
17:28:41 <walters> yeah
17:29:00 <walters> which exist in kola form https://github.com/containers/bootc/tree/main/tests/kolainst
17:29:33 <dustymabe> ok
17:29:39 <dustymabe> we're getting close to time
17:30:11 <dustymabe> I think the idea here is to have an experimental stream with at least bootc installed that people can play around with and we can get some early feedback on
17:30:13 <walters> but the thing that bootc install --takeover combined with some basic logic for scraping injected ssh key auth will enable is we can even try out testing via tmt but having it replace any existing image
17:30:25 <dustymabe> I'll put up a proposed
17:31:31 <dustymabe> #proposed We'll create a new `experimental` FCOS development stream for testing out highly experimental new features and add bootc as the first iteration of experimentation in that stream.
17:31:55 <walters> +1
17:31:56 <dustymabe> I think there are some more details (including implementation details) it would be good to discuss further (maybe outside of the meeting)
17:31:56 <jlebon> ack
17:32:03 <jbrooks> +1
17:32:15 <ravanelli> +1
17:32:25 <bgilbert> +1
17:32:29 <spresti[m]> +1
17:32:32 <jlebon> walters: fyi, over IRC on my client, (what i think is) your backticks isn't rendering correctly
17:32:56 <jlebon> it looks like a control character
17:32:58 <jbrooks> same
17:33:06 <dustymabe> I think one thing I might be able to make more clear in the proposed is that bootc isn't necessarily a foregone conclusion for FCOS.. I'm definitely interested to see how the experimentation goes and also interested in a lot of the longer term "migration" type details.
17:33:34 <jlebon> +1
17:33:41 <dustymabe> worth doing a re-propose? or do you think that's clear enough from us just calling it `experimental` ?
17:34:21 <dustymabe> #agreed We'll create a new `experimental` FCOS development stream for testing out highly experimental new features and add bootc as the first iteration of experimentation in that stream.
17:34:26 <dustymabe> either way - marked it ^^
17:34:30 <dustymabe> #topic open floor
17:34:36 <dustymabe> anyone with anything for open floor?
17:35:44 <dustymabe> #endmeeting