16:31:13 <travier> #startmeeting fedora_coreos_meeting 16:31:13 <zodbot> Meeting started Wed Aug 31 16:31:13 2022 UTC. 16:31:13 <zodbot> This meeting is logged and archived in a public location. 16:31:13 <zodbot> The chair is travier. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 16:31:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:31:13 <zodbot> The meeting name has been set to 'fedora_coreos_meeting' 16:31:17 <travier> #topic roll call 16:31:18 <bgilbert> .hi 16:31:19 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net> 16:31:22 <travier> .hello siosm 16:31:23 <spresti[m]> hi 16:31:23 <zodbot> travier: siosm 'TimothΓ©e Ravier' <travier@redhat.com> 16:31:28 <mnguyen_> .hello mnguyen 16:31:28 <fifofonix> .hi 16:31:28 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com> 16:31:31 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com> 16:31:34 <spresti[m]> .hi 16:31:34 <zodbot> spresti[m]: Sorry, but user 'spresti [m]' does not exist 16:32:10 <lucab> .hi 16:32:10 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com> 16:33:15 <pehunt> hello 16:33:32 <aaradhak> .hi 16:33:33 <zodbot> aaradhak: aaradhak 'Aashish Radhakrishnan' <aaradhak@redhat.com> 16:33:38 <pehunt> .hi 16:33:39 <zodbot> pehunt: Sorry, but user 'pehunt' does not exist 16:33:42 <davdunc> .hello2 16:33:43 <zodbot> davdunc: davdunc 'David Duncan' <davdunc@amazon.com> 16:35:04 <travier> π 16:35:21 <spresti[m]> π 16:35:37 <travier> We have a good bunch of folks, let's start 16:35:56 <travier> #chair bgilbert lucab aaradhak pehunt mnguyen_ davdunc 16:35:56 <zodbot> Current chairs: aaradhak bgilbert davdunc lucab mnguyen_ pehunt travier 16:36:07 <travier> #chair spresti[m] fifofonix 16:36:07 <zodbot> Current chairs: aaradhak bgilbert davdunc fifofonix lucab mnguyen_ pehunt spresti[m] travier 16:36:21 <travier> (hope I did not forgot someone) 16:36:28 <travier> forget* 16:36:33 <travier> #topic Action items from last meeting 16:36:55 <travier> No actions π 16:37:05 <travier> Let's move one 16:37:06 <travier> one* 16:37:08 <travier> on* 16:37:25 <travier> #topic New Package Request: conmon-rs 16:37:31 <travier> #link https://github.com/coreos/fedora-coreos-tracker/issues/1288 16:37:48 <travier> We have guest(s?) today so let's start with this one 16:38:01 <travier> It's about adding a new package to FCOS 16:38:14 <travier> pehunt: can you give us a short overview? 16:38:18 <pehunt> Hey all 16:38:35 <travier> π 16:38:44 <pehunt> We're looking to add conmon-rs to fcos. the plan is to have it replace conmon as the oci runtime monitor cri-o and podman use 16:39:25 <pehunt> it's been a long time since we've done this, so we're not really familiar with the whole process. Via the ticket I see we need to request a fedora package first 16:39:48 <travier> yes, all packages that are included in FCOS come from Fedora 16:40:08 <travier> So the first step would probably be to get it into Fedora 16:40:14 <pehunt> makes sense :) I don't think I have any specific items to discuss, does anyone have any questions for me on it? 16:40:39 <travier> From the issue tracker: how would the migration happen? 16:41:03 <travier> How do users try out the new one? 16:41:17 <travier> Are they 100% compatible? 16:41:57 <pehunt> so the migration would happen from the crio config. Once we have full support in (we're looking at experimental support now), a user would update the crio config to use a runtime of type `pod` instead of `oci` (naming is hard) 16:42:19 <cyberpear> sounds like a win to me! can both be installed in parallel? podman config? 16:42:28 <lucab> I was looking at the current COPR package and I think it currently installs a binary with a different name. I think the interface compatibility is at the CLI level, is that right? 16:42:54 <pehunt> both can be installed in parallel. I think containers.conf will need a new field, or it would be specified via podman CLI 16:43:14 <pehunt> CLI compatiblity is not in the cards. It's a whole new way of interacting with a container monitor 16:43:43 <pehunt> both podman and cri-o require an update to know how to communicate (over a RPC mechanism called cap'n proto) 16:44:00 <pehunt> cri-o's had support since 1.24, podman is approaching support now 16:45:09 <travier> can you run a container with the new conmon and then go back to classic conmon? 16:45:33 <travier> or do you need to re-create it? 16:45:38 <pehunt> need to re-create it 16:45:48 <travier> ouch 16:45:57 <pehunt> conmon{,rs} requires being the direct parent of the container process, so there's no simple migration path 16:46:18 <travier> Let's consider this scenario: 16:46:50 <travier> FCOS with conmon only, I create a container. I update & reboot to FCOS with conmon-rs only. Wht happens to my container? 16:47:04 <pehunt> since you rebooted, the container would restart anyway 16:47:16 <travier> restart yes, but rm & create? 16:47:26 <travier> maybe that's more for podman than cri-o 16:47:48 <pehunt> I don't know exactly how podman handles a restart, but cri-o removes all containers on a restart, as there's state in tmpfs that is needed to track the execution, and that's obviously lost 16:48:49 <pehunt> under the hood, AFAICT, podman containers are recreated in a similar mechanism. 16:49:27 <pehunt> it just looks like they're the same because they're the same name and everything, but the underlying container process is different. I am not sure how the podman community intends on handling it, but I imagine there may need to be a `podman system migrate` call or the like 16:49:59 <travier> OK, so we will have to make sure that we reach out to the podman team 16:50:38 <travier> #action travier Reach out to the podman team or the conmon-rs transition 16:51:07 <travier> This makes it a bit weird to include it in FCOS before we get podman support 16:51:28 <travier> But submitting this for packaging etc. will take some time anyway 16:52:00 <pehunt> I see a situation where cri-o uses conmon-rs experimentally before podman uses it at all. podman would still use conmon (which we don't intend on dropping for a while). is that problematic? 16:52:03 <bgilbert> #undo 16:52:03 <zodbot> Removing item from minutes: ACTION by travier at 16:50:38 : travier Reach out to the podman team or the conmon-rs transition 16:52:07 <bgilbert> #action travier Reach out to the podman team for the conmon-rs transition 16:52:16 <bgilbert> (typo might cause semantic confusion) 16:52:29 <lucab> pehunt: thanks for the additional details, I didn't initially notice that it has its own new RPC mechanism 16:52:48 <travier> bgilbert: π 16:54:41 <travier> What do other folks think? 16:54:48 <bgilbert> it sounds as though adding the new package won't create any compat concerns, but removing the old one will? 16:54:51 <bgilbert> assuming we don't change the default 16:55:02 <pehunt> correct, and removing the old is not yet on our radar 16:56:05 <pehunt> there would need to be full feature completeness in cri-o/podman, a while of conmorns being the default (with conmon being a backup to revert back to) and migration paths for both managers before we'd consider removing conmon 16:56:32 <bgilbert> but the plan is definitely to remove conmon in the long run, right? 16:56:50 <pehunt> yeah 16:57:14 <bgilbert> it seems pretty harmless to ship conmon-rs as an aid to testing/migration, provided that we clearly document the situation 16:57:34 <bgilbert> we generally favor long, careful migration paths, which this is 16:58:05 <travier> +1 16:58:43 <bgilbert> without podman support, this would be a leaf package, which is a bit odd 16:58:46 <bgilbert> since we don't ship cri-o 16:58:57 <travier> Maybe we might wait until we have initial podman support to include it and let users give it a try? 16:59:15 <pehunt> that's a good point 16:59:34 <bgilbert> yeah, that's what I'm wondering. any rough sense of ETA for (at least experimental) podman support? 17:00:26 <pehunt> I don't have an accurate estimate. Some of the work has been done, but podman needs more than cri-o so there are a number of pieces still missing 17:00:39 <pehunt> on the magnitude of months, not weeks 17:01:22 <pehunt> I am fine if we want to wait, and only install conmon-rs optionally alongside the cri-o rpm. We can still go through the process of adding conmon-rs to fedora so we're ready to include it when podman is ready 17:02:10 <lucab> it sounds like OKD may want to start providing this at first, maybe even earlier than FCOS itself? 17:02:32 <pehunt> I can see that being the case 17:02:35 <bgilbert> if there are users who actively want to use their own cri-o with conmon-rs (such as OKD), I'm not 100% opposed to including it sooner. but without podman support, we'd have no way to know whether the conmon-rs we're shipping even works 17:02:40 <travier> +1 for lucab comment. Maybe this is of interest for early inclusion in OKD 17:03:02 <pehunt> that route works for me too 17:03:30 <travier> And they are working on CentOS Stream CoreOS for OKD too so they will be interested in that too 17:03:53 <travier> alright, should we vote? 17:04:27 <bgilbert> travier: need a proposal :-) 17:04:31 <travier> yep :) 17:05:05 <lucab> IIRC cri-o comes from a module, right? 17:05:28 <pehunt> correct! 17:05:59 <travier> # proposal We will not add conmon-rs to FCOS now but we will as soon as we have basic support in podman to help with the transition 17:06:13 <travier> (open to changes 17:06:14 <travier> ) 17:06:19 <lucab> if yes, version-coupling the two in the same module may also help stabilization (if unforeseen RPC/API breakage may still happen) 17:07:03 <bgilbert> travier: that may be stronger than necessary 17:07:23 <bgilbert> we could add conmon-rs as soon as podman supports it, even if there's no migration support 17:07:28 <travier> sure, I'm bad at proposals 17:07:29 <bgilbert> and just document that -rs is a separate world for now 17:07:53 <travier> agree 17:08:48 <lucab> proposal: suggest shipping conmon-rs in the cri-o module for now, and let OKD pick it up directly bypassing FCOS. Re-evaluate FCOS inclusion once stabilized and podman supports that? 17:09:03 <bgilbert> lucab: once what is stabilized? 17:09:07 <walters> OKD and RHCOS? 17:09:37 <travier> We'll probably add it to RHCOS but that's another discussion? 17:09:43 <pehunt> I do intend on having an RHCOS conversation, but i guessed it wasn't in scope here 17:09:44 <lucab> bgilbert: conmon-rs API surface, sorry 17:10:04 <travier> yes, I think the RHCOS discussion can happen after this one 17:11:20 <lucab> I guess the interesting discussion is for Centos9 stream, i.e. either to have it in the main stream or in a dedicated module/sig/copr. 17:11:32 <bgilbert> I'm unclear on the relationship between podman gaining support and the API surface stabilizing 17:11:46 <bgilbert> i.e., whether Luca's proposal is stronger or weaker 17:12:19 <lucab> which is similar to the discussion here about plain Fedora or side-module 17:12:25 <pehunt> podman having full support implies the API surface has stabalized. At that point, any new features would be purely additive, which is handled by the IPC mechanism 17:12:37 <bgilbert> pehunt: +1 17:13:10 <pehunt> I am guessing for C9S we'd include in OKD SIG until it is ready for podman, same as for FCOS 17:13:17 <travier> # proposal We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The transition timeline will be discussed then. 17:13:29 <lucab> bgilbert: that was a minor detail for version-compatibility. I agree it isn't well formulated and we drop that part 17:13:37 <bgilbert> lucab: +1 17:14:20 <bgilbert> travier: for the last sentence, maybe "The timeline for migrating away from legacy conmon will be discussed then." for clarity? 17:14:29 <travier> π 17:14:59 <travier> #proposal We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The timeline for migrating away from legacy conmon will be discussed then. 17:15:32 <lucab> ack 17:15:34 <bgilbert> +1 17:16:34 <bgilbert> pehunt: does that meet your needs? 17:16:40 <pehunt> +1 17:16:41 <travier> other folks? anybody can vote π 17:17:10 <spresti[m]> +1 17:18:22 <travier> #accepted We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The timeline for migrating away from legacy conmon will be discussed then. 17:18:35 <travier> not sure about the meetbot commands anymore 17:18:38 <pehunt> thanks everyone :) 17:18:41 <travier> Thanks pehunt for joining us 17:18:48 <bgilbert> pehunt: thanks for the info and discussion! 17:18:56 <bgilbert> #agreed We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The timeline for migrating away from legacy conmon will be discussed then. 17:19:06 <travier> π 17:19:34 <walters> (not strictly related but worth bearing in mind that testing out conmon-rs with crio on FCOS should be as easy as writing a Dockerfile that installs both and then booting it, don't even need an RPM of conmon-rs either, could just drop the binary out of a CI build) 17:20:04 <travier> I don't think we have to discuss the other tickets as we already did last week so I think I'll go to open floor 17:20:10 <travier> +1 walters 17:20:14 <bgilbert> one moment travier 17:20:18 <travier> ok 17:20:24 <bgilbert> #undo 17:20:24 <zodbot> Removing item from minutes: AGREED by bgilbert at 17:18:56 : We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The timeline for migrating away from legacy conmon will be discussed then. 17:20:25 <bgilbert> #undo 17:20:25 <zodbot> Removing item from minutes: ACCEPTED by travier at 17:18:22 : We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The timeline for migrating away from legacy conmon will be discussed then. 17:20:27 <bgilbert> #agreed We will not add conmon-rs to FCOS yet as it won't be usable until podman gets support. We will include it once podman gets initial support for conmon-rs. The timeline for migrating away from legacy conmon will be discussed then. 17:20:34 <bgilbert> both cmds exist but are slightly different 17:20:37 <bgilbert> ok, carry on 17:20:40 <travier> ok thanks 17:20:59 <travier> #topic Open Floor 17:21:18 <lucab> #link https://github.com/coreos/fedora-coreos-tracker/issues/1287 17:22:03 <lucab> this was initially a request to change FCOS / systemd defaults, but I think we steered away from that 17:23:17 <lucab> let's see how this progress on the musl side, but I expect this to become a common annoyance in musl-based images in the future 17:24:05 <travier> π 17:26:21 <lucab> (that's all from my side) 17:26:36 <travier> probable needs to be reported to musl :/ 17:26:41 <travier> probably* 17:29:28 <travier> OK, will close this one if we don't have anything else 17:29:37 <travier> Nice digging indeed lucab for this one 17:30:12 <travier> #endmeeting