16:29:48 <dustymabe> #startmeeting fedora_coreos_meeting 16:29:48 <zodbot> Meeting started Wed Sep 29 16:29:48 2021 UTC. 16:29:48 <zodbot> This meeting is logged and archived in a public location. 16:29:48 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 16:29:48 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:29:48 <zodbot> The meeting name has been set to 'fedora_coreos_meeting' 16:29:53 <dustymabe> #topic roll call 16:29:58 <dustymabe> .hi 16:30:00 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com> 16:30:27 <lucab> .hi 16:30:30 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com> 16:30:33 <bgilbert> .hi 16:30:34 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net> 16:30:41 <jlebon> .hello2 16:30:42 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com> 16:31:10 <dustymabe> #chair lucab bgilbert jlebon 16:31:10 <zodbot> Current chairs: bgilbert dustymabe jlebon lucab 16:32:08 <dustymabe> light attendance today.. I know there is a conference going on that some people are attending 16:32:21 <lorbus> .hi 16:32:22 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com> 16:32:30 <dustymabe> welcome lorbus :) 16:32:53 <dustymabe> #chiar lorbus 16:33:00 * lorbus waves 16:33:03 <dustymabe> will wait another minute and then get started 16:34:14 <dustymabe> #topic Action items from last meeting 16:34:20 <dustymabe> * lorbus and jlebon to reach out to the containers team to discuss what cri-o versions will be supported and how at the modular level to pull it off (context: https://github.com/coreos/fedora-coreos-tracker/issues/767) 16:34:42 <jlebon> can you re-action that one? 16:34:50 <dustymabe> #action lorbus and jlebon to reach out to the containers team to discuss what cri-o versions will be supported and how at the modular level to pull it off (context: https://github.com/coreos/fedora-coreos-tracker/issues/767) 16:34:51 <jaimelm> .hello2 16:34:52 <zodbot> jaimelm: jaimelm 'Jaime Magiera' <jaimelm@umich.edu> 16:34:55 <lorbus> I haven't come around to doing that. Not sure if jlebon has. Maybe re-action this one? 16:34:57 <dustymabe> #chair jaimelm 16:34:57 <zodbot> Current chairs: bgilbert dustymabe jaimelm jlebon lucab 16:35:01 <dustymabe> done :) 16:35:07 <jlebon> lorbus: let's get together at some point to chat :) 16:35:29 <dustymabe> #topic [F35] Fedora CoreOS Test Day 16:35:36 <lorbus> jlebon: yes, definitely :) 16:35:40 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/973 16:35:45 <dustymabe> jlebon++ 16:35:47 <dustymabe> lorbus++ 16:36:19 <dustymabe> we're going to try to have a FCOS test day for the f35 rebase here soon 16:36:31 <dustymabe> since our `next` stream is switched over it should be a good candidate for testing 16:37:08 <dustymabe> in the past we've added test cases that help test our documentation 16:37:19 <dustymabe> since most of our CI automates other testing that we care about 16:37:44 <dustymabe> does that sound reasonable for this iteration? 16:38:10 <jlebon> +1 16:38:20 <jlebon> esp. platforms we don't have any coverage on 16:38:49 <dustymabe> i think sumantro was suggesting we possibly use next week for the "test day" 16:39:08 <dustymabe> we could possibly try to go later if we think that's a bit ambitious 16:39:32 <lucab> It would be good to note down the last F34 release on next and manually test the upgrades from that 16:39:33 <dustymabe> probably need to spend some time before hand identifying where our docs have changed or adding new test cases for new docs that have been added 16:39:45 <dustymabe> lucab: indeed 16:40:47 <jaimelm> +1 on testing day, +1 on identifying docs changes beforehand. Next week sounds ambitious. 16:41:19 <dustymabe> ok, maybe we can push to the following week and schedule a day next week for syncing on test case changes 16:41:26 <jaimelm> ^^ 16:41:59 <dustymabe> #action dustymabe to talk to sumantro to try to get FCOS testing on the week of oct 11 16:42:35 <dustymabe> jaimelm: we should schedule some time now to try to identify test case changes (and interested parties) 16:42:42 <dustymabe> i'm interested 16:43:00 <dustymabe> jaimelm: would love some help organizing too if you're able/willing 16:43:07 <dustymabe> I think last time we may have caught you at a bad time 16:44:07 <dustymabe> for time: next Monday (morning US) works for me - any other suggestions? 16:45:22 <dustymabe> is anyone else that's interested in looking at the docs test cases able to make it Monday (anytime)? 16:45:32 <jlebon> yup, WFM too! 16:45:35 <jlebon> i can help 16:45:54 <jaimelm> dustymabe: Monday works for me 16:46:14 * jaimelm is in two meetings at once. responses may be delayed. 16:46:18 <dustymabe> #action dustymabe to schedule some time on Monday to identify docs test cases and prepare for F35 test day 16:46:33 <dustymabe> #topic tracker: Fedora 35 changes considerations 16:46:39 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues?q=is%3Aissue+is%3Aopen+label%3AF35-changes 16:46:56 <dustymabe> looks like we only have two subtickets left! 16:46:58 <dustymabe> almost done 16:47:05 <jlebon> woohoo! 16:47:20 <dustymabe> looks like travier is the brave soul assigned to both of them 16:47:23 <walters> .hello2 16:47:24 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com> 16:47:38 <dustymabe> #chair walters 16:47:38 <zodbot> Current chairs: bgilbert dustymabe jaimelm jlebon lucab walters 16:47:40 <dustymabe> welcome! 16:47:58 <jlebon> travier: feels like we can drop the rpmautospec one? definitely nice to have, but no need to tie to f35 16:48:24 <jlebon> (i.e. let's keep the ticket, but drop the label) 16:48:33 <dustymabe> yeah, I think there's less discuss needed on that one 16:48:48 <dustymabe> once we close off the SSSD ticket I'll drop the meeting label and close out the top level ticket 16:48:57 <dustymabe> so we won't bring it up any longer 16:49:25 <dustymabe> I think he may be AFK (didn't see him in roll call) so we'll punt on this one for now 16:49:31 <copperi[m]1> .hello2 16:49:32 <zodbot> copperi[m]1: Sorry, but user 'copperi [m] 1' does not exist 16:49:40 <dustymabe> #chair copperi[m]1 16:49:40 <zodbot> Current chairs: bgilbert copperi[m]1 dustymabe jaimelm jlebon lucab walters 16:49:49 <dustymabe> #topic Actually move iptables to the nft backend 16:49:54 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/676 16:50:28 <dustymabe> this came up again recently because we switched to f35 and they broke it out to an iptables-legacy package 16:50:54 <dustymabe> I thought we had done this before, but let's get agreement on the proposal in https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-824225675 and then execute 16:51:22 <dustymabe> migrate new nodes 16:51:23 <dustymabe> declare iptables-legacy in a deprecation period and provide instructions on how to migrate (possibly with doc links for e.g. additional OKD/k8s-specific instructions) 16:51:25 <jaimelm> is on board with jlebon's proposal. Did we define x? 16:51:25 <dustymabe> wait X months 16:51:27 <dustymabe> force migrate old nodes 16:51:29 <dustymabe> no longer consider iptables-legacy supported 16:51:33 <jlebon> was thinking about this, and it's a little trickier than the proposal makes it seem 16:51:42 <jlebon> the "migrate new nodes" part is the hard bit 16:52:03 <jlebon> i'm not very confident in the proposal at https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-732514979 16:52:18 <jlebon> so my proposal now is https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-732515968 16:53:11 <jlebon> notably, the first 2-barrier proposal is complex, and even then not foolproof 16:53:34 <jlebon> the steps would still roughly be: 16:53:43 <jlebon> - declare iptables-legacy in a deprecation period 16:53:47 <jlebon> - wait X months 16:53:58 <jlebon> - force migrate old and new nodes 16:54:07 <jlebon> so that'd be the key difference 16:54:30 <jlebon> (unless explicitly opted into not migrating) 16:55:03 <dustymabe> seems reasonable to me 16:55:22 <dustymabe> basically users have the ability to opt out if they touch the stamp file 16:55:26 <walters> I think there's two classes of user here; the "small scale non-kube" case where they may use iptables manually, or equally well may not (in cloud) versus kubernetes where it's an important part of networking setup 16:55:37 <jaimelm> ^^ 16:55:40 <dustymabe> obviously requires them to be engaged and reading notices 16:55:41 <walters> the forced migration seems like a risk for Typhoon 16:55:53 <walters> (and OKD?) 16:56:11 <jaimelm> Not so much for OKD 16:56:19 <jlebon> yeah, this approach would require e.g. Typhoon to touch the stamp file 16:56:23 <jaimelm> I ran it by the team a few months ago. Not a big concern. 16:56:40 <dustymabe> jlebon: good point. the integration platform could do it 16:56:41 <jlebon> then it's in their hands when they want to move to nft 16:57:40 <dustymabe> well hmm. trying to think about how easy that would be 16:58:02 <dustymabe> not exactly sure how updates to typhoon are handled 16:58:28 <walters> i thought the stamp file was about marking use of nft? is there another stamp file for "I want iptables-legacy"? 16:58:32 <jlebon> we can run the proposal by Dalton to see what he thinks 16:58:43 <dustymabe> yeah, +1 for running the proposal by Dalton 16:58:50 <lucab> to the best of my knowledge, the usual FCOS way through zincati 16:58:58 <jlebon> walters: in https://github.com/coreos/fedora-coreos-tracker/issues/676#issuecomment-732515968, the stamp file would be "i want to stay on legacy" 16:59:18 <dustymabe> lucab: but what about the platform on top? i.e. could they define new containers in a new version that put a file on the host? 16:59:21 <walters> ok 17:00:54 <lucab> dustymabe: in theory yes, but I think it would still require manual intervention as typhoon down not auto-upgrade itself 17:01:42 <dustymabe> #proposal 1. declare iptables-legacy in a deprecation period 2. allow people to opt out of automigration 3. wait 3 months 4. force migrate old and new nodes 17:01:56 <dustymabe> hmm - should have added in there talking with typhoon 17:02:07 <lucab> https://typhoon.psdn.io/topics/maintenance/#upgrades 17:02:23 <dustymabe> #proposal Talk with typhoon about proposed migration. The proposed migration is: 1. declare iptables-legacy in a deprecation period 2. allow people to opt out of automigration 3. wait 3 months 4. force migrate old and new nodes 17:02:50 <jlebon> +1 17:02:52 <dustymabe> jlebon: does that reflect your proposal? 17:02:59 <jlebon> yup 17:03:13 <dustymabe> +1 from my side 17:03:28 <copperi[m]1> +1 17:03:47 <jaimelm> +1 17:04:25 <dustymabe> walters lucab bgilbert.. look good? 17:04:29 <walters> +1 17:04:45 <dustymabe> #agreed Talk with typhoon about proposed migration. The proposed migration is: 1. declare iptables-legacy in a deprecation period 2. allow people to opt out of automigration 3. wait 3 months 4. force migrate old and new nodes 17:05:04 <walters> as a side note I'm in the "BPF is the future" camp and NFT vs iptables is...eh, not very interesting 17:05:19 * dustymabe wants to learn more about BPF - like a lot more 17:05:20 <bgilbert> wfm 17:06:04 <dustymabe> anybody want to reach out to dalton to check this plan once I add it to the ticket (jlebon might also be able to add a little more design detail about how the "opt-out" service would operate 17:06:27 <jlebon> dustymabe: was thinking mostly of pinging dalton on the ticket :) 17:06:42 <jlebon> i can do that if you'd like 17:06:45 <dustymabe> that works!! 17:07:37 <dustymabe> #action jlebon to add some design details to the proposal in the ticket and also reach out to dgubble to see if this can be handled in typhoon 17:07:51 <dustymabe> jaimelm: ^^ also worth confirming 100% with OKD this won't be a problem 17:07:59 <lucab> (my IRC client is having large latency/lagging issues) 17:07:59 <dustymabe> #undo 17:07:59 <zodbot> Removing item from minutes: ACTION by dustymabe at 17:07:37 : jlebon to add some design details to the proposal in the ticket and also reach out to dgubble to see if this can be handled in typhoon 17:08:17 <dustymabe> #action jlebon to add some design details to the proposal in the ticket (#676) and also reach out to dgubble to see if this can be handled in typhoon 17:08:25 <dustymabe> lucab: noted 17:08:29 <bgilbert> *dghubble 17:08:37 <dustymabe> bgilbert: boo 17:08:39 <dustymabe> #undo 17:08:39 <zodbot> Removing item from minutes: ACTION by dustymabe at 17:08:17 : jlebon to add some design details to the proposal in the ticket (#676) and also reach out to dgubble to see if this can be handled in typhoon 17:08:47 <dustymabe> #action jlebon to add some design details to the proposal in the ticket (#676) and also reach out to dghubble to see if this can be handled in typhoon 17:08:53 <dustymabe> better :) 17:08:57 <jaimelm> dustymabe: straight from the man himself... https://github.com/openshift/okd/discussions/613 17:09:07 <jaimelm> but I'll ask again 17:09:18 <dustymabe> +1 17:09:23 <jaimelm> He's OOTO but will be back next week-ish 17:09:29 <dustymabe> sound sgood 17:09:31 <dustymabe> thanks jaimelm 17:09:40 <dustymabe> #topic Handle re-invocations of coreos-installer on a new disk 17:09:44 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/976 17:09:47 * sumantrom is here 17:09:53 <dustymabe> bgilbert: want to set the stage for this one? 17:10:05 <bgilbert> not my ticket, but I'll be happy to chime in 17:10:13 <dustymabe> oops - let me check 17:10:20 <bgilbert> walters? 17:10:27 <dustymabe> +1 walters? 17:10:49 <walters> We've had a lot of problems I'm pretty sure are rooted in races on multiple disks with partitions that have `LABEL=boot` 17:11:29 <walters> It is *possible* for this to happen in cloud, in theory but I think it's mostly a metal thing 17:12:12 <walters> I haven't checked other distributions but at least the way Anaconda works for example it injects a uuid for `/boot` into both the kernel and grub cfg 17:12:43 <walters> (hmm actually Fedora Cloud presumably inherits this too) 17:13:30 <walters> anyways that's the intro 17:14:06 <jlebon> hmm, i'm not sure that's the case re. Anaconda setting it in the kernel kargs. though definitely seeing it in the grub config 17:15:35 <walters> right, sorry it's not on the kargs for Anaconda, it's in `/etc/fstab` - because most uses of Anaconda-generated-systems don't need /boot in the initramfs 17:15:38 <dustymabe> so basically the problem is when you have more than one disk attached to a system and both of them have a label of `boot` on one of the filesystems? 17:15:43 <jlebon> what's so insidious about it is that the way problems manifest it makes you question everything 17:15:48 <bgilbert> jlebon: +1 17:16:18 <bgilbert> I would tend to lean toward solutions that keep coreos-installer out of it, since that'd be another cloud vs. metal divergence 17:16:44 <walters> yeah, i worked on https://blog.verbum.org/2020/12/01/committed-to-the-integrity-of-your-root-filesystem/ partially because of this, which was a lot of fun and valuable for other reasons obviously 17:16:59 <walters> (thinking there was some race on shutdown/update around power failures) 17:17:06 <bgilbert> e.g. failing if we find multiple /boot partitions, or burning in a per-build FS UUID + adding it to kargs + replacing it with a generated UUID on first boot 17:17:32 <walters> failing in the OS boot, or failing in coreos-installer? 17:17:36 <lucab> the two cases I've seen so far are mostly human/hardware failures, and most of the pain is that it become apparent way too late (on the first upgrade) 17:17:38 <bgilbert> failing in the OS boot 17:18:03 <bgilbert> (which is fundamentally subject to races of course) 17:18:07 <dustymabe> we could fail in both places 17:18:18 <dustymabe> though coreos-installer isn't always run directly on the target hardware 17:18:26 <jlebon> i don't think we should fail in coreos-installer 17:18:30 <dustymabe> sometimes the disk is written to and then "moved" 17:18:43 <dustymabe> jlebon: what about a warning then? 17:18:44 <bgilbert> +1, we can't assume c-i is run on the target 17:18:46 <jlebon> https://github.com/coreos/coreos-installer/pull/642#issuecomment-930351108 17:19:13 <bgilbert> that warning will fire if c-i is run from inside FCOS, which feels a bit odd 17:19:26 <bgilbert> *an installed FCOS 17:19:35 <jlebon> yeah, maybe more of a "Hey, I noticed that ..." 17:20:04 <lucab> I do generally agree with the directions walters want to go (more uuids), but at the same time I find valuable that we can say "by-label/boot" 17:20:22 <dustymabe> +1 lucab 17:20:28 <walters> lucab: it'd be good to enumerate the things we know today are using the label 17:20:49 <walters> note the proposal is just *adding* and using a uuid, not dropping the label though 17:21:21 <lucab> I only did a quick grep through the initrd, and we use that for ordering units 17:21:54 <walters> (side thread: all agreed we do need boot=UUID=$x on the kernel cmdline unlike anaconda because we need access to it in the initramfs?) 17:22:14 <jlebon> kinda feels like it's going out of our way to try to be resilient something that most likely indicates something wrong that ideally should be rectified 17:22:15 <lucab> sure, but the underlying story is "we move to uuid so that there can be multiple 'boot' labels" 17:22:31 <walters> hmm side note, the RHEL FIPS stuff requires this which is why we already have https://github.com/openshift/os/blob/9cd6b20bf01aee14c4c96946b84b14dce35d5c87/overlay.d/05rhcos/usr/lib/dracut/modules.d/40rhcos-fips/rhcos-fips.sh#L58 17:22:35 <bgilbert> jlebon: +1 17:22:46 <walters> so this is basically moving that bit of FIPS to be default 17:22:46 <jlebon> we want to own the machine and we need users to know that too 17:23:23 <jlebon> if there's two boot partitions, is it possible they booted from the wrong device? 17:23:28 <bgilbert> what's the scope of the problem we're trying to solve? I can see several options: 1. bootloader disk doesn't match boot disk, 2. root doesn't match boot, 3. user booted not from the disk they thought 17:24:08 <bgilbert> e.g. it should be easy to fail on 2 17:24:41 <bgilbert> 2 rephrased: / and /boot come from different disks 17:24:42 <walters> hmm true... 17:25:28 <walters> it'd still be racy though 17:25:33 <bgilbert> it would 17:26:32 <lucab> we reached here through reported symptoms, so we basically only know about cases of 2. 17:27:11 <dustymabe> any ways to make it less racy? is there a moment in time where we are sure all fs with label boot should be up? 17:27:28 <dustymabe> even if it's late and we've already done some $work 17:28:18 <lucab> I guess when we say racy here we mean "somebody can add another 'boot' partition at any point at runtime" 17:28:18 <bgilbert> well, no. hotplug can happen at any time 17:28:43 <dustymabe> let's say we're up and we get a boot 17:28:52 <jlebon> hmm, we don't need to do any waiting. we could have a stamp in /boot and / which match each other and verify they match at mount time 17:28:58 <dustymabe> and then we mount a root fs - and it doesn't match (through some heuristic) 17:29:07 <dustymabe> jlebon: +1 17:29:45 <jlebon> we dynamically generate the stamp on first boot. so whatever devices we agreed on first boot, those are what they have to remain for the rest of its life 17:29:51 <bgilbert> jlebon: stamp isn't necessary if we check for a common backing device 17:29:59 <walters> yeah but we could also find /boot from the same block device 17:30:01 <jlebon> bgilbert: we support moving rootfs to a different device 17:30:07 <bgilbert> ...right 17:30:09 <walters> ok true 17:30:46 <dustymabe> time check 17:31:13 <walters> jlebon: can you drop the stamp proposal in the ticket for discussion point? 17:31:25 <dustymabe> sounds like we're coalescing (spelling) around trying to figure out a good way to error in this case rather than change our UUID design for this particular instance 17:31:28 <jlebon> sure thing! 17:31:55 <walters> by error do we actually fail the boot, or do we just loudly warn? 17:32:28 <dustymabe> #proposal attempt to find a good way to show the user an error and fail the boot rather than change our UUID design for this particular invalid setup. 17:32:30 <jaimelm> loudly warn 17:32:35 <lucab> fail the boot in initrd, on each boot? 17:32:43 <dustymabe> i was thinking people were leaning towards error 17:32:43 <bgilbert> walters: fail? it's a pretty bad race 17:33:04 <dustymabe> yeah basically it's a condition people would want to rectify 17:33:12 <jlebon> +1 to fail. way nicer error than other possible failure modes 17:33:23 <jaimelm> true 17:33:24 <jlebon> because at some point, boot will fail 17:33:27 <dustymabe> could we do even more? 17:33:43 <jlebon> with an obscure error like ostree failing to find the rootfs 17:33:43 <walters> dustymabe: like, wipe out the other disk(s) partitions automatically? 17:33:45 <dustymabe> so basially we detect boot/root matching but also CLHM if we find more than one part with boot label 17:33:54 <dustymabe> walters: ^^ 17:33:59 <lucab> dustymabe: format one of the boot partition, random choice :) 17:34:09 <dustymabe> let's say we get lucky and boot/root match on this boot 17:34:10 <jlebon> hehe 17:34:20 <dustymabe> but next boot it might fail to match (race) 17:34:31 <dustymabe> CLHM could help us show the user the potential problem 17:34:56 <dustymabe> just an idea, either way current proposal stands 17:34:57 <jlebon> openshift users probably don't see CLHM on every boot 17:35:02 <dustymabe> jlebon: good point 17:35:07 <dustymabe> votes? 17:35:07 <bgilbert> even if root/boot match, we should probably check for other boot partitions 17:35:15 <walters> In theory if we detect a mismatch, we could e.g. only delete/mark notable the invalid boot partition that we used 17:35:17 <bgilbert> we may not find them in time (racy) but we could fail if we did 17:35:19 <walters> and then reboot 17:35:25 <walters> that would lead to eventual reconcilation 17:35:33 <walters> without arbitrary waiting 17:35:42 <bgilbert> walters: mark ourselves invalid, or the other one? 17:35:49 <walters> ourselves, only if we detect it was invalid 17:36:08 <jlebon> unless we don't know if it's boot that's wrong or root 17:36:11 <lucab> it may be valid but still on the "other" disk 17:36:18 <dustymabe> hmm.. auto reconciliation here seems potentially problematic 17:36:22 <dustymabe> :) 17:36:30 <bgilbert> isn't it true that damage may have already been done? 17:36:37 <bgilbert> i.e. stabilizing on one outcome isn't enough 17:36:57 <walters> can you elaborate on "damage"? 17:36:59 * dustymabe is really enjoying this detailed discussion 17:36:59 <jlebon> it's mostly on first boot that we do a lot of stuff with /boot 17:37:05 <bgilbert> walters: update skew? 17:37:09 <dustymabe> sorry jaimelm for $time 17:37:11 <bgilbert> also, we don't know which disk the user intended us to use 17:37:29 <walters> agreed, this proposal is picking one arbitrarily 17:37:51 <dustymabe> #proposal attempt to find a good way to show the user an error and fail the boot rather than change our UUID design for this particular invalid setup. 17:37:55 <jlebon> can bootloader order change on each boot without explicit action? 17:37:56 <dustymabe> can we stick with this ^^ for now 17:38:08 <bgilbert> sure would be nice if we had a reliable way to get that info from the bootloader 17:38:24 <lucab> for reference, one of the reporter was happy enough to know there was a node problem coming from this duplicate partitions, and re-provision the node in a correct way 17:38:29 <jaimelm> for now, and figure out a helpful way to notify the user 17:38:35 <walters> jlebon: I think this depends, but for a server with e.g. two nvme drives both with EFI partitions, it might simply race 17:38:36 <jlebon> dustymabe: maybe let's keep chatting in the ticket for now? 17:38:48 <bgilbert> jlebon: +1 17:38:48 <dustymabe> jlebon: i.e. no agreed upon outcome? 17:38:52 <jlebon> yeah 17:38:55 <walters> yeah, continue discussion 17:39:00 <dustymabe> that's fine, but i'll atleast do a summary 17:39:04 <jaimelm> ^^ 17:39:07 <jlebon> +1 17:39:14 <dustymabe> well summary of the current proposed 17:39:18 <dustymabe> #topic open floor 17:39:23 <dustymabe> welcome all to open floor 17:39:26 <dustymabe> sorry about $time 17:39:43 <dustymabe> #info help wanted to chase down a containers-common issue in rawhide https://github.com/coreos/fedora-coreos-tracker/issues/981 17:40:02 <dustymabe> #info the `next` stream has been moved to Fedora Linux 35 content - please please test! 17:40:05 <pjones> walters: it should race and then /iterate/ 17:40:32 <pjones> walters: assuming you're talking about the case where no variables are configured 17:40:33 <dustymabe> any other topics for open floor? 17:41:23 <dustymabe> for the containers-common issue, don't feel like you need to be an expert.. we can help guide you 17:41:26 * jaimelm will look at 981 but is busy 17:41:43 <dustymabe> +1 17:41:45 * jaimelm isn't mediumxpert 17:41:56 <dustymabe> will close out the meeting in 60s unless new topic come up 17:42:49 <jaimelm> jlebon: reminder for the CI walkthrough - or anyone else that wants to give me the tour 17:43:09 <jlebon> jaimelm: let's chat in the channel 17:43:09 <dustymabe> #endmeeting