16:30:31 <dustymabe> #startmeeting fedora_coreos_meeting 16:30:31 <zodbot> Meeting started Wed May 12 16:30:31 2021 UTC. 16:30:31 <zodbot> This meeting is logged and archived in a public location. 16:30:31 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:30:31 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:31 <zodbot> The meeting name has been set to 'fedora_coreos_meeting' 16:30:36 <dustymabe> #topic roll call 16:30:39 <dustymabe> .hi 16:30:39 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com> 16:30:40 <bgilbert> .hi 16:30:40 <travier> .hello siosm 16:30:42 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net> 16:30:45 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com> 16:30:59 <jbrooks> .hello jasonbrooks 16:31:00 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com> 16:31:16 <lucab> .hello2 16:31:16 <darkmuggle> .hello 16:31:16 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com> 16:31:19 <zodbot> darkmuggle: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1". 16:31:38 <darkmuggle> .hello2 16:31:38 <zodbot> darkmuggle: darkmuggle 'None' <me@muggle.dev> 16:32:04 <jlebon> .hello2 16:32:05 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com> 16:32:05 <fifofonix> .hello2 16:32:08 <zodbot> fifofonix: fifofonix 'Fifo Phonics' <fifofonix@gmail.com> 16:33:47 <dustymabe> #chair travier jbrooks lucab darkmuggle jlebon fifofonix 16:33:47 <zodbot> Current chairs: darkmuggle dustymabe fifofonix jbrooks jlebon lucab travier 16:34:05 <dustymabe> #topic Action items from last meeting 16:34:31 <dustymabe> looks like we had a clean slate. No action items from last time. 16:34:51 <skunkerk> .hello sohank2602 16:34:55 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com> 16:34:57 <dustymabe> #chair skunkerk 16:34:57 <zodbot> Current chairs: darkmuggle dustymabe fifofonix jbrooks jlebon lucab skunkerk travier 16:35:22 <dustymabe> I'll go to meeting tickets. Please add the meeting label or otherwise comment in a ticket if you'd like to have it brought up in a meeting. 16:35:32 <dustymabe> #topic Mounts permission denied starting in 33.20210426.3.0 16:35:38 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/818 16:35:59 <dustymabe> I wanted to bring this up to raise awareness 16:36:28 <dustymabe> the most succinct and relevant information is in this comment: https://github.com/coreos/fedora-coreos-tracker/issues/818#issuecomment-834539146 16:37:09 <dustymabe> unfortunately we didn't catch this before it hit our stable stream 16:37:25 <dustymabe> even people running `next`/`testing` woulnd't have seen this unless they deployed fresh nodes 16:37:38 <dustymabe> OR placed new containers that needed this on existing nodes 16:38:34 <dustymabe> podman upstream has landed a fix, but they're working through release candidate releases for podman 3.2 and not likely to do a backport to 3.1 16:39:02 <dustymabe> if we had caught it in `testing` we would have froze podman (probably) 16:40:08 <dustymabe> hopefully we get the podman 3.2 release into next week's `next`/`testing` builds 16:40:17 <travier> We have had a lot of podman issues recently 16:40:33 <travier> wondering how we could improve test coverage 16:40:43 <dustymabe> the pace of development is pretty high in podman land 16:41:02 <travier> which is good 16:41:21 <travier> but feels like test coverage is not there 16:41:26 <dustymabe> travier: there's nothing like actual users running `next`/`testing` 16:41:31 <fifofonix> fifofonix: (note to self, improve lower env testing by scheduling replacement of at least some nodes) 16:41:39 <walters> .hello2 16:41:40 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com> 16:41:42 <dustymabe> but in this case it was tricky 16:42:00 <dustymabe> because if it was an existing node the labels were already correct 16:42:38 <dustymabe> travier: I think we can add tests for things as they come up. like we could add a test for this now, but I did get dan to add a test upstream in podman to try to make sure it doesn't go the other way again 16:42:57 <dustymabe> #chair walters 16:42:57 <zodbot> Current chairs: darkmuggle dustymabe fifofonix jbrooks jlebon lucab skunkerk travier walters 16:43:01 <travier> I don't think it's on us. I've requested release testing for toolbox upstream 16:43:07 <travier> for example 16:43:11 <dustymabe> +1 16:43:48 <dustymabe> #info we hit a podman bug in our stable stream. Fix will come in the podman 3.2 release. Details in https://github.com/coreos/fedora-coreos-tracker/issues/818#issuecomment-834539146 16:43:56 <travier> #link https://github.com/containers/podman/issues/10296 16:44:17 <dustymabe> anthing else we'd like to investigate? 16:44:39 <jlebon> there's also another podman bug we're currently avoiding by pinning to 3.1.2 16:44:54 <jlebon> so whenever we unpin we need to make sure all the fixes we need are in 16:45:10 <dustymabe> jlebon: +1 16:45:17 <jlebon> #link https://github.com/coreos/fedora-coreos-config/pull/1002 16:45:26 * dustymabe moving on to next meeting item in 30s 16:45:28 <fifofonix> more broadly than podman issue, how thin is 'next', 'testing' use? is there merit in shining a light on importance of it? 16:45:55 <dustymabe> fifofonix: that's a good question. I don't know the answer to that 16:46:08 <dustymabe> there's always merit in shining a light of importance on it IMHO 16:46:11 <bgilbert> +1 16:46:44 <fifofonix> personally i'm interested in knowing more about FCOS usage stats although that may be closely guarded? 16:46:48 <dustymabe> jlebon: can you open that PR against `next-devel` too? 16:47:11 <dustymabe> fifofonix: not closely guarded. though we don't have a lot of data right now 16:47:13 <jlebon> hmm, will the countme data give us per-stream data? 16:47:17 <dustymabe> see https://fedoramagazine.org/getting-better-at-counting-rpm-ostree-based-systems/ 16:47:25 <jlebon> i guess only if it's on a different releasever 16:47:36 <jlebon> dustymabe: ack 16:48:36 <travier> We will only get numbers about how many users are running a given "Fedora release" and since how long 16:49:05 <dustymabe> ok moving on to next ticket 16:49:08 <travier> won't be including the stream they are on unfortunately 16:49:22 <travier> +1 16:49:41 <dustymabe> #topic Platform Request: PowerVS (Power Virtual Server) 16:49:47 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/817 16:50:18 <dustymabe> Looks like a request for a new hardware platform (Power PC architecture in IBM Cloud) 16:50:48 <dustymabe> bgilbert: seems like you've had some dialogue. Any advice how to proceed here? 16:51:43 <bgilbert> AFAICT this is a routine platform request at this point. I think I agree that `powervs` is a reasonable platform name, and we've been told that the closed-source agent is not mandatory. 16:52:03 <bgilbert> so it should just be a matter of getting the remaining details and implementing the usual things 16:53:13 <dustymabe> yeah, including a ppc64le version of FCOS to be useful 16:53:18 <bgilbert> well, yeah 16:53:25 <dustymabe> :) 16:54:13 <jlebon> so for now this would be just about RHCOS then, right? 16:54:22 <dustymabe> Anybody have any concerns about supporting this platform in our tooling? 16:54:55 <dustymabe> jlebon: I think the goal for them is OpenShift, so yes, but I appreciate trying to be upstream first 16:55:34 <dustymabe> though, if we have some 'ppc machines running in the cloud' then we could possibly do pipeline builds on them and actually have ppc64le content 16:55:37 <lucab> I'm mildly concerned because there are no real references in that thread, and IBM Cloud Gen1 was a mess from a networking point of view 16:55:54 <jlebon> dustymabe: oh yeah, definitely. we should clarify in the ticket though that the outcoem for this does not include outputing FCOS media 16:56:44 <dustymabe> bgilbert: do you have the same concerns as lucab ? 16:57:25 <bgilbert> I'll happily defer to lucab on ibmcloud networking :-) 16:57:52 <lucab> but I guess we'll only discover going forward. If powervs actually has proper DHCP, then it's probably good. 16:58:31 <bgilbert> jlebon: wouldn't we want to produce FCOS media? 16:58:38 <dustymabe> ok. any action for us on this ticket right now? I notice they have an open PR. who can properly review that? 16:59:33 <jlebon> bgilbert: eventually yes once we're multi-arch enabled 16:59:47 <bgilbert> +1 17:00:42 <dustymabe> #info powervs as a new platform seems reasonable 17:00:54 <dustymabe> good with that ^^ ? 17:01:21 <bgilbert> +1 17:01:21 <jlebon> that #info seems reasonable 17:01:29 <lucab> dustymabe: seems reasonable 17:01:33 <dustymabe> :) 17:01:55 <dustymabe> if there is someone who would like to volunteer to work with them it would probably make things go smoother 17:02:02 <lucab> (jlebon is too fast) 17:02:24 <jlebon> :) 17:02:27 <dustymabe> they have an open PR and it's probably best if we don't just let that linger in "unknown status" land 17:03:11 * dustymabe moves on to next meeting topic soonish 17:04:06 <dustymabe> #topic Cannot upgrade from N-2 releases due to missing GPG key 17:04:12 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/749 17:04:40 <dustymabe> we discussed this last week and I think the discussion lead us to a few options 17:04:53 <dustymabe> jlebon summarized in https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-832975742 17:05:15 <dustymabe> Sounds like 3. seemed reasonable enough and he added in support to rpm-ostree for that 17:05:33 <dustymabe> remaining item is: https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-834801971 17:06:13 <dustymabe> so the question is whether the logic for that bit should live in zincati or rpm-ostree 17:06:23 <jlebon> spoke to lucab about this, and he had a variation on this which was cleaner 17:06:26 <jlebon> let me make a comment 17:06:31 <dustymabe> +1 17:07:46 <jlebon> https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-839947093 17:08:13 <dustymabe> jlebon: ok, so no real issue/question where the logic should reside? 17:08:29 <travier> my concerns are pretty much as summarized by walters in https://github.com/coreos/rpm-ostree/pull/2819#pullrequestreview-656121731 17:09:40 <dustymabe> hmm. but the commit we actually deploy is gpg verified, right? 17:10:16 <travier> not necessarily afaik 17:10:35 <jlebon> this is not turning off GPG verification 17:10:35 <walters> another thing I forgot about is we have the same signing key across editions, so I think this opens up MITM rebasing (again omitting TLS) between them 17:11:28 <dustymabe> but zincati is going to do the check? 17:11:44 <travier> what check? 17:11:49 <dustymabe> as suggested here: https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-839947093 17:11:53 <travier> zincati is not checking GPG signatures 17:11:58 <walters> no, rpm-ostree does gpg verification via the remote config 17:12:09 <dustymabe> travier: the branch check 17:12:27 <lucab> to my understanding, the commits themselves are signed and there are multiple levels of rollback prevention. Zincati can check the "on same stream" condition 17:12:32 <travier> that does not resolve the gpg check issue 17:13:06 <jlebon> again, to be sure this is understood, GPG verification is still done as usual 17:13:40 <jlebon> libostree still verifies everything it pulls before continuing, and we're not changing that 17:14:23 <jlebon> all this is changing is allowing zincati to tell rpm-ostree to deploy to a specific commit without verifying that it's on the same branch the node is on 17:14:45 <dustymabe> and then zincati will do the branch verification (in a different way) 17:15:07 <travier> then I don't understand how that helps us with GPG verifications errors 17:15:10 <jlebon> but IMO branches are mostly an implementation detail, and we wouldn't be discussing this if we matched the MCO and did pinned commits 17:15:40 <dustymabe> travier: when rpm-ostree does the branch verification it walks the history from HEAD 17:15:41 <walters> i agree with that 17:16:12 <dustymabe> if HEAD is signed with an f34 key and you are on f31 then the gpg verification of the HEAD commit will fail 17:16:26 <dustymabe> (it verifies the commits as it walks the tree) 17:17:16 <dustymabe> what we're changing now is just telling it not to worry about branch verification in rpm-ostree itself, so it won't attempt to walk the tree 17:17:42 <travier> so this still means that we need release barriers for every fedora releases 17:17:46 <travier> to get the new keys 17:17:55 <dustymabe> N-2, yes 17:17:56 <jlebon> for more information, this is actually re-enabling something that rpm-ostree used to do by default: https://github.com/coreos/rpm-ostree/pull/495 17:18:11 <lucab> travier: yes, https://docs.fedoraproject.org/en-US/fedora-coreos/update-barrier-signing-keys/ 17:18:43 <travier> because we did not do that for the latest round of release for the F34 rebase 17:18:57 <dustymabe> right, we decided not to because of this very issue we're discussing 17:19:00 <dustymabe> :) 17:19:05 <travier> yes :) 17:19:15 <jlebon> it's useless right now, because the code for it won't make it into nodes which need it until we at f36 or something :) 17:19:39 <travier> so this is what I was confused about. Sounds reasonable to me now 17:19:45 <lucab> we can retro-add barriers at any time 17:19:46 <travier> oh, indeed 17:19:49 <dustymabe> ok i'll try to wind this topic down 17:19:59 <walters> though hmm actually...ostree has "ref bindings" from https://github.com/ostreedev/ostree/commit/cf16805a2f70026f24eb0b245666efcd4a7c8e44 but...I think rpm-ostree never used them 17:20:35 <walters> which is probably just an oversight 17:20:39 <dustymabe> the one thing we haven't decided (which we brought up last time) is "how old of nodes do we care about?" - will open a separate issue for that if there isn't already one 17:20:45 <walters> but they're clearly better than walking the commit history 17:21:11 <dustymabe> the outcome of that topic will help inform our decision on "what do we do about current really old nodes hitting this problem?" 17:21:19 <jlebon> walters: yeah, although it doesn't guarantee either that the commit comes lives on that branch 17:21:39 <walters> it does because the signature covers that metadata 17:21:51 <lucab> dustymabe: those need a manual intervention anyway 17:21:57 <walters> anyways i'll follow up there on GH 17:22:23 <dustymabe> lucab: yeah 17:22:53 <jlebon> walters: hmm, not following 17:23:06 <dustymabe> anything we could do on the zincati side to "deadend" them so maybe they would get a message ? 17:23:10 <jlebon> but yeah, we can chat on GH (I did look at ref-binding in https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-781453429) 17:23:36 <lucab> dustymabe: I'd say we first fix the issue and the post instructions to manually help stuck nodes 17:23:43 <dustymabe> k 17:23:54 * dustymabe flips to open floor 17:23:57 <dustymabe> #topic open floor 17:24:32 <dustymabe> Timothee is going to post an email to coreos-status soon about CountMe enablement in FCOS 17:24:45 <walters> I have one on https://github.com/coreos/fedora-coreos-tracker/issues/828 ! 17:24:50 <lucab> dustymabe: we can mark deadends yes, but very ancient nodes may not yet have the logic to show that to MOTD 17:25:02 <dustymabe> lucab: yeah, we'd have to investigate if it was possible 17:25:14 <walters> So far it's going pretty well, just wanted to highlight it. I am working on supporting the vmware ova which is the last FCOS artifact not handled (in upstream git master) 17:25:21 <dustymabe> walters: want to add a `meeting` label so we bring it up next time? 17:25:43 <walters> yeah we can talk about it next week 17:26:00 <lucab> dustymabe: I think it may be better not to mark deadends and see if "wget new GPG key to /etc" is enough to bring them back into the usual flow 17:26:08 <jlebon> walters: is this providing bit-for-bit matching of the uncompressed artifacts? 17:26:22 <walters> yes-ish 17:26:35 <jlebon> hehe 17:26:43 <bgilbert> most-bits-for-bit? 17:26:53 <walters> gcp is missing https://github.com/coreos/coreos-assembler/pull/2158 to validate 17:26:53 <bgilbert> :-) 17:27:01 <walters> the ISO for example is indeed bit-for-bit validated 17:27:18 <walters> but that's simple because we ship the squashfs which is 95% of the ISO 17:27:36 <dustymabe> i wonder if this topic would be a good candidate for a video meeting 17:27:37 <travier> lucab: agree with GPG key import 17:27:40 <walters> artifacts which have "internal compression" are harder (AWS and vmware have VMDK) 17:27:58 <dustymabe> seems like a lot of details to work through that could be good for higher bandwidth (and maybe a short presentation) 17:28:00 <travier> dustymabe: good idea, a rehydrator demo would be great! 17:28:06 <walters> yeah agree 17:28:10 <travier> (saying the guy not making the demo :D) 17:28:21 <bgilbert> #info Git repository default branches have been renamed to main; see the coreos@l.fp.o post for a list of repos and instructions on updating local checkouts 17:28:24 <bgilbert> #link https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/thread/RLSS4ZUMAUL7ZPZPFTZERSSBWGZFM4VU/ 17:28:31 <walters> nice work on that everyone! 17:28:32 <dustymabe> we could either wait for the first meeting in June (scheduled video meeting) OR schedule one ad-hoc for next week 17:28:43 <dustymabe> bgilbert++ 17:28:43 <zodbot> dustymabe: Karma for bgilbert changed to 1 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:28:46 <dustymabe> travier++ 17:28:51 <bgilbert> dustymabe++ 17:28:51 <zodbot> bgilbert: Karma for dustymabe changed to 2 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:28:51 <dustymabe> thanks for working on the branch renaming 17:28:54 <travier> dustymabe++ 17:28:54 <zodbot> travier: Karma for dustymabe changed to 3 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:29:08 <dustymabe> 🍪🍪🍪🍪 17:29:15 <jlebon> agreed, nice work! 17:29:18 <bgilbert> travier++ 17:29:18 <zodbot> bgilbert: Karma for siosm changed to 2 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:29:42 <travier> I guess making video meetings tied to a schedule was not a good idea 17:29:43 <bgilbert> and thanks to everyone else who helped! 17:29:54 <travier> let's make an ad-hoc one 17:29:58 <dustymabe> WFM 17:30:06 <travier> bgilbert++ 17:30:06 <zodbot> travier: Karma for bgilbert changed to 2 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:30:20 <dustymabe> travier: do you want to organize and run it next week (or even the following week)? 17:30:43 <travier> Not making any promise for the upcoming weeks (moving, etc.) 17:30:54 <dustymabe> busy busy 17:31:03 <dustymabe> ok, well we'll see what comes naturally then 17:31:16 <travier> would prefer someone else do it :) 17:31:23 <dustymabe> anything else for open floor? 17:31:48 * dustymabe goes to create a video meeting label 17:32:39 <travier> #link https://fedoramagazine.org/getting-better-at-counting-rpm-ostree-based-systems/ 17:32:54 <dustymabe> thanks travier for working on that ^^ 17:33:19 <bgilbert> +1 17:33:27 <dustymabe> #endmeeting