16:29:57 <bgilbert> #startmeeting fedora_coreos_meeting
16:29:57 <zodbot> Meeting started Wed Mar 10 16:29:57 2021 UTC.
16:29:57 <zodbot> This meeting is logged and archived in a public location.
16:29:57 <zodbot> The chair is bgilbert. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:29:57 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:57 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:04 <bgilbert> #topic roll call
16:30:08 <bgilbert> .hello2
16:30:09 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:49 <jlebon> .hello2
16:30:50 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:23 <cverna> hello o/
16:31:55 <dustymabe> .hello2
16:31:56 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:32:39 <PanGoat> .hello jaimelm
16:32:39 <bgilbert> #chair jlebon cverna dustymabe
16:32:39 <zodbot> Current chairs: bgilbert cverna dustymabe jlebon
16:32:39 <zodbot> PanGoat: jaimelm 'Jaime Magiera' <jaimelm@umich.edu>
16:32:45 <bgilbert> #chair PanGoat
16:32:45 <zodbot> Current chairs: PanGoat bgilbert cverna dustymabe jlebon
16:33:02 <jbrooks> .hello jasonbrooks
16:33:03 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:33:33 <jlebon> there's the container plumbing days virtual conference going on right now, so we might have less people than usual
16:33:51 <darkmuggle> .hello2
16:33:52 <zodbot> darkmuggle: darkmuggle 'None' <me@muggle.dev>
16:34:01 <bgilbert> #chair jbrooks darkmuggle
16:34:01 <zodbot> Current chairs: PanGoat bgilbert cverna darkmuggle dustymabe jbrooks jlebon
16:34:13 <bgilbert> #topic Action items from last meeting
16:34:24 <bgilbert> - bgilbert to investigate FCCT check for too-small rootfs
16:34:42 <bgilbert> #info bgilbert filed https://github.com/coreos/fcct/issues/211, will track the issue there
16:34:52 <bgilbert> - lucab to refresh the existing nm-cloud-setup PR, dropping the auto-enable part
16:35:02 <bgilbert> looks like this hasn't been done yet
16:35:06 <bgilbert> #action lucab to refresh the existing nm-cloud-setup PR, dropping the auto-enable part
16:35:17 <bgilbert> - lucab to track the nm-cloud-setup kola testing in a ticket and followup on that
16:35:20 <bgilbert> likewise
16:35:22 <bgilbert> #action lucab to track the nm-cloud-setup kola testing in a ticket and followup on that
16:35:28 <bgilbert> - travier to note that we generally like the tiers idea, and that we'll figure out the details in a followup
16:35:44 <walters> .hello2
16:35:45 <bgilbert> #info travier updated https://github.com/coreos/fedora-coreos-tracker/issues/738
16:35:45 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:35:46 <travier> .hello siosm
16:35:50 <bgilbert> #chair walters travier
16:35:50 <zodbot> Current chairs: PanGoat bgilbert cverna darkmuggle dustymabe jbrooks jlebon travier walters
16:35:51 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:36:03 <bgilbert> - jbrooks to note the result of the twitter poll, and followup for logistic steps
16:36:12 <bgilbert> #info jbrooks posted https://github.com/coreos/fedora-coreos-tracker/issues/695#issuecomment-791733875
16:37:04 <bgilbert> #topic Create confidence/testing tiers for platforms supported by Fedora CoreOS
16:37:12 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/738
16:37:23 <bgilbert> travier: this still has the meeting label; do we want to continue the discussion?
16:37:41 <travier> We can if people think we need to
16:37:51 <travier> Last time we were over time
16:38:43 <dustymabe> no opinion here
16:38:49 <bgilbert> same
16:39:20 <travier> I've done naming changes and updated the description
16:39:43 <travier> We could then decide that:
16:39:43 <travier> Tier 3 platform artifacts are only built after Tier 1 & 2 ones are done for a release and failures are non blocking nor critical which would reduce release overhead.
16:39:43 <travier> Tier 2 & 3 platform artifacts are not built by CI for testing PRs or for testing / mechanical streams which would reduce CI overhead.
16:39:52 <travier> How do people feel about that?
16:40:31 <dustymabe> No strong opinions. LGTM
16:40:33 <travier> Maybe that's step 3 and step 2 is how do we present that to the community?
16:40:47 <bgilbert> would we ship tier 3 images with CI failures, or skip them?
16:41:41 <bgilbert> sorry, looks like only tier 1 has CI
16:41:47 <travier> yes
16:42:00 <travier> if you have CI you're tier 1
16:42:02 <jlebon> travier: re. 'Tier 2 & 3 not built by CI for testing PRs' i'd make an exception for coreos-assembler and f-c-c CI, otherwise agreed!
16:42:12 <PanGoat> yes
16:42:40 <bgilbert> okay, rephrasing: what happens to the stream metadata if a tier 3 artifact doesn't build?
16:43:18 <bgilbert> I think the correct answer is "it sticks on the old release", and I think that requires code changes
16:44:13 <jlebon> it shouldn't really fail to build though (that dovetails into what i mentioned above re. CI)
16:45:05 <dustymabe> bgilbert: so possible pre-requisite for this is slight modification to our tooling to handle this case?
16:45:08 <jlebon> i'd say it depends on the failure.  assuming it's covered in cosa's CI, it shouldn't be hard to pinpoint what's breaking it
16:45:39 <bgilbert> IMO we should either revise the policy to make tier 3 build failures release blocking, or have a plan for releasing without them
16:46:09 <bgilbert> dustymabe: I'm not sure about "slight".  the stream metadata generator would need to understand history.
16:46:53 <dustymabe> cool. thanks for the correction
16:46:58 <bgilbert> there's also the policy question of whether we e.g. continue shipping updates to a platform where the current release images are broken
16:47:31 <cverna> in practice how many times a tier 3 platforms fails  to build when tier 1&2 pass ?
16:47:37 <bgilbert> right
16:47:52 <bgilbert> if we don't anticipate having this problem, maybe we shouldn't design for it yet
16:48:00 <jlebon> IMO, it's not worth worrying about this (i.e. I'd stick with the status quo of making them release blocking)
16:48:09 <bgilbert> i.e. make all images release blocking until such time as that becomes a problem
16:48:09 <bgilbert> yeah
16:48:14 <cverna> jlebon: +1
16:49:45 <bgilbert> travier: do you want us to reach a conclusion in this meeting?
16:49:51 <jlebon> so for presenting to users... are we thinking just a table in https://docs.fedoraproject.org/en-US/fedora-coreos/ ?
16:50:03 <jlebon> + announcing it on the list probably
16:50:04 <bgilbert> travier: about the "we could then decide" part
16:50:16 <PanGoat> A simple table is good, announcing with a link to it.
16:50:30 <bgilbert> jlebon: we'd probably want some indication on the download page, right?
16:50:43 <bgilbert> and on the getting-started docs page for each platform
16:50:45 <PanGoat> above the table, a clear explanation of what it actually means
16:50:51 <jlebon> bgilbert: good point
16:52:21 <bgilbert> okay, anyone want to propose anything?  otherwise we can just update the ticket with the discussion and move on
16:52:40 <travier> bgilbert: should be good. I don't think we have immediate actions
16:52:44 <bgilbert> +1
16:53:01 <cverna> +1 to move on
16:53:06 <travier> +1
16:53:07 <PanGoat> onward
16:53:13 <lorbus> +1
16:53:15 <bgilbert> #topic Help from Fedora QA
16:53:26 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/762
16:53:30 <bgilbert> #chair lorbus
16:53:30 <zodbot> Current chairs: PanGoat bgilbert cverna darkmuggle dustymabe jbrooks jlebon lorbus travier walters
16:53:32 <bgilbert> cverna?
16:53:41 <cverna> o/
16:54:16 <cverna> So sumantro from Fedora QA reached out and offered to help with FCOS
16:55:09 <cverna> In particular he has a good knowledge of how things are done in Fedora and he wanted to look for opportunities to have better collaboration
16:55:57 <cverna> I suggested that it would be good for him to help with running releases since that would make him understand how FCOS releases works
16:56:26 <cverna> but I am not sure how we decide who is added in the release executors ?
16:56:58 <dustymabe> I don't think we have any formal requirements - other than the people need appropriate credentials for certain things
16:57:18 * dustymabe notes it would be nice to have kevin or mohan run through a release one time too
16:57:49 <walters> no objections from me
16:57:51 <jlebon> more release executors = win!
16:58:13 <bgilbert> sgtm
16:58:30 <bgilbert> cverna, can you coordinate?
16:58:32 <walters> /giphy picard make it so
16:58:37 <travier> +1
16:58:45 <cverna> yes :)
16:59:01 <cverna> I ll work with him on getting the pre-requisites
16:59:08 <jlebon> maybe sumantro can then also help us with hooking kola into bodhi :)
16:59:15 <bgilbert> #action cverna to coordinate adding sumantro to release executors
16:59:44 <cverna> jlebon: that's a topic we discussed but I think it is wise to start with baby steps :P
16:59:52 <jlebon> cverna: +1
17:00:23 <bgilbert> cool, any other pieces to discuss here?
17:01:10 <bgilbert> cverna ^
17:02:13 <cverna> nope
17:02:16 <bgilbert> +1
17:02:18 <cverna> sorry :)
17:02:22 <bgilbert> np
17:02:28 <bgilbert> anyone have other tickets to discuss before open floor?  this is the last one with `meeting`
17:03:31 <bgilbert> #topic Open Floor
17:03:51 <jlebon> there's an email from Matt on the coreos list
17:03:56 * jlebon finds links
17:03:59 <PanGoat> I'll be getting back to #239 after this upcoming OKD event.
17:04:11 <jlebon> https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/thread/A6QLMB2SYMNCROUIZDCJ7UGLCSYN3EUW/
17:04:15 <PanGoat> (which folks are encouraged to attend)
17:04:26 <bgilbert> #link https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/thread/A6QLMB2SYMNCROUIZDCJ7UGLCSYN3EUW/
17:04:35 <jlebon> he wants to know if we have any talking points for the f34 beta
17:05:03 <jlebon> which goes into our discussions recently about changing the process for FCOS
17:05:07 <cverna> we are not really following this release cycle tho :P
17:05:24 <dustymabe> cverna: maybe it's just a chance for us to highlight the big changes we've made since f33
17:05:35 <dustymabe> even if we've already released them into the existing streams
17:05:49 <cverna> dustymabe: yeah sound good
17:05:49 <PanGoat> +1 dustymabe
17:05:54 <PanGoat> hype it
17:05:59 <walters> Yeah, I am not aware of any features held for f34 that we have now
17:06:36 <cverna> for example I don't think we talked advertised things like bootup
17:06:44 <bgilbert> doesn't seem to be anything listed in https://github.com/coreos/fedora-coreos-tracker/issues/704
17:07:10 <jlebon> cverna: right yeah. even if it's not a match, we should use the platform :)
17:07:34 <bgilbert> hmm, I have some reservations about saying "feature X already exists" in a message about _upcoming_ features
17:08:00 <dustymabe> bgilbert: yeah. Maybe it's a chance to do some education about how the releases for FCOS works?
17:08:08 <bgilbert> dustymabe: +1
17:08:18 <dustymabe> we could discuss this proposal with mattdm and see what he thinks
17:08:24 <PanGoat> nice
17:08:39 <bgilbert> we could possibly combine both even.  "FCOS has a rolling release structure.  recent changes in the rolling release include..."
17:09:03 <dustymabe> WFM
17:09:07 <jlebon> yeah SGTM
17:09:11 <cverna> +1
17:09:41 <bgilbert> anyone want to take an action?
17:09:45 <jlebon> should we hash that out right now in a hackmd?
17:09:50 <PanGoat> ^^
17:09:59 <bgilbert> sure
17:10:09 <jbrooks> I have a separate open floor item
17:10:09 <jlebon> https://hackmd.io/ppvBI50IQ_KQw0xCJrtM4w
17:10:16 <bgilbert> jbrooks: go ahead
17:10:22 <jbrooks> Red Hat Summit is coming up, and we'd like to have a demo video about Fedora CoreOS available for the Fedora booth. That's due Mar 16.
17:10:50 <jbrooks> Do we have something already done that we could use?
17:11:39 <PanGoat> The video is due by then or the committment to providing a video?
17:11:56 <jbrooks> The video
17:12:05 <bgilbert> jbrooks: I'm not aware of anything existing
17:12:09 <jlebon> there's dustymabe's recording where we showcase the install flow
17:12:40 <jbrooks> Is this on his blog?
17:12:41 <jlebon> i don't think there's a video just showing a booted FCOS and e.g. zincati, rpm-ostree, podman, etc...
17:12:43 <dustymabe> I think josh berkus has created a video for kubecon (that he cut down from a preso/demo I did) in the past
17:12:53 <jbrooks> I'll ask Josh about that
17:13:09 <PanGoat> jlebon: that would be ideal
17:13:20 <dustymabe> jbrooks: there's this also, but it's a bit more targeted on a specific feature: https://dustymabe.com/2020/11/18/coreos-install-via-live-iso-copy-network/
17:13:22 <walters> To me though it's not just about the in-OS tools, it's also the whole concept that e.g. we have stream metadata that includes AMIs that we tested end to end
17:13:23 <PanGoat> what are the constraints on the video? (duration, etc.)
17:13:32 <bgilbert> walters: +1
17:13:35 <walters> IOW the video should aspects of https://getfedora.org/en/coreos/
17:13:54 <dustymabe> yeah, I don't know if we have anything specific to this use
17:14:12 <PanGoat> Start with the overview like https://getfedora.org/en/coreos/, then transition into the booted OS
17:14:25 <jbrooks> mp4, 1GB size limit, 16:9 ratio -- those are the constraints they're listing
17:14:26 <bgilbert> seems like FCOS is stable enough now that anything we create could be reused for a while
17:14:39 <PanGoat> ^^
17:15:26 <PanGoat> hackmd of talking points?
17:15:50 <jbrooks> I'll follow up on the video
17:16:00 <bgilbert> jbrooks: +1
17:16:12 <bgilbert> feel free to file a tracker ticket for coordination
17:16:26 <jbrooks> ack
17:16:52 <travier> Would be interested if doing the video if need be :)
17:16:58 <travier> in doing*
17:17:01 <PanGoat> same
17:17:11 <jbrooks> awesome
17:17:15 <travier> 👍
17:17:37 <jlebon> travier++ PanGoat++ jbrooks++
17:17:42 <bgilbert> cool
17:17:58 <bgilbert> re the beta notes, anything else we want to discuss during the meeting itself?
17:19:13 <bgilbert> if not, and unless anyone else has open floor items, I'll close the meeting in two minutes
17:19:37 <travier> 👍
17:19:56 <lorbus> +1. Thanks for hosting bgilbert++
17:19:57 <zodbot> lorbus: Karma for bgilbert changed to 4 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:20:09 <walters> I have one open floor topic
17:20:13 <bgilbert> walters: go ahead
17:20:16 <walters> https://hackmd.io/GPB-x3XHToiYHdEnYNA1kA
17:21:31 <walters> In particular, I'd like to start making use of a portion of the budget OpenShift spends on cloud CI to also cover parts of FCOS and openshift/os - adding things like `/test azure` to Ignition, afterburn and the config git.
17:22:26 <bgilbert> hmm, this proposal lists some components but doesn't really say how they'd come together
17:22:51 <bgilbert> what's the overarching idea?
17:22:53 <walters> For sure, there are some important details here
17:23:39 <walters> Well to start the overarching idea is this is actually the current state, we have at least these 3 overlapping CI systems (plus Travis in a few cases)
17:24:03 <walters> I'd like to drop Travis (going to be a bit painful for ostree)
17:24:15 <bgilbert> yeah, we've been heading in the direction of dropping Travis
17:24:59 <walters> Then the idea is we just formalize a bit that we move forward with these 3 - there's overlap/duplication but we iterate on that
17:25:02 <lorbus> I do like the idea of doing for in Prow. E.g. we could trigger an OKD machine-os rebuild for each FCOS build, and actually run some OKD E2E tests for FCOS there
17:25:08 <darkmuggle> If we're dropping CI systems, I'd like to go all in on Prow for OS builds.
17:25:21 <walters> yes, we absolutely could add `/test e2e-okd-aws` to fedora-coreos-config
17:25:24 <bgilbert> FYI you're going to get some pushback on Prow
17:25:51 <walters> Oh for sure, I am also conflicted on Prow
17:26:20 <bgilbert> though your point about OpenShift CI resources is a good one
17:27:07 <walters> Right, that feature is really a key distinction
17:27:17 <lorbus> * I do like the idea of doing more in Prow. E.g. we could trigger an OKD machine-os rebuild for each FCOS build, and actually run some OKD E2E tests for FCOS there
17:27:19 <darkmuggle> For OS builds, having prow would massively benefit both RHCOS/OCP and FCOS/OKD.
17:27:34 <PanGoat> +1 lorbus
17:27:35 <walters> I think extracting the creds from Prow to somewhere else is *theoretically* possible but I don't want to go there myself
17:28:00 <bgilbert> walters: is there a particular point you wanted to surface, or is this more of a general heads-up?
17:28:10 <jlebon> also happy to leverage more prow to do cloudy things
17:28:33 <walters> I think more the latter, definitely not a final decision this moment kind of thing given I literally just typed up this doc =)
17:28:41 <bgilbert> +1
17:28:56 <jlebon> to me the primary appeal of CoreOS CI is the similarity between CI and prod
17:29:11 <walters> Mainly just wanted to gauge sentiment and see whether I've missed anything obvious (e.g. if there was some other CI system someone really likes)
17:29:39 <bgilbert> walters: once you have the doc more filled out, could you file a tracker ticket?
17:30:13 <walters> We have one linked at the top in the doc but...maybe it should just be closed and a "2021 CI" ticket filed around this?
17:30:38 <darkmuggle> I think Ganglank is really close to helping here. Gangplank was designed to play really nice with Prow.
17:30:41 <bgilbert> walters: ideally the ticket would be in the form of a concrete proposal
17:30:47 <bgilbert> for an action to be taken
17:30:50 <PanGoat> ^^
17:30:50 <walters> darkmuggle: that sounds great!  I'd love to delve into that intersection more
17:31:29 <travier> Sorry to drop that but shouldn't FCOS be built on Fedora Infra?
17:31:52 <travier> testing for everything else in prow looks fine but the final builds?
17:31:55 <walters> bgilbert: hmm right but realizing this is a lot of individual pieces and some per-repository decisions
17:32:27 <dustymabe> travier: it's currently built in CentOS infra, but pretty close to Fedora infra now that those teams are working more closely together.
17:32:28 <bgilbert> walters: well, an overarching proposal, anyway.  it's hard to get a "discuss CI" ticket  to converge :-)
17:32:33 <walters> Maybe the proposal is just "Let's use these 3" to start
17:32:43 <walters> no mandate about specific repositories using all 3
17:32:50 <dustymabe> whatever we do, we should include Fedora releng in discussions/proposals
17:33:06 <travier> (I like the idea)
17:33:11 <travier> dustymabe++
17:33:11 <zodbot> travier: Karma for dustymabe changed to 8 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:33:48 <walters> travier: agree FCOS is lifecycled with Fedora and needs to be tied to that infra
17:34:12 <walters> OK I will try to write up a new tracker ticket, the old one is a mess
17:34:25 <walters> #action walters to write up a ticket for https://hackmd.io/GPB-x3XHToiYHdEnYNA1kA
17:34:29 <bgilbert> walters: +1
17:34:33 <travier> +1
17:34:37 <walters> #action walters to go have lunch
17:34:46 <lorbus> heh
17:34:50 <bgilbert> :-P
17:34:50 <darkmuggle> I'll volunteer to collaborate with you on that walters.
17:35:01 <walters> darkmuggle: cool!  feel free to edit the doc
17:35:16 <bgilbert> walters: he might have meant lunch
17:35:22 <walters> haha
17:35:28 <travier> :D
17:35:29 <lorbus> I'll call my dinner lunch and do the same
17:35:34 <bgilbert> okay, we're over time.  closing in 60s
17:35:46 <travier> bgilbert++ thanks for running
17:35:50 <lorbus> thanks all!
17:35:51 <darkmuggle> lol
17:36:17 <dustymabe> thanks for running the meeting!
17:36:38 <jlebon> thanks bgilbert, thanks all!
17:36:38 <bgilbert> thanks for the discussion, all!
17:36:40 <bgilbert> #endmeeting