16:29:31 <dustymabe> #startmeeting fedora_coreos_meeting
16:29:31 <zodbot> Meeting started Wed Feb 23 16:29:31 2022 UTC.
16:29:31 <zodbot> This meeting is logged and archived in a public location.
16:29:31 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:29:31 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:31 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:37 <dustymabe> #topic roll call
16:30:01 <dustymabe> .hi
16:30:02 <zodbot> dustymabe: Something blew up, please try again
16:30:05 <zodbot> dustymabe: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:21 <jaimelm> .hello2
16:30:22 <zodbot> jaimelm: Something blew up, please try again
16:30:25 <zodbot> jaimelm: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:37 <jlebon> .hello2
16:30:38 <zodbot> jlebon: Something blew up, please try again
16:30:39 * dustymabe wonders when zodbot is going to get their act together
16:30:41 <bgilbert> hi
16:30:41 <zodbot> jlebon: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:57 <ravanelli> .hi
16:30:58 <zodbot> ravanelli: Something blew up, please try again
16:30:59 <lucab> .hi
16:31:02 <zodbot> ravanelli: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:31:05 <zodbot> lucab: Something blew up, please try again
16:31:08 <zodbot> lucab: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:31:15 <jaimelm> lots of explosions today
16:32:09 <miabbott> .hello miabbott
16:32:10 <zodbot> miabbott: Something blew up, please try again
16:32:11 <dustymabe> #chair jaimelm jlebon bgilbert ravanelli lucab miabbott
16:32:11 <zodbot> Current chairs: bgilbert dustymabe jaimelm jlebon lucab miabbott ravanelli
16:32:13 <zodbot> miabbott: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:32:16 <saqali> .hello saqali
16:32:17 <zodbot> saqali: Something blew up, please try again
16:32:20 <zodbot> saqali: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:32:22 <dustymabe> #chair saqali
16:32:22 <zodbot> Current chairs: bgilbert dustymabe jaimelm jlebon lucab miabbott ravanelli saqali
16:32:37 <saqali> .hi
16:32:38 <zodbot> saqali: Something blew up, please try again
16:32:41 <zodbot> saqali: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:33:00 <jbrooks> o/
16:33:33 <dustymabe> ok let's get started
16:33:36 <dustymabe> #chair jbrooks
16:33:37 <zodbot> Current chairs: bgilbert dustymabe jaimelm jbrooks jlebon lucab miabbott ravanelli saqali
16:33:40 <dustymabe> #topic Action items from last meeting
16:33:47 <dustymabe> * jlebon to open issue to investigate "102 Introduce module Obsoletes and EOL"
16:33:49 <dustymabe> * ravanelli to look into the F36 changes 127 to see if this change affects us on ppc64le
16:33:51 <dustymabe> * dustymabe to create a f36 changes ticket for podman 4.0 investigation/announcement
16:33:53 <dustymabe> * jlebon to send out an announcement about podmanv4 coming to F36 and the beta coming to next in the coming weeks
16:34:06 <jlebon> #info jlebon opened https://github.com/coreos/rpm-ostree/issues/3465
16:34:30 <travier> .hi siosm
16:34:31 <zodbot> travier: Something blew up, please try again
16:34:33 <jlebon> #info jlebon started email draft in https://github.com/coreos/fedora-coreos-tracker/issues/1106#issuecomment-1048901881
16:34:34 <zodbot> travier: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:34:59 <dustymabe> #info dustymabe opened a ticket to discuss podmanv4 coming to FCOS in F36 https://github.com/coreos/fedora-coreos-tracker/issues/1106
16:35:14 <dustymabe> thanks jlebon
16:35:32 <dustymabe> ravanelli: did you have a chance to look at the f36 change regarding ppc?
16:35:38 <dustymabe> #chair travier
16:35:38 <zodbot> Current chairs: bgilbert dustymabe jaimelm jbrooks jlebon lucab miabbott ravanelli saqali travier
16:37:32 <dustymabe> ok we'll re-action and move on
16:37:41 <dustymabe> #action ravanelli to look into the F36 changes 127 to see if this change affects us on ppc64le
16:38:09 <dustymabe> note - we are light on topics today - if there is something you've been wanting to discuss now is a great time to mention it
16:38:13 <dustymabe> #topic RFC: A new continuous mechanical FCOS stream
16:38:18 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/910
16:38:55 <dustymabe> cc jlebon
16:39:01 <jlebon> i can introduce this one
16:39:04 <dustymabe> #chair gursewak
16:39:04 <zodbot> Current chairs: bgilbert dustymabe gursewak jaimelm jbrooks jlebon lucab miabbott ravanelli saqali travier
16:39:16 <jmarrero> .hi
16:39:17 <zodbot> jmarrero: Something blew up, please try again
16:39:20 <zodbot> jmarrero: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:39:26 <miabbott> #chair jmarrero
16:39:26 <zodbot> Current chairs: bgilbert dustymabe gursewak jaimelm jbrooks jlebon jmarrero lucab miabbott ravanelli saqali travier
16:39:47 <jlebon> we now have COPR hooked up to CoreOS packages so they build continuously rebuild git main
16:40:25 <jlebon> I'd like to create a continuous stream which pulls all those in and runs the tests
16:40:41 <jlebon> it wouldn't build all the artifacts. but at least an AMI
16:41:19 <jlebon> the idea is that (1) it becomes trivial to try out the latest version of our software and (2) we catch interdependent regressions faster
16:41:43 <jlebon> the downside here of course is it's yet another stream to maintain
16:42:16 <jlebon> that said, it'd be based on testing-devel so shouldn't be rawhide-level maintainance
16:42:42 <jlebon> and as such we make it clear that it's not as prioritized
16:42:49 <lucab> do you need/want all the git-binaries in the same image? Would a per-PR image build flow work instead, without a full FCOS stream?
16:42:52 <jlebon> this is purely a devel stream for our own purposes
16:43:33 <jlebon> lucab: i think there's value in being able to bring up an AMI with everything already in it
16:44:00 <dustymabe> we already don't create an AMI for rawhide (though we probably should)
16:44:34 <jlebon> we should also feel ok to turn it off in the future if it doesn't seem worth the overhead
16:45:00 <travier> I like it. Should be really useful to validate fresh changes and basing that on testing-devel should enable us to focus on our changes.
16:45:29 <lucab> (my quick-manual-testing flow is through `qemu` images)
16:46:03 <jlebon> lucab: the qemu flow is also faster, because now you just need to download the qcow2. that's just `buildfetch && decompress && kola run`
16:46:06 <dustymabe> I think this is a good idea but I really don't want to chase the failures
16:46:31 <jlebon> s/kola run/cosa run/
16:46:35 <dustymabe> we already added a test case for kubernetes and then promptly disabled it
16:46:59 <travier> Hopefully we don't have false positive failures
16:47:18 <jlebon> dustymabe: any failures specific to the continuous stream would likely mean an issue that's coming to testing-devel
16:48:09 <dustymabe> probably :) - my sentiment still remains
16:48:16 <miabbott> so what's the action on those failures, then?  would we be monitoring both testing-devel + continuous for failures and reacting?
16:48:58 <jlebon> miabbott: ideally yes, but prioritized differently
16:49:41 <jlebon> if i had to order them, i'd say testing-devel > continuous > rawhide
16:49:43 <miabbott> continuous, rawhide, testing-devel...the amount of streams we would be reacting to feels like it is getting a bit much for our small team
16:50:35 <jlebon> hmm, maybe i'm not framing this right
16:50:50 <miabbott> i think if we have the continuous stream, the SLA needs to be crisply defined.  it already seems like we are increasing the amount of time we are spending on rawhide, so another stream could be more than we want to handle
16:50:53 <jlebon> the goal shouldn't be to have another healthy stream going
16:51:01 <travier> I think we should publicize rawhide failures upstream in Fedora as part of the becoming an edition process.
16:51:20 <travier> That would also allow us to focus on those streams
16:51:22 <jlebon> it's to use it as a signal for what's to come and a tool for faster development
16:51:51 <jlebon> it gives us more time to react, but we don't have to react as fast if we're busy
16:51:59 <dustymabe> ok let me ask a question
16:52:18 <miabbott> if we aren't reacting as quickly, doesn't that stream just continuously fail and become less useful?
16:52:25 <dustymabe> how often do the projects we work on (the ones that would be built continuously) break our testing stream in an adverse way?
16:52:31 <travier> If Fedora wants us to take more part in testing they should also take part in seeing and resolving breakage that come from their side
16:53:14 <jlebon> dustymabe: definitely less often than other components in the OS
16:53:29 <jlebon> but it does happen
16:53:43 <dustymabe> yeah. i'm just thinking we CI most of our stuff anyway at the PR level.
16:54:21 <dustymabe> my main objection is around "I don't want to increase our load of looking at pipeline failures". If you think that it's not really going to be increased that much then I could get on board
16:54:39 <lucab> in the "faster development" folder, I think this all-from-git may be more useful when hooked to a per-PR (fallible) job
16:54:41 <miabbott> i see the value of the continuous stream, but i worry in practice we are not great at ignoring failures that we could categorize as "less important"
16:56:13 <jlebon> dustymabe: if flakes weren't an issue, i think it'd be easier to guarantee this :(
16:57:11 <jlebon> lucab: that'd be good to do also, yes
16:57:36 <dustymabe> maybe we can keep a running tally of issues that would have been caught earlier if we had a continuous stream in the ticket?
16:57:44 <dustymabe> and re-evaluate in the future if it would be useful
16:57:52 <jlebon> dustymabe: but overall i think we need to be better at prioritizing things
16:57:57 <bgilbert> it feels like the value of this is pretty small?  if we already have good PR CI upstream (including for building an RPM), we should be catching most things before merge.  and we're probably not going to manually test the artifact after a change of interest has already merged.
16:58:45 <dustymabe> jlebon: indeed on the prioritization front
16:58:50 <jlebon> bgilbert: PR artifacts are ephemeral though
16:59:07 <jlebon> and doesn't cover cross-project issues
16:59:53 <bgilbert> yeah, I accept the cross-project part, but we're unlikely to go manually run artifacts _after_ a change lands unless we have a reason to
17:00:02 <bgilbert> so the lack of an artifact to run doesn't affect us much
17:00:10 <travier> At the same time as we have good PR CI, we shouldn't have that many failures too
17:00:17 <miabbott> so maybe we just produce artifacts with just `kola run --basic-qemu-scenarios` run against it?
17:01:27 <bgilbert> so +1 to examples of problems this would have caught, I guess
17:01:31 <dustymabe> time check.. any proposals anyone would like to make
17:01:39 <jlebon> bgilbert: fair. though it would provide an easy way for others to try new things out
17:01:45 <bgilbert> true
17:02:10 <jlebon> to me, the idea that i can just `kola spawn -p aws` and get everything at the latest sounds pretty awesome
17:02:22 <jlebon> but anyway, i don't want to push on this too hard. we can keep chatting in the ticket for now
17:03:00 <dustymabe> jlebon: what do you say we continue discussion there and also agree to collect issues we think would have been caught by this stream there?
17:03:27 <jlebon> dustymabe: sure SGTM
17:04:09 <dustymabe> #proposed While we see value in a continuous stream we aren't wholly convinced the benefits qualify adding another stream at this point. We'd like to continue discussion in the ticket and collect examples of issues we think would have been caught by a continuous stream before landing in testing-devel.
17:04:18 <miabbott> +1
17:04:27 <bgilbert> +1
17:04:29 <jaimelm> +1
17:04:44 <jmarrero> +1
17:05:13 <jlebon> +0.5 :)
17:05:26 <dustymabe> jlebon: :)
17:05:35 <dustymabe> #agreed While we see value in a continuous stream we aren't wholly convinced the benefits qualify adding another stream at this point. We'd like to continue discussion in the ticket and collect examples of issues we think would have been caught by a continuous stream before landing in testing-devel.
17:05:43 <travier> +1
17:06:00 <dustymabe> #topic tracker: Fedora 36 changes considerations
17:06:05 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/918
17:06:27 <dustymabe> just doing a periodic checkin on this
17:06:52 <dustymabe> looks like all remaining issues have an open ticket or an action item for someone to investigate
17:07:04 <dustymabe> anything anyone wants to mention before we move on?
17:08:05 <dustymabe> ok we are out of topics so I'm going to grab a few things.. if there is anything pressing that needs to be brought up let me know!
17:08:10 <dustymabe> #topic tracker: This Month in Fedora CoreOS February 2022
17:08:16 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1107
17:08:43 <dustymabe> anything happened over the past few weeks that we should highlight in our monthly newsletter
17:08:46 <dustymabe> cverna++
17:09:25 * dustymabe knows there has been a lot of work around coreos layering
17:09:29 <fifofonix_> not pressing.  just a +1 for a recorded (or live) demo of the new layering stuff when available.  esp if it were a selinux policy mod type example.
17:10:02 <dustymabe> fifofonix_: indeed
17:10:05 <miabbott> one addition to the monthly newsletter that i think we should do is highlighting first time contributors (i'll add the idea to the ticket)
17:10:10 <dustymabe> we should have a video meeting coming up soon
17:10:20 <dustymabe> miabbott: good idea
17:10:22 <jlebon> cverna: thanks for starting that!
17:10:41 <dustymabe> I know nemric was working on systemd user units in Ignition
17:10:47 <dustymabe> could be a cool thing to highlight when it lands
17:10:52 <bgilbert> +1
17:11:08 * dustymabe moves to open floor soonish
17:11:25 <jaimelm> a layering demo at the video meeting would be fantastic
17:11:44 <dustymabe> #topic open floor
17:11:49 <fifofonix_> sorry sort of thought we were already open floor.  also interested in the toolbox stuff i've read about here and there but never experimented with.  a video example would be nice.
17:12:10 <dustymabe> fifofonix_: no worries :)
17:12:27 <dustymabe> anyone with anything for open floor?
17:12:57 <dustymabe> I had a talk with the podman team again recently about their plans for podman-machine and "Desktop" support
17:13:29 <dustymabe> which includes the use of FCOS
17:14:04 <dustymabe> one thing they touched on was a RFE for zincati. There is currently an issue for it
17:14:17 <dustymabe> but it needs some clarification for what's required
17:14:23 <dustymabe> https://github.com/coreos/zincati/issues/539
17:14:36 <dustymabe> might try to get some people together soon to try to hammer those down
17:15:04 <dustymabe> that's it from me - anyone else?
17:16:12 * dustymabe closes out the meeting soon
17:16:37 <dustymabe> though.. jaimelm i'm interested if there is anything from the OKD side worth bringing up
17:17:28 <bgilbert> oh, I do have something
17:17:31 <dustymabe> yay
17:17:41 <bgilbert> #link https://github.com/coreos/coreos-installer/issues/792
17:18:03 <bgilbert> apparently at least OKD 4.9 is still recommending an FCOS release based on Fedora 34
17:18:13 <bgilbert> and the newest release of coreos-installer dropped the F34 signing key
17:18:36 <dustymabe> interesting
17:18:54 <bgilbert> I am 100% okay with us having done so, but it's really unfortunate that OKD is pinning an 8-month-old FCOS release
17:19:42 <dustymabe> bgilbert: but does it just use the F34 FCOS as a starting point and then updates flow?
17:19:50 <dustymabe> i could be OK with that, though still not ideal
17:20:41 <bgilbert> I haven't checked, but I assume so?  however there is also https://github.com/openshift/okd/issues/1020
17:20:59 <jlebon> it's not far from the situation on the OCP side. but then the instructions should probably include the coreos-installer version to use
17:21:25 <dustymabe> interesting
17:21:42 <dustymabe> jaimelm: any thoughts here?
17:21:44 <bgilbert> yeah, pinning the installer seems like an okay workaround
17:22:40 <dustymabe> thanks for bringing this up bgilbert
17:23:10 <dustymabe> We *could* modify the instructions in a release checklist to drop that EOL key later (i.e. f36 GA in this case)
17:23:14 <dustymabe> which might help a little
17:23:24 <dustymabe> in this case I just did them both at the same time
17:24:07 <bgilbert> we don't support old releases, by policy
17:24:49 <bgilbert> I'm in favor of some overlap so people can do some additional QA on a new Fedora major if they want, but dropping support for EOL releases is the right thing to do IMO
17:25:04 <dustymabe> yeah
17:25:18 <dustymabe> i don't think you'll find much objection there
17:25:34 <dustymabe> though.. I guess we should probably be careful with the term EOL
17:25:45 <jlebon> we can drop it at the same time as the rest of Fedora "drops" it
17:25:53 <dustymabe> in terms of greater Fedora F34 isn't EOL
17:25:53 <jlebon> i.e. one month after N+2 GAs
17:26:05 <bgilbert> jlebon: we don't maintain two releases in parallel though
17:26:12 <bgilbert> dustymabe: right
17:26:41 <bgilbert> jlebon: in that sense, we gave it two months after stable rebase, which is longer than the Fedora policy
17:26:52 <dustymabe> ultimately it would be nice to help OKD not rely on something from f34 so maybe let's try to have that conversation with them
17:27:18 <jlebon> bgilbert: i mean matching Fedora's definition of EOL
17:27:23 <dustymabe> anything else for today?
17:27:29 <fifofonix_> since we're talking about k8s can i say that i love the simplicity of typhoon
17:27:37 <dustymabe> fifofonix_: you can :)
17:27:44 <bgilbert> jlebon: right, I don't think that makes sense for us though
17:27:52 <fifofonix_> i don't know whether it is a sensitive topic but i feel like not mentioning typhoon more prominently is meh
17:28:12 <travier> Why would it be sensitive?
17:28:13 <bgilbert> fifofonix_: I don't think it's a sensitive topic.  FCOS is for running containers, regardless of orchestrator :-)
17:28:14 <fifofonix_> i'm talking about in the context of fcos - in the faq etc
17:28:23 <dustymabe> fifofonix_: I haven't personally used it but I appreciate the work being done by dghubble there
17:28:24 <fifofonix_> redhat/openshift/okd...
17:28:25 <jlebon> bgilbert: i think i probably agree :)
17:28:38 <travier> PRs or suggestions welcomed 🙂
17:28:47 <dustymabe> fifofonix_++
17:28:59 <fifofonix_> good point travier but i've held off a bit there for fear of offending.
17:29:11 <bgilbert> fifofonix_: yeah, please do send PRs!
17:29:17 <bgilbert> the FAQ in particular is pretty stale
17:29:30 * dustymabe closes meeting in 30s (on time today)
17:29:38 <lucab> bgilbert: for this specific case, I think it won't hurt keeping the oldstable key a bit longer. Both FCOS and OKD by default will auto-update to something more recent.
17:29:39 <travier> We certainly won't be offended at contributions
17:29:41 <fifofonix_> i don't know - i still go back to your faq and it has moved forward.  amazing job everyone as usual.
17:30:02 <dustymabe> #endmeeting