16:31:24 <dustymabe> #startmeeting fedora_coreos_meeting
16:31:24 <zodbot> Meeting started Wed Nov 17 16:31:24 2021 UTC.
16:31:24 <zodbot> This meeting is logged and archived in a public location.
16:31:24 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:31:24 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:31:24 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:31:28 <dustymabe> #topic roll call
16:31:30 <dustymabe> .hi
16:31:31 <zodbot> dustymabe: Something blew up, please try again
16:31:34 <zodbot> dustymabe: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:31:54 <nemric> hi
16:32:05 <davdunc> jo
16:32:08 <davdunc> hi
16:32:09 <dustymabe> 👋
16:32:24 <dustymabe> man zodbot is having a bad week
16:32:33 <davdunc> bless their heart.
16:32:35 <travier[m]> .hello siosm
16:32:36 <zodbot> travier[m]: siosm 'Timothée Ravier' <travier@redhat.com>
16:32:37 <dustymabe> davdunc: did the meeting get logged last week when we were having similar issues?
16:32:44 <dustymabe> .hello dustymabe
16:32:46 <miabbott> .hello miabbott
16:32:46 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:32:48 <davdunc> it did dustymabe
16:32:49 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:33:12 <walters> .hello2
16:33:13 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:33:27 <lucab> .hi
16:33:28 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com>
16:33:36 <davdunc> .hui
16:33:38 <davdunc> .hi
16:33:39 <zodbot> davdunc: davdunc 'David Duncan' <davdunc@amazon.com>
16:33:45 <davdunc> i just can't type today.
16:35:41 <jlebon> .hello2
16:35:41 <jmarrero> .hi
16:35:41 <ravanelli> .hi
16:35:41 <mnguyen_> .hello mnguyen
16:35:41 <dustymabe> #chair nemric davdunc travier[m] miabbott walters lucab jlebon jmarrero ravanelli mnguyen_
16:35:41 <zodbot> Current chairs: davdunc dustymabe jlebon jmarrero lucab miabbott mnguyen_ nemric ravanelli travier[m] walters
16:35:47 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:35:50 <zodbot> jmarrero: jmarrero 'Joseph Marrero' <jmarrero@redhat.com>
16:35:52 <dustymabe> quite the crew here today! 🎉
16:35:53 <zodbot> ravanelli: ravanelli 'Renata Renata Andrade Matos Ravanelii' <renata.ravanelli@gmail.com>
16:35:56 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:36:21 <dustymabe> #topic Action items from last meeting
16:36:28 <dustymabe> We had one action item from last meeting
16:36:34 <dustymabe> * jlebon to file a ticket about aligning testing with GA for f36
16:36:48 <jlebon> #info jlebon filed https://github.com/coreos/fedora-coreos-tracker/issues/1024
16:36:58 <jlebon> ohh cool, hadn't noticed the issue number
16:37:06 <jdoss> .hi
16:37:07 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:37:12 <dustymabe> :) - may the powers of two be with you
16:37:30 <jlebon> :)
16:37:53 <bgilbert> .hi
16:37:54 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:37:56 <lucab> kiloticket vs kibiticket
16:37:58 <dustymabe> Let's move into topics
16:38:08 <dustymabe> #chair bgilbert
16:38:08 <zodbot> Current chairs: bgilbert davdunc dustymabe jlebon jmarrero lucab miabbott mnguyen_ nemric ravanelli travier[m] walters
16:38:23 <dustymabe> #topic During major version rebases, align FCOS testing with GA
16:38:32 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1024
16:38:32 <jdoss> #chair jdoss
16:38:56 <dustymabe> #chair jdoss
16:38:56 <zodbot> Current chairs: bgilbert davdunc dustymabe jdoss jlebon jmarrero lucab miabbott mnguyen_ nemric ravanelli travier[m] walters
16:39:03 <jdoss> \o/
16:39:03 <dustymabe> ha - how could I miss jdoss
16:39:07 <dustymabe> jlebon: want to intro this one?
16:39:18 <jlebon> dustymabe: sure
16:39:57 <jlebon> currently, during a new Fedora release, testing is delayed by one week and stable by three weeks to get the new content
16:40:07 <jlebon> we've said in the past already that we wanted to shorten this
16:40:19 <jlebon> since things have been going smooth lately, I think it's a good time to work on that
16:40:33 <jlebon> the proposal is to make testing on release week have the new content
16:41:02 <travier[m]> 👍️
16:41:10 <dustymabe> i.e. do we feel comfortable enough with the process now to move `testing` over a week earlier
16:41:39 <jdoss> +1 fail forward
16:41:39 <travier[m]> But we're not fully rebased to F35 right now so maybe this is a bit early to discuss
16:42:24 <dustymabe> travier[m]: IOW, we don't know if a bunch of people will hit issues once F35 hits `stable` ?
16:42:31 <jlebon> i think implementing this is mostly straightforward; just promote `next` content to `testing` on release week, but freeze exceptions handling needs some discussing i think
16:42:35 <travier[m]> I think we should wait at least for a month after we release the first F35 based version to get some perspective about how well the rebase went this time
16:42:58 <travier[m]> I'm +1 for the change but I think we should re-discuss that later
16:43:33 <jlebon> i'm ok with that. this is for f36, so we have time
16:43:34 <dustymabe> I'm comfortable discussing it now, but also comfortable waiting a month (since there is nothing pressing this)
16:44:18 <travier[m]> dustymabe: yes, we don't know _now_ if the work we did to prepare the F35 rebase will be bearing fruits
16:44:49 <dustymabe> jlebon: want to say we'll discuss this in jan 2022?
16:45:13 <jlebon> SGTM
16:45:38 <dustymabe> +1
16:46:19 <dustymabe> #info we'll discuss this in january 2022 when the F35 rebase has landed in our stable stream and any currently unknown issues have been fleshed out
16:47:01 <dustymabe> ok we discussed the nutanix one last time and jaimelm isn't here for the other one
16:47:10 <dustymabe> let's dive into something else
16:47:22 <dustymabe> #topic work items awaiting some action
16:47:34 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues?q=is%3Aissue+is%3Aopen+label%3Astatus%2Fpending-action
16:48:00 <dustymabe> we have quite a few work items that have been discussed and we've decided some sort of direction on
16:48:09 <dustymabe> but we have yet to pick up and do the work
16:48:23 <dustymabe> if you click on the link you can see the list.
16:48:36 <dustymabe> is there anything in that list (that isn't already assigned) that anyone is interested in
16:49:12 <dustymabe> if it's an area you're interested in but not comfortable with yet i'm happy to be a mentor - and I assume other senior members are willing to mentor as well
16:49:28 <dustymabe> if no one volunteers we'll just assign everything to jdoss
16:49:44 <jdoss> What!
16:50:11 <walters> haha
16:50:21 <jdoss> I got enough on my plate. I am trying to shove AWX into FCOS right now and it is emotionally hurting me.
16:50:35 <dustymabe> oh also - one item in there is related to azure - we recently got credits to do CI testing in Azure
16:51:04 <dustymabe> jdoss: ha
16:51:05 <travier[m]> :D
16:51:35 <dustymabe> ok we'll move off this topic - please reach out if there's anything that looks enticing :)
16:51:40 <dustymabe> #topic - call for topics
16:51:58 <dustymabe> any other items that could use the `meeting` label or that need to be brought up for discussion today?
16:53:30 <dustymabe> :) - I'll do a better job of fishing for appropriate topics in the future.. one that comes to mind for me is...
16:53:52 <dustymabe> #topic Make publicly accessible coreos-assembler builds for architectures != x86_64
16:53:57 <dustymabe> #link https://github.com/coreos/coreos-assembler/issues/2470
16:54:17 <dustymabe> so we currently only produce/deliver x86_64 COSA images in quay.io
16:54:33 <dustymabe> but we recently added multi-arch builders to our pipeline(s)
16:54:51 <dustymabe> right now those builders are just building cosa locally once a day
16:55:27 <dustymabe> There are a couple of questions here..
16:55:54 <dustymabe> Well.. maybe not questions
16:56:10 <dustymabe> My current thoughts on the subject are in https://github.com/coreos/coreos-assembler/issues/2470#issuecomment-970679843
16:56:48 <dustymabe> so 1. preferrably build/push from non-internal RH
16:57:11 <walters> I am still somewhat in the camp of moving closer to OSBS
16:57:21 <dustymabe> but access to container build infra is super limited (unless you have an openshift on different architectures)
16:57:29 <dustymabe> walters: i.e. the OSBS that is in Fedora ?
16:57:30 <walters> https://github.com/coreos/enhancements/pull/7 makes the OSBS alignment much stronger
16:57:35 <walters> yes
16:57:56 <davdunc> I have access to one dustymabe
16:58:06 <jlebon> the thing with OSBS is that they really want us to ship everything in RPMs first
16:58:18 <walters> I think we need to break that requirement
16:58:27 <jlebon> if we can, that'd be great
16:58:58 <dustymabe> so is OSBS an openshift cluster and is on various arches?
16:59:07 <dustymabe> do we get ppc64le, s390x, aarch64 ?
16:59:14 <walters> I believe so yes
16:59:25 <jlebon> but also... IIUC, this will introduce a huge lag between a git merge and it actually being usable in pipelines
16:59:35 <walters> (Long term of course I also think koji should stop being a cluster, mock should stop existing, and we do RPM builds in Kubernetes pods in the same way we do container builds too)
16:59:53 <jlebon> unless we can get continuous building like we currently do
16:59:56 <dustymabe> walters: yeah I think I agree with that
17:00:06 <jlebon> without having to do anything in dist-git
17:00:18 <dustymabe> jlebon: can you descirbe the contraints around OSBS that make you think the lag will be increased?
17:00:24 <dustymabe> i'm in "fact finding" mode right now
17:00:32 <dustymabe> I really know very little about OSBS
17:01:16 <jlebon> I'm not completely sure either, so let me dig into it and report back
17:01:33 <walters> well I think for sure our CI (e.g. per-pull request builds) would continue to run outside of OSBS, and we'd probably continue to use the CI infra to push output from `main` somewhere
17:01:35 <dustymabe> jlebon: is this worth us setting up some time to talk with the Fedora folks on $topic?
17:02:01 <walters> so for example if push came to shove and we needed to ship a fix to uploads to azure (x86_64  only) we could have the x86_64 pipeline use the "CI cosa" that didn't come from osbs
17:02:20 <jlebon> dustymabe: maybe, yeah
17:03:13 <dustymabe> ok, who wants to lead up that effort?
17:03:21 <lucab> jlebon: I think builds can get directly triggered from git content via webhooks, without going through dist-git
17:04:37 * dustymabe adds this to the list of tasks
17:04:53 <jlebon> lucab: that'd be great
17:05:17 <dustymabe> #info we'll investigate OSBS further to see if that solution can possibly solve our needs here without too many contraints
17:05:52 <dustymabe> #topic open floor
17:06:11 <dustymabe> any topics for open floor?
17:06:23 <jlebon> dustymabe: but to circle back, i think it wouldn't be too bad to just have the arch builders push daily to quay.io
17:06:41 <travier[m]> Be careful, non Matrix users do not see message reactions (👍️)
17:06:55 <dustymabe> jlebon: yeah, I thought about that too - but if we do that, what's the benefit?
17:07:03 <jlebon> the advantage is that other developers can pull them, and we'd be able to drop the :tag hack we just added
17:07:07 <dustymabe> travier[m]: indeed - I see no message reactions
17:07:37 <dustymabe> ehh - i'm not really sure how many people are pulling non-x86_64 container images TBH
17:07:55 <dustymabe> the extream is probably on the s390x side
17:08:22 <dustymabe> to put it in a statement: if we do the build on the s390x builder and push to quay.io no one would probably ever pull it
17:08:33 <jlebon> well... aren't we talking about the same advantages here regardless of how it's achieved?
17:08:34 <dustymabe> the s390x builder already has it local
17:08:44 <dustymabe> jlebon: not exactly
17:08:49 <jlebon> are you questioning whether we should do this at all? whether it be OSBS or otherwise
17:09:09 <dustymabe> yes and no
17:09:24 <dustymabe> I think if we have something else coordinating the builds and doing the uploads it has benefits
17:09:53 <walters> I think having it on quay.io is making things feel more seamless for the time someone who works mainly on e.g. x86_64 needs to test a fix for $arch and can follow instructions to access a $arch kube cluster or single $arch coreos instance
17:10:00 <dustymabe> I think if we're doing the builds on the builders anyway (we're coordinating the build/uploads ourselves) then it doesn't make a lot of sense (since we're just pulling them back to those same builders)
17:10:41 <dustymabe> walters: yeah, but git clone and podman build isn't too bad IMO
17:10:50 <jmarrero> We could also just document that if on arch you build the container first and set COREOS_ASSEMBLER_CONTAINER
17:10:52 <walters> Longer term I suspect aarch64 is going to make sufficient inroads into the cloud space that our major images will need to have at least those two, and if you have two you might as well have all of them
17:11:09 <travier[m]> I only think it make sense to have images in quay for the arch that we have in FCOS so x86_86 and aarch64
17:11:21 <dustymabe> travier[m]: the plan is to add the other arches
17:11:32 <dustymabe> so we'd have ppc64le and s390x fcos
17:11:47 <jlebon> dustymabe: right yeah, IOW if it's too complex to do ourselves then it's less worth it. that's why i was saying it wouldn't be too bad to just add a `podman push` to that systemd unit/timer
17:11:48 <dustymabe> though it's definitely arguable how much impact that will have to most users
17:11:58 <travier[m]> but will we have fcos builds too?
17:12:30 <dustymabe> travier[m]: full out streams - just like aarch64 and x86_64 today
17:13:25 <dustymabe> and eventually... riscv FCOS anyone?
17:13:37 <travier[m]> Well if we have the infra to do the builds then OK
17:14:10 <dustymabe> sorry I closed off that topic too soon :)
17:14:15 <dustymabe> any other topics for open floor ?
17:15:27 <dustymabe> #endmeeting