16:30:01 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:01 <zodbot> Meeting started Wed May 18 16:30:01 2022 UTC.
16:30:01 <zodbot> This meeting is logged and archived in a public location.
16:30:01 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:01 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:04 <dustymabe> #topic roll call
16:30:11 <lucab> .hi
16:30:12 <zodbot> lucab: Something blew up, please try again
16:30:15 <zodbot> lucab: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:33 <jlebon> .hello2
16:30:35 <zodbot> jlebon: Something blew up, please try again
16:30:38 <zodbot> jlebon: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:41 <dustymabe> .hi
16:30:42 <zodbot> dustymabe: Something blew up, please try again
16:30:44 <travier_> .hello siosm
16:30:45 <zodbot> dustymabe: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:48 <zodbot> travier_: Something blew up, please try again
16:30:51 <zodbot> travier_: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:31:24 <dustymabe> .fire zodbot
16:31:24 <zodbot> adamw fires zodbot
16:31:31 <travier> :)
16:32:14 <jmarrero> .hi
16:32:15 <zodbot> jmarrero: Something blew up, please try again
16:32:18 <zodbot> jmarrero: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:32:33 <dustymabe> #chair dustymabe lucab travier jlebon jmarrero
16:32:33 <zodbot> Current chairs: dustymabe jlebon jmarrero lucab travier
16:33:45 <dustymabe> reminder up front - if you have any topics you want to discuss please add the meeting label to the ticket or mention it up front here
16:34:41 <miabbott> .hello miabbott
16:34:42 <zodbot> miabbott: Something blew up, please try again
16:34:46 <zodbot> miabbott: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:34:47 <mnguyen_> .hello mnguyen
16:34:49 <zodbot> mnguyen_: Something blew up, please try again
16:34:50 <ravanelli> .hi
16:34:52 <zodbot> mnguyen_: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:34:54 <mnguyen_> .hi
16:34:55 <zodbot> ravanelli: Something blew up, please try again
16:34:58 <zodbot> ravanelli: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:35:01 <zodbot> mnguyen_: Something blew up, please try again
16:35:04 <dustymabe> zodbot: is explosive today!
16:35:04 <zodbot> mnguyen_: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:35:04 <miabbott> 🔥🔥🔥🔥
16:35:09 <aaradhak> .hi
16:35:10 <zodbot> aaradhak: Something blew up, please try again
16:35:14 <zodbot> aaradhak: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:35:22 <mnguyen_> .bye
16:35:32 <dustymabe> #chair miabbott mnguyen_ ravanelli aaradhak
16:35:32 <zodbot> Current chairs: aaradhak dustymabe jlebon jmarrero lucab miabbott mnguyen_ ravanelli travier
16:35:43 <gursewak> .hi
16:35:44 <zodbot> gursewak: Something blew up, please try again
16:35:47 <zodbot> gursewak: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:36:00 <dustymabe> #chair gursewak
16:36:00 <zodbot> Current chairs: aaradhak dustymabe gursewak jlebon jmarrero lucab miabbott mnguyen_ ravanelli travier
16:36:05 <dustymabe> have I missed anyone?
16:36:29 <dustymabe> #topic Action items from last meeting
16:36:42 <dustymabe> No action items from last meeting
16:36:44 <dustymabe> :)
16:37:17 <aaradhak> .hi
16:37:18 <dustymabe> we only have one meeting ticket but the NM reps aren't here to discuss that one again so we'll skip it
16:37:18 <zodbot> aaradhak: Something blew up, please try again
16:37:21 <zodbot> aaradhak: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:37:30 <dustymabe> let's go through a few alternative meeting tickets
16:37:35 <dustymabe> #topic May Edition of "This Month in FCOS"
16:37:42 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1188
16:37:53 <dustymabe> anyone with any items to add to the list of happenings in May?
16:38:04 <dustymabe> we just had an ignition release - anything significant we want to mention
16:38:56 <jlebon> support for kubevirt
16:39:12 <dustymabe> jlebon: yeah.. did we push that completely over the finish line yet?
16:39:17 <miabbott> is that fully plumbed in?
16:39:21 <dustymabe> or should we make it a point to
16:39:36 <jlebon> the disabling service bit which is minor, but was super long standing papercut
16:39:53 <dustymabe> jlebon: +1 for that too
16:39:53 <jlebon> i think it still needs plumbing elsewhere yeah
16:40:56 <dustymabe> IIUC the remaining work is on us to complete
16:40:56 <dustymabe> accurate? ^^
16:40:56 <jlebon> yeah, agreed
16:41:15 <dustymabe> who wants to volunteer as tribute to push kubevirt over the edge?
16:41:20 <jlebon> the ignition-validate container now available for aarch64 is pretty nice too
16:41:37 <dustymabe> oh? we should mention that too
16:41:37 <lucab> is the hackmd link the wrong one or is it just the title that needs a s/April/May/ update?
16:41:48 <dustymabe> lucab: good question - we should ask cverna
16:42:04 <dustymabe> I usually just update the ticket with info and then he updates the hackmd
16:43:25 <lucab> ack, will do that
16:43:37 <dustymabe> ok here is what I have so far:
16:43:43 <dustymabe> - New Ignition release
16:43:45 <dustymabe> - At least mention that this fixes #392
16:43:47 <dustymabe> - Mention other features
16:43:49 <dustymabe> - Support for kubevirt (we need to push this over the finish line)
16:43:51 <dustymabe> - aarch64 `ignition-validate`.
16:43:57 <dustymabe> container
16:43:59 <lucab> from my side: all Fedora CoreOS services have been migrated to the new OCP4 fedora-infra cluster
16:44:06 <dustymabe> lucab: +1
16:44:26 <dustymabe> ahh - COSA has been moved to F36 t oo
16:44:50 <dustymabe> and we can mention `testing` and `stable` (by the end of the month) have also been moved
16:45:08 <jlebon> +1
16:45:23 <dustymabe> anything else?
16:45:54 <lucab> https://github.com/coreos/rpm-ostree/releases/tag/v2022.9
16:46:23 <lucab> specifically, new features related to IMA and containerized flow
16:47:11 <dustymabe> +1
16:47:24 <dustymabe> i'll make a comment with some of this to the ticket. if you see anything I missed please do add it
16:47:27 <lucab> https://github.com/ostreedev/ostree/releases/tag/v2022.3
16:47:44 <lucab> the part about SELinux policy customizations
16:47:44 <dustymabe> lucab: maybe you can make a comment for the rpm-ostree/ostree releases and relevant highlights
16:48:15 <lucab> dustymabe: sure, don't worry, I was going through recent releases right now looking for juicy stuff
16:48:17 <dustymabe> lucab: yep I think that's https://github.com/coreos/fedora-coreos-tracker/issues/1188#issuecomment-1119192248
16:48:50 <lucab> yes indeed
16:49:15 <dustymabe> ok shall we move to next topic? hopefully someone will fall from the sky and finish kubevirt support :)
16:49:39 <dustymabe> #topic develop strategy around organization and naming for our containers in quay.io
16:49:44 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1171
16:49:55 <dustymabe> so for this one we are mostly unblocked on strategy
16:50:18 <dustymabe> at this point we need to get creds from the people who have them to push to the `fedora` ord on quay
16:50:32 <dustymabe> #action dustymabe jaimelm to reach out to fedora infra about creds for pushing to `quay.io/fedora/fedora-coreos` and `quay.io/fedora/fedora-coreos-kubevirt`.
16:50:55 <jlebon> awesome +1
16:51:50 <dustymabe> #topic open floor
16:52:05 <dustymabe> ok. we didn't really have any meeting tickets so here we are
16:52:08 <dustymabe> any items for open floor?
16:52:30 <lucab> I have a question related to the previous topic
16:52:46 <dustymabe> lucab: go ahead, sorry I didn't wait long enough
16:53:03 <lucab> assuming we'll do https://github.com/coreos/fedora-coreos-cincinnati/issues/9 soonish
16:53:21 <lucab> would that go in the `fedora` namespace or in `coreos` one?
16:54:07 <lucab> or more generally, are we trying to put new things in the fedora namespace and stop adding it to the coreos one?
16:54:19 <dustymabe> good question
16:54:31 <jlebon> i think anything that's more internal should stay in `coreos`
16:54:38 <dustymabe> I personally think the containers that we create that deliver "fedora" go under `fedora`
16:54:50 <jlebon> and anything we consider a public API in `fedora`
16:55:05 <dustymabe> I think ones that are the product of upstream projects (like butane) stay under coreos
16:55:25 <dustymabe> jlebon: define "public API"?
16:55:44 <jlebon> a public API that FCOS users can use
16:55:54 <lucab> ok, so the general feedback seems to that this is an upstream project and would thus go into `coreos`
16:56:04 <lucab> *seems to be that
16:56:06 <dustymabe> jlebon: would you put `butane` under `fedora` ?
16:56:43 <jlebon> dustymabe: i think your criteria is better yeah
16:56:53 <dustymabe> +1
16:56:57 <lucab> (yes I think butane or ignition-validate are similar, but this one is both 1) new and 2) FCOS-specific)
16:57:34 <lucab> ok thanks, I'm good, I'll note this in the ticket
16:57:39 <dustymabe> lucab: +1
16:57:56 <dustymabe> i mean would you call the container `quay.io/coreos/fedora-coreos-cincinnati`
16:57:58 <dustymabe> ?
16:58:10 <dustymabe> I think that makes it clear it might be specific to FCOS
16:58:33 <lucab> probably, yes, that matched the GH repo name
16:58:36 <lucab> *matches
16:58:41 <dustymabe> +1
16:58:59 <dustymabe> other topics for open floor ?
16:59:33 <dustymabe> I've been noticing some buzz from Kubecon - kind of bummed we don't have anyone there presenting FCOS.. Should we submit something for the next kubecon?
16:59:52 <lucab> I may have a couple
17:00:17 <miabbott> CFP for kubecon NA closed next week on friday may 27
17:00:21 <miabbott> closes*
17:00:46 <dustymabe> lucab++
17:00:47 <zodbot> dustymabe: Karma for lucab changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:01:05 <lucab> first one is that we had a bit of unexpected downtime in coreos-cincinnati
17:01:05 <dustymabe> miabbott: can you open a ticket for presos for kubecon?
17:01:16 <miabbott> sure thing
17:01:26 <lucab> maybe the F36 bump, or maybe something else, I'm currently bisecting
17:01:42 <lucab> #link https://github.com/coreos/fedora-coreos-cincinnati/pull/65#issuecomment-1130055600
17:02:02 <lucab> if anybody saw errors in Zincati logs yesterday/today, it's this
17:02:42 <dustymabe> lucab: when you saw you'll have to try smaller image bumps - what does that mean?
17:02:52 <dustymabe> s/saw/say
17:03:13 <dustymabe> try 34->35 instead of 34->36?
17:03:23 <lucab> dustymabe: we did a F33->F36 bumps, I'm now doing F33->F34->F35->F36
17:03:33 <dustymabe> oh,
17:03:37 <dustymabe> +1
17:03:39 <dustymabe> thanks
17:03:46 <lucab> yes kinda
17:04:05 <lucab> F33 was the max that the old cluster could run
17:04:14 <dustymabe> This one will be interesting to watch. I'll grab my popcorn 🍿
17:04:34 <lucab> F34 is already deployed right now and seems fine
17:04:55 <lucab> #link https://github.com/coreos/fedora-coreos-cincinnati/pull/66#issuecomment-1130199951
17:05:20 <miabbott> Ticket for KubeCon NA 2022 Proposals
17:05:23 <lucab> the next one is F35 but I'll try that tomorrow, PR is at https://github.com/coreos/fedora-coreos-cincinnati/pull/67
17:05:25 <miabbott> #link https://github.com/coreos/fedora-coreos-tracker/issues/1203
17:05:41 <dustymabe> approved!
17:05:45 <dustymabe> thanks lucab
17:05:56 <dustymabe> i guess another topic is monitoring
17:06:06 <dustymabe> we realized this because a user reported it
17:06:21 <lucab> ah, not really
17:06:31 <dustymabe> oh, i was mistaken then
17:06:40 <dustymabe> i just saw IRC and assumed
17:06:41 <lucab> according to metrics everything was fine
17:06:53 <dustymabe> hmm so a gap in metrics then?
17:07:02 <lucab> I'm still not sure what was going on
17:07:35 <lucab> well we don't have insights on the ingress
17:08:01 <dustymabe> yeah - almost like we'd need our canaries reporting back to the metrics server too
17:08:26 <lucab> plus staging was fine, and three of the four production endpoints were fine too
17:08:56 <lucab> so dunno, let's see
17:09:17 <lucab> no the other topic was the setup of the "nobody" user that I saw today
17:09:25 <lucab> #link https://github.com/coreos/fedora-coreos-tracker/issues/1201
17:10:06 <lucab> it feels like it's something from Atomic maybe?
17:11:01 <lucab> I think we can just go and fix the definition, but I was curious if anybody knew more about the context/backstory
17:11:02 <dustymabe> jlebon: miabbott: mnguyen_: ring any bells?
17:11:23 <travier> yes, https://fedoraproject.org/wiki/Changes/RenameNobodyUser
17:11:44 <travier> I don't know about fixing it. Maybe we need an alias
17:11:53 <travier> about how* we should fix it
17:11:55 <dustymabe> travier: IOW this is a Fedora Change we didn't properly pick up
17:12:03 <travier> a long time ago
17:12:12 <dustymabe> which is why we now watch the changes more closely
17:13:24 <lucab> uh interesting
17:13:28 <lucab> well F28 was indeed long time ago :)
17:13:40 <jlebon> ideally we'd be able to apply the change to new machines only like the Fedora Change, but that's tricky for this
17:13:52 <travier> yes
17:14:03 <travier> this is also related to the sysusers rpm-ostree change
17:14:27 <lucab> do we feel the need to keep reserving the 99 uid/gid?
17:14:28 <jlebon> yup, indeed
17:14:44 <travier> in that we need to remove nss-altfiles & our hardcoded passwd & group entries
17:14:46 <dustymabe> lucab: big thanks for bringing up this issue
17:15:08 <jlebon> lucab: i'm not sure we have a choice. if we remove it and a user gets added with that uid/gid, they then have access to files with that uid
17:15:40 <lucab> but I wouldn't expect things to be owned by "nobody", I think
17:16:14 <travier> None should use it but some bad security advice guides use it as unprivileged user
17:16:19 <jlebon> I wouldn't either, but I'm not sure how safe it is to assume it.
17:16:59 <dustymabe> i mean, we could just assign 99 to a non-existant user
17:17:15 <dustymabe> fcos-dontuse
17:17:17 <dustymabe> :)
17:17:21 <lucab> but then I think 99 should be reserved at the whole Fedora level too
17:17:35 <jlebon> maybe we can just rename it to oldnobody, and then add the new nobody with the new id
17:17:36 <dustymabe> lucab: i was speaking only for upgrading systems
17:17:45 <travier> The "easy" wait to get out of this is to move to sysusers only. This involves moving passwd & group to /etc only. This would keep existing system as is and only apply to new systems
17:17:47 <dustymabe> jlebon: yeah, basically what you are saying
17:17:57 <dustymabe> travier++
17:17:58 <lucab> (or the may re-allocate for a random package user/group in the future)
17:18:06 * dustymabe have been wanting to close the sysuers ticket for a really long time
17:18:06 <travier> s/wait/way
17:18:16 <jlebon> travier: yeah, agreed
17:18:34 <jlebon> if this isn't actually causing any issues right now, it might be best to just let it be until we get to sysusers
17:18:43 <lucab> travier: yes sure, but we need to fix existing users first, or this would just disappear
17:19:09 <lucab> I'm doubtful
17:19:30 <dustymabe> lucab: "fix exisint users first" -> what problem are we seeing exactly?
17:19:31 <travier> existing users would get their current /etc + our /usr defaults
17:19:38 <lucab> even with sysusers, it will conflict with the sysusers entry from the the "setup" package
17:20:39 <travier> The migration path for existing systems with sysusers is to merge /etc & /usr before we remove /usr and then do the switch
17:21:24 <travier> in that we no longer have anything in /usr in the end and the files in /etc are only populated on first boot
17:21:43 <lucab> but we'll end up with a weird mix of nobody being both 99 and 65534
17:21:45 <travier> (potential migration path*)
17:21:54 <jlebon> lucab: that's the case for Fedora as a whole today IIUC
17:22:03 <travier> sysusers does not re-create users that already exist
17:22:29 <dustymabe> travier: ^^ that's the lightbulb for me
17:22:31 <lucab> jlebon: indeed, which is way I'd rather have Fedora reserve 99 first
17:22:32 <travier> but yes, we would need to provide a script/instructions for users to update their systems to new defaults
17:22:48 <lucab> *why
17:23:55 <dustymabe> IIUC it wouldn't be required, just nice to get them back in line
17:23:59 <lucab> I think I agree on general case
17:24:00 <travier> I don't think they will want to do that now has it has been a while
17:24:24 <travier> they're more interested in not adding new static users
17:24:50 <lucab> on this specific one, to me it looks like F28 forgot to reserve the old uid/gid AND we forgot to update
17:25:14 <travier> They did not need to reserve it for existing systems
17:25:24 <travier> it just stayed there on the old ones
17:25:32 <dustymabe> should I summarize the proposed solution?
17:25:47 <dustymabe> - migrate to sysusers "merge /etc & /usr before we remove /usr and then do the switch"
17:25:49 <dustymabe> - user noboddy is still 99 (new user won't be created)
17:25:51 <dustymabe> - FCOS user can run a script to update system and files to have noboddy user be 65534
17:26:07 <dustymabe> that last part is optional ^^
17:26:16 <lucab> (I wasn't honestly expecting to come to solutions, I was mostly looking for background as git-log didn't help)
17:26:31 <dustymabe> lucab: fair, but while we are here
17:26:51 <travier> Another option is to keep 99 as oldnobody until we move on with sysusers and add a new nobody to fix things now
17:27:02 <travier> then we can remove it
17:27:24 <dustymabe> is there anything blocking us (other than time) from making progress on sysusers?
17:27:45 <travier> some one working on it I think :)
17:27:46 <dustymabe> I vaguely remember some Fedora change coming soon that was going to force us to move anyway
17:28:00 <lucab> it's progressing at the Fedora level
17:28:11 <dustymabe> lucab: for F37?
17:28:38 <lucab> most of FCOS system users are covered already, I think chrony and another one are the missing ones
17:28:58 <lucab> dustymabe: without a target, but the building blocks are all in place in F36
17:29:28 <dustymabe> IOW we could do this anytime we like if we do the work
17:29:58 * dustymabe wishes he had an army
17:30:14 <lucab> with plenty of stragglers and corner-cases like this "nobody" one
17:30:18 <dustymabe> ok we're running out of time.. shall I add some of this context and possible proposed solution to the ticket?
17:30:49 <lucab> yes let's add more details
17:30:57 <dustymabe> will do
17:31:09 <dustymabe> I guess this was open floor so we can just close the meeting...
17:31:16 <dustymabe> will close in 60s unless new topics appear
17:31:34 <lucab> all from my side
17:31:49 <dustymabe> lucab: thanks for making this a super productive meeting
17:32:03 <travier> lucab++
17:32:17 <dustymabe> #endmeeting