16:30:34 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:34 <zodbot> Meeting started Wed May 19 16:30:34 2021 UTC.
16:30:34 <zodbot> This meeting is logged and archived in a public location.
16:30:34 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:34 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:34 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:38 <dustymabe> #topic roll call
16:30:41 <dustymabe> .hi
16:30:42 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:50 <copperi_> .hello copperi
16:30:51 <zodbot> copperi_: copperi 'Jan Kuparinen' <copper_fin@hotmail.com>
16:31:31 <jdoss> .hi
16:31:32 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:31:38 <jaimelm> .hello
16:31:38 <zodbot> jaimelm: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:31:49 <cyberpear> .hi
16:31:49 <jlebon> .hello2
16:31:49 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:31:52 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:55 <jaimelm> .hello2
16:31:56 <zodbot> jaimelm: jaimelm 'Jaime Magiera' <jaimelm@umich.edu>
16:32:12 <jbrooks> .hello jasonbrooks
16:32:13 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:32:32 <lorbus> .hi
16:32:32 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:33:15 <dustymabe> #chair copperi_ jdoss jaimelm cyberpear jlebon jbrooks
16:33:15 <zodbot> Current chairs: copperi_ cyberpear dustymabe jaimelm jbrooks jdoss jlebon
16:33:15 <darkmuggle> .hello2
16:33:17 <zodbot> darkmuggle: darkmuggle 'None' <me@muggle.dev>
16:33:19 <dustymabe> #chair darkmuggle
16:33:19 <zodbot> Current chairs: copperi_ cyberpear darkmuggle dustymabe jaimelm jbrooks jdoss jlebon
16:33:39 <dustymabe> we have a lot of representation on the 'J' letter names in this meeting
16:33:54 <dustymabe> c c d d d j j j j
16:34:07 <lucab> .hello2
16:34:08 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:34:28 <dustymabe> #chair lucab
16:34:28 <zodbot> Current chairs: copperi_ cyberpear darkmuggle dustymabe jaimelm jbrooks jdoss jlebon lucab
16:34:58 * lorbus is here too :) making it two L's in the list
16:34:59 <lorbus> .hi
16:35:00 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:35:27 <dustymabe> #chair lorbus
16:35:27 <zodbot> Current chairs: copperi_ cyberpear darkmuggle dustymabe jaimelm jbrooks jdoss jlebon lorbus lucab
16:35:31 <dustymabe> ok let's get started
16:35:37 <dustymabe> #topic Action items from last meeting
16:35:45 <jaimelm> #couch lorbus
16:35:50 <bgilbert> .hello2
16:35:51 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:35:58 <dustymabe> #chair bgilbert
16:35:58 <zodbot> Current chairs: bgilbert copperi_ cyberpear darkmuggle dustymabe jaimelm jbrooks jdoss jlebon lorbus lucab
16:36:11 <dustymabe> IIUC We had no action items from last meeting.
16:36:14 <dustymabe> So nothing to discuss here
16:36:20 <lorbus> jaimelm: that's about right :D
16:36:42 <dustymabe> ok, moving on to meeting tickets
16:36:52 <dustymabe> #topic Cannot upgrade from N-2 releases due to missing GPG key
16:36:59 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/749
16:37:15 <dustymabe> I think we've mostly discussed this and come to a conclusion on how to handle it in the future
16:37:21 <dustymabe> I posted a workaround in the bug
16:37:33 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-843446996
16:37:54 <dustymabe> #info posted a workaround for systems that hit this problem in the bug at https://github.com/coreos/fedora-coreos-tracker/issues/749#issuecomment-843446996
16:38:21 <jaimelm> pretty straight forward
16:38:29 <dustymabe> the final piece of the puzzle I think (for the future) is some state checking on zincati's part? Anything left to the discuss there? Is there an open ticket for that work?
16:38:56 <jlebon> not yet, want me to file one lucab?
16:41:10 <lucab> I think we had one but can't find it, so yes please
16:41:40 <dustymabe> jlebon: if you would link to that ticket from #749 and we'll close it when it lands in zincati
16:42:02 <jlebon> done :)
16:42:20 <jlebon> #link https://github.com/coreos/zincati/issues/558
16:42:45 <dustymabe> woot woot
16:42:48 <dustymabe> ok next topic then
16:43:01 <dustymabe> #topic systemd-oomd for Fedora CoreOS
16:43:07 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/840
16:43:32 <dustymabe> in an effort to keep more in sync with Fedora, I'm bringing up systemd-oomd
16:43:45 <dustymabe> I thought it required swap, but that's not true any longer
16:44:51 <dustymabe> any thoughts on us enabling systemd-oomd?
16:44:57 <jdoss> +1
16:45:05 <jdoss> I also think the zswap stuff is pretty great too
16:45:42 <dustymabe> jdoss: yeah, we previously had that discussion and I think we can re-visit zswap in the future. kubernetes just accepted a KEP to support swap enabled systems.
16:46:38 <dustymabe> hmm. everyone is quiet today
16:46:51 <dustymabe> let's see if this gets any action
16:47:29 <dustymabe> #proposed considering systemd-oomd can work on systems without swap we will enable it in Fedora CoreOS like other editions of Fedora do
16:47:34 <dustymabe> ack/nack?
16:47:58 <jlebon> what are the implications for OKD/k8s?
16:48:20 <jlebon> if there are any
16:48:52 <dustymabe> jaimelm: might know, i'm not sure
16:49:02 <jaimelm> I can't think of any, but will bring it to the group.
16:49:02 <lorbus> Afaik there's swap support coming to K8s soon, but it's not there yet. Idk what impact using oomd would have tho
16:49:30 <dustymabe> also I need to understand a little more to know if systemd-oomd has requirements (like cgroups v2 for some reason)
16:49:38 <jaimelm> I'll take that as a task
16:49:42 <lorbus> thanks jaimelm, raising that in the OKD WG would be great
16:49:53 <cyberpear> oomd seems nicer than the OOM Killer. I wonder what its corner cases are...
16:49:57 <jlebon> apart from that in general I don't see why we shouldn't match
16:50:02 <dustymabe> #action jaimelm to ask the OKD working group if there are any implications that systemd-oomd would have on k8s/OKD
16:50:03 <darkmuggle> In the case of FCOS, I would urge caution
16:50:08 <lorbus> jlebon: agree :100\
16:50:12 <lorbus> * jlebon: agree :100:
16:50:44 <darkmuggle> Matching makes sense to be me, but we might need some config to handle the headless nature of FCOS
16:51:11 <dustymabe> darkmuggle: meaning the defaults might need to be tweaked?
16:51:30 <darkmuggle> Correct.
16:51:43 <darkmuggle> I'm imagining a situation where rpm-ostree or sshd is OOM'd
16:51:44 <dustymabe> i can check to see if Server has different defaults than Workstation
16:52:13 <dustymabe> darkmuggle: indeed. I think you can specify in your systemd unit if you want your process to be ignored from oomd consideration
16:52:17 <darkmuggle> That said, in general, I'm in favor. Just make sure that the defaults are sensible
16:53:00 <bgilbert> hmm, the oomd manpage recommends swap 'to give oomd enough time to react'
16:54:12 <dustymabe> bgilbert: meaning the system might run out of memory before oomd can kill something?
16:54:43 <darkmuggle> More like it gives systemd-oomd time to react
16:54:50 * dustymabe just assumes the kernel OOM would be the fallback in that case
16:54:51 <bgilbert> the manpage doesn't make it clear whether oomd livelocks or whether the kernel OOM killer takes over
16:55:04 <bgilbert> that seems... extremely hacky though
16:55:18 * bgilbert has not dug into this at all
16:55:25 <dustymabe> no worries. It's new
16:55:31 <dustymabe> let's regroup on this next week?
16:55:39 <jlebon> feels like we should have someone spike on this and play around with it
16:55:48 <darkmuggle> I think that would be the sensible course to play with it.
16:55:49 <jaimelm> might be good enable it, throw some memory tests at FCOS, and see what defaults work best under various scenarios.
16:55:51 <lucab> I don't think it really works without cgroupsv2 though
16:56:04 <jlebon> otherwise, it's more or less the blind leading the blind :)
16:56:10 <lucab> the underlying PSI primitive does not really exist on v1
16:56:17 <darkmuggle> Having dealt with OOM on a massive fleet of servers, this is topic that makes my spidey sense tingle.
16:56:19 <dustymabe> lucab: I was interested in knowing that
16:56:26 <jaimelm> dustymabe: more investigation is needed. Let's move on?
16:56:38 <travier> .hello siosm
16:56:39 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:56:48 <dustymabe> jaimelm: correct, but one more thing
16:57:17 <dustymabe> i would really like to get some volunteers at some point. I'd like to not let things like this linger forever
16:57:27 <dustymabe> because then we diverge from Fedora because "we don't know what we don't know"
16:57:55 <dustymabe> we'll circle back in a week. hopefully there is some more information by then.
16:58:02 <darkmuggle> If someone wants to swarm with me on that, I'll throw my hat in the ring for this one.
16:58:14 <dustymabe> #dustymabe to find out how systemd-oomd acts under cgroups v1 (if at all)
16:58:19 <jaimelm> right. let's rangle some volunteers. I've been working on a template for Calls to Action. Happy to leverage it once we know more so that we know what to ask folks to do.
16:58:20 <lorbus> #chair siosm
16:58:20 <zodbot> Current chairs: bgilbert copperi_ cyberpear darkmuggle dustymabe jaimelm jbrooks jdoss jlebon lorbus lucab siosm
16:58:25 <dustymabe> darkmuggle: that'd be great!
16:58:26 <jaimelm> wrangle*
16:58:38 <dustymabe> bgilbert: are you interested too, or you good with whatever darkmuggle finds?
16:58:47 <dustymabe> #action dustymabe to find out how systemd-oomd acts under cgroups v1 (if at all)
16:58:59 <bgilbert> I'll trust darkmuggle :-)
16:59:22 <dustymabe> #action darkmuggle to investigate systemd-oomd (in use without swap) and report back next week
16:59:39 <jlebon> darkmuggle++ dustymabe++
16:59:41 <zodbot> jlebon: Karma for darkmuggle changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:59:43 <dustymabe> #info jaimelm dustymabe and darkmuggle will look into some open questions about systemd-oomd and report back next week
16:59:44 <zodbot> jlebon: Karma for dustymabe changed to 4 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:00:00 <jaimelm> good stuff
17:00:22 * dustymabe skipping the remaining topic (tagged for our next video meeting)
17:00:26 <dustymabe> #topic open floor
17:00:55 <dustymabe> #info the stable stream has been rebased to Fedora 34
17:01:00 <lucab> dustymabe: https://github.com/systemd/systemd/blob/v248/src/oom/oomd.c#L154
17:01:34 <jaimelm> dustymabe: that was a well-written email btw.
17:02:13 <jlebon> yay f34 🎉
17:02:16 <dustymabe> lucab: nice. I'll still look into whether the systmed-oomd service blows up or not
17:02:27 <dustymabe> or if it just gracefully does nothing
17:02:31 <cyberpear> 🎉 happy to see the rebase happen faster this time!
17:02:35 <lorbus> :tada:
17:02:36 <dustymabe> jaimelm: thanks :)
17:03:24 <dustymabe> any other topics for open floor?
17:03:31 <lorbus> That emoji now seemingly triggers a full-chat-screen-celebration on the Element client 🎉 😂
17:03:49 * lorbus < https://matrix.org/_matrix/media/r0/download/matrix.org/cesNDFVyVGrxEgxHSYxClbJg/message.txt >
17:04:16 <jlebon> and we already have the next ticket for tracking f35 (https://github.com/coreos/fedora-coreos-tracker/issues/824)
17:04:40 <dustymabe> #info we re-enabled the rawhide mechanical stream and are opening bugs when our CI tests fail
17:04:41 <cyberpear> lorbus: IRC folks just see a paste link
17:04:52 <bgilbert> lorbus: I tend to think we should respond in some way, but consistent with what Fedora as a whole does
17:04:56 <dustymabe> this will help us track fedora changes and react much more quickly
17:05:13 <bgilbert> e.g. we don't control where meetbot runs
17:06:25 * lorbus < https://matrix.org/_matrix/media/r0/download/matrix.org/iuLYjLqKujJapGhjRkYIVvrp/message.txt >
17:06:38 <bgilbert> lorbus: still a link
17:06:56 <bgilbert> because of the long message AIUI
17:07:15 <dustymabe> yeah. I can read it though
17:07:19 <dustymabe> This is an FYI:
17:07:21 <dustymabe> You may have heard about the hostile takeover of Freenode that took place today (!)
17:07:23 <dustymabe> https://blog.bofh.it/debian/id_461
17:07:25 <lorbus> You may have heard about the hostile takeover of Freenode that took place today (!)
17:07:25 <dustymabe> Most of the freenode staff have resigned and moved to Libera.Chat:
17:07:27 <dustymabe> https://libera.chat/news/welcome-to-libera-chat
17:07:29 <dustymabe> There is already a card on the Fedora Council tracker for this as well:
17:07:31 <dustymabe> https://pagure.io/Fedora-Council/tickets/issue/371
17:07:33 <dustymabe> Does (or should) the FCOS community have a stance on this?
17:07:35 <dustymabe> Full Disclosure: I'm in favour of moving all Fedora channels from Freenode to Libera in view of these developments
17:07:41 <cyberpear> thanks
17:07:48 <lorbus> thanks dustymabe :)
17:08:06 <dustymabe> my personal opinion
17:08:10 <dustymabe> let's do whatever fedora does
17:08:29 <jlebon> dustymabe: +1
17:08:33 <dustymabe> if #fedora-meeting-1 is now on something other than freenode, then we'll move our channels to that new server platform
17:08:42 <jaimelm> Those are lots of words to throw at a person (e.g. "Trumpian"). I haven't investigated it. If true, support packing up and moving. That said, I trust but verify.
17:08:55 <bgilbert> in https://pagure.io/Fedora-Council/tickets/issue/371 it looks like a consensus is developing to move to libera
17:08:58 <dustymabe> in this community I'd like to focus on the tech.. and thankfully we have Fedora to lean on for the other stuff
17:09:41 <darkmuggle> Going with Fedora just makes sense here since its where others in the greater ecosystem will play
17:09:48 <jaimelm> true
17:09:51 <dustymabe> WFM
17:09:51 <bgilbert> darkmuggle: +1
17:10:13 * dustymabe knows there is some matrix bridging and stuff going on. I've been wanting to play with it for a long time
17:10:20 <dustymabe> but it's always been at the bottom of my priority list
17:10:35 <dustymabe> maybe now.. maybe not
17:10:42 <dustymabe> any other topics for open floor?
17:10:56 <lorbus> fwiw I agree with all the statements/concerns above, just wanted to raise awareness. Thanks all!
17:11:05 <dustymabe> lorbus: thanks for doing that!
17:11:07 <dustymabe> lorbus++
17:11:07 <zodbot> dustymabe: Karma for lorbus changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:11:15 <copperi_> +1
17:12:17 * dustymabe will close meeting in 30 seconds if no other chatter happens
17:12:29 <cyberpear> countme should integrate with the updates polling, IMO
17:12:43 <cyberpear> but I can take that to #fedora-coreos
17:12:51 <jaimelm> we were thiiiiissss close
17:13:04 <bgilbert> lorbus: thanks for raising!
17:13:07 <dustymabe> haha.. yeah #fedora-coreos if you don't mind
17:13:10 <dustymabe> #endmeeting