16:30:13 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:13 <zodbot> Meeting started Wed Jul  6 16:30:13 2022 UTC.
16:30:13 <zodbot> This meeting is logged and archived in a public location.
16:30:13 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:13 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:18 <dustymabe> #topic roll call
16:30:22 <dustymabe> .hi
16:30:23 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:48 <jbrooks> .hello jasonbrooks
16:30:51 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:31:01 <ravanelli> .hi
16:31:03 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:31:07 <jlebon> .hello2
16:31:08 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:24 <jmarrero> .hi
16:31:25 <zodbot> jmarrero: jmarrero 'Joseph Marrero' <jmarrero@redhat.com>
16:31:42 <gursewak> .hi
16:31:43 <zodbot> gursewak: gursewak 'Gursewak Singh' <gurssing@redhat.com>
16:31:49 <jdoss> .hi
16:31:52 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:32:27 <lorbus> .hi
16:32:28 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:32:45 <dustymabe> #chair jbrooks ravanelli jlebon jmarrero gursewak jdoss lorbus
16:32:45 <zodbot> Current chairs: dustymabe gursewak jbrooks jdoss jlebon jmarrero lorbus ravanelli
16:32:56 <mnguyen> .hi
16:32:57 <zodbot> mnguyen: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:33:08 <jdoss> I have a meeting overlap in 30 so I might be in and out here.
16:33:29 <dustymabe> jdoss +1
16:33:34 <dustymabe> ok let's get started
16:33:42 <dustymabe> #topic Action items from last meeting
16:33:46 <dustymabe> * cverna to open ticket for FCOS as an edition and update
16:33:49 <dustymabe> * jlebon to open investigation tickets for the IMA/FIDO changes
16:33:59 <dustymabe> this was from two meetings ago (last week's meeting was canceled)
16:35:02 <jlebon> sorry, I was away for a while. let's re-action mine
16:35:19 <dustymabe> #action jlebon to open investigation tickets for the IMA/FIDO changes
16:35:34 <lucab> .hi
16:35:35 <zodbot> lucab: lucab 'Luca BRUNO' <lucab@redhat.com>
16:36:06 <walters> .hi
16:36:07 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:36:42 <dustymabe> #info there was an existing ticket for becoming an edition (https://github.com/coreos/fedora-coreos-tracker/issues/915) and cverna opened a ticket related to becoming an edition (https://github.com/coreos/fedora-coreos-tracker/issues/1239).
16:37:09 <dustymabe> we also filed a change request to try to make it happen this time: https://fedoraproject.org/wiki/Changes/FedoraCoreOS
16:37:31 <dustymabe> we can move on to meeting topics now
16:37:42 <dustymabe> #topic New Package Request: Crash
16:37:46 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1238
16:38:02 <jlebon> cverna++
16:38:02 <zodbot> jlebon: Karma for cverna changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:38:49 <dustymabe> as travier mentions in comment #1 to this issue: "Due to the size of the kernel debuginfo packages, it is extremely unlikely that this will ever be included in the image."
16:39:16 <dustymabe> basically the crash program needs kernel-debuginfo installed in order to do analysis IIUC
16:39:29 <jdoss> Ugh
16:40:49 <jdoss> If they are having issues, they should just layer the kernel-debuginfo and debug their specific problems.
16:41:35 <dustymabe> jdoss: I think I agree. though it looks like they are coming from the OKD world where maybe that isn't as obvious to them how to do
16:42:17 <dustymabe> I think there is a followup here that we could do (documenting this specific workflow), but before we start discussing that
16:42:30 <dustymabe> does anyone want to advocate FOR inclusion of crash?
16:43:33 <dustymabe> mic check.. is this thing on?
16:44:08 <jmarrero> not me
16:44:11 <jlebon> it's on :)
16:44:15 <lucab> I think it was a "no one" :)
16:44:16 <dustymabe> :)
16:44:45 <lucab> (not directly related, but I think we already had the same discussion for OCP/RHCOS and we may already have some instructions/steps there)
16:44:47 <dustymabe> ok let's move to a secondary topic then.. should we document this workflow for a user
16:45:35 <jdoss> on how to debug kernel problems/
16:45:39 <jdoss> ?*
16:45:41 <dustymabe> this would be a great candidate (I think) for some documentation that uses `toolbox` to install the necessary packages inside the toolbox container and analyzing the crashdump
16:45:45 <jbrooks> It's worth documenting, and it sounds like steps from ocp/rhcos could be  reused
16:46:27 <dustymabe> does anyone have a link to those steps?
16:46:55 <lucab> I guess this fits into https://docs.fedoraproject.org/en-US/fedora-coreos/debugging-kernel-crashes/
16:47:25 <dustymabe> lucab: indeed. We'd extend that documentation
16:47:28 <lucab> dustymabe: let me see if I can find again the RFE/Jira, if I'm not actually mis-remembering
16:48:03 <jlebon> i guess there's nothing actually FCOS-specific to this, right?
16:48:25 <dustymabe> jlebon: i.e. could apply to silverblue too?
16:48:28 <jlebon> cool to have this be in our docs, though could be interesting to have it upstream too if there's a good place for it
16:48:41 <jlebon> s/too/instead/
16:48:41 <dustymabe> or any fedora variant, yeah
16:48:52 <walters> (adding debugging tools into a toolbox container also applies to yum-based systems)
16:49:10 <lucab> the "how to find matching debuginfo" part, I think that depends on distro/repos layout
16:49:22 <jlebon> dustymabe: i meant upstream crash, but Fedora-level makes more sense since... yes what lucab said
16:49:24 <dustymabe> yeah. I'm cool with putting the steps somewhere else and linking to them from our docs
16:50:22 <jlebon> it's possible the debuginfod work in Fedora would make this really streamlined
16:50:58 <dustymabe> #proposed We don't think including the crash program in Fedora CoreOS is extremely useful because it needs the kernel-debuginfo packages in order to do crash dump analysis and you can grab crash at the same time those packages are downloaded. Though, we do think it would be useful to document how to do the crashdump analysis from say a toolbox container.
16:51:08 <lucab> ah, found the bugzilla RFE, but we didn't document the actual workflow there
16:51:28 <lucab> #link https://bugzilla.redhat.com/show_bug.cgi?id=2013888
16:51:42 <jlebon> ack to proposal
16:52:00 <jdoss> You are not authorized to access bug #2013888. Womp womp
16:52:02 <dustymabe> #undo
16:52:02 <zodbot> Removing item from minutes: <MeetBot.items.Link object at 0x7fe2d6a10e48>
16:52:12 <dustymabe> yeah unfortunately it's locked for some reason
16:52:35 <jdoss> OK BZ... keep your secrets
16:52:51 <dustymabe> +1 -1 to proposed?
16:52:54 <jdoss> +1
16:52:56 <jbrooks> +1
16:53:04 <jbrooks> it contains a link to https://docs.openshift.com/container-platform/4.8/support/gathering-cluster-data.html#about-toolbox_gathering-cluster-data
16:53:17 <jlebon> huh, apparently it 'Provides: bundled(gdb)' which is interesting
16:53:52 <lucab> there isn't much juice in that BZ anyway, it is basically a "no, please try running this containerized/in toolbox instead"
16:54:00 <dustymabe> #agreed We don't think including the crash program in Fedora CoreOS is extremely useful because it needs the kernel-debuginfo packages in order to do crash dump analysis and you can grab crash at the same time those packages are downloaded. Though, we do think it would be useful to document how to do the crashdump analysis from say a toolbox container.
16:54:21 <lucab> (sorry I didn't realize the whole ticket was private)
16:54:32 <jdoss> All good.
16:54:36 <dustymabe> anyone want to volunteer to do a brief test to see if they can trigger a crash dump and use toolbox to install crash and kernel-debuginfo and analyze the crashdump?
16:54:51 <dustymabe> and post the results to the ticket
16:54:58 * dustymabe moves on to next topic soon
16:55:36 <dustymabe> #topic installing non-x86_64 emulators
16:55:40 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1237
16:56:11 <dustymabe> TL;DR do we need to install emulators for non x86_64
16:56:53 <dustymabe> The original reason for doing this for x86_64 was to 'allow "out of the box" access to the large inventory of containers only built for x86_64'
16:59:13 <dustymabe> let's see who advocated for it originally
16:59:37 <dustymabe> looks like bgilbert and travier
16:59:42 <dustymabe> and rhatdan
17:00:09 <jdoss> They can't be that big right?
17:00:51 <dustymabe> according to https://github.com/coreos/fedora-coreos-tracker/issues/1088#issuecomment-1150393245 about 12M
17:01:33 <dustymabe> my concern here isn't as much "installing aarch64 emulator on x86_64" it's more about having a good reason too. Otherwise people start asking for every other arch on every arch
17:01:39 <lucab> I'm inclined to say that we don't need to, until there is some actual user asking about that
17:01:50 <jlebon> dustymabe: +1
17:02:00 <dustymabe> i.e. x86_64 now have aarch64, ppc64le, and s390x emulators
17:02:11 <dustymabe> and same for each arch
17:02:27 <jmarrero> I would enable people to use coreos to run more containers and test stuff.
17:02:34 <jmarrero> It*
17:03:11 <lucab> it's unlikely that anything using this will sit in the firstboot hotpath, so there is always client-side (and containerized) package layering covering this
17:03:21 <dustymabe> jmarrero: you are right, but i think we need to weigh the tradeoffs
17:03:35 <jlebon> if we do include it, we should check if we really need both aarch64 and aarch64_be or if one is much more prevalent than the other
17:03:41 <dustymabe> i think there is a much smaller percentage of users on x86_64 that want to run aarch64 containers
17:03:45 <jdoss> They weight about 12mb
17:04:04 <dustymabe> jlebon: the answer is no.. I asked crobinso and he said he would accept a PR to split those two out
17:04:05 <jlebon> containers that are built for aarch64 are usually also built for x86_64
17:04:12 <dustymabe> jlebon: ^^ bingo
17:04:13 <jlebon> dustymabe: +1
17:04:29 <dustymabe> usually if people are building for aarch64 they are already doing it "multi-arch" anyway
17:06:08 <jdoss> I can see users on a non-x86_64 arch needed to run x86_64 but I think the reverse is going to be very few use-cases but this is just a guess without more data.
17:06:24 <dustymabe> indeed
17:06:53 <jmarrero> I am ok saying layer the package if you needed. My only example would be running podman on mac but that is aarch64 which we agreed to add the emulator for.
17:06:55 <dustymabe> one case I can think of is if you have a development team with M1 macs targetting aarch64 EC2 isntances and some devs on your team don't have M1 macs
17:07:14 <dustymabe> but that is a stretch
17:07:47 <jmarrero> s/needed/need it/
17:07:52 <jbrooks> yeah, layering is always an option
17:08:10 <dustymabe> #proposed We included the x86_64 emulator on non-x86_64 architectures to get access to the large library of containers that are only built for x86_64 today. We don't think there is a significant enough user base wanting aarch64/ppc64le/s390x emulators for containers built just for those architectures (usually if you are building for aarch64/ppc64le/s390x you are building for all
17:08:13 <dustymabe> architectures). We'll leave the installation of the emulators up to the user for now.
17:08:22 <jdoss> +1
17:08:28 <jbrooks> +1
17:08:42 <jmarrero> +1
17:08:53 <jlebon> +1
17:08:58 <gursewak> +1
17:09:10 <dustymabe> #agreed We included the x86_64 emulator on non-x86_64 architectures to get access to the large library of containers that are only built for x86_64 today. We don't think there is a significant enough user base wanting aarch64/ppc64le/s390x emulators for containers built just for those architectures (usually if you are building for aarch64/ppc64le/s390x you are building for all
17:09:13 <dustymabe> architectures). We'll leave the installation of the emulators up to the user for now.
17:09:21 <dustymabe> #topic tracker: Fedora 37 changes considerations
17:09:26 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1222
17:09:34 <jlebon> this should be reconsidered if it's a pain point for more than a few users
17:09:52 <dustymabe> I added some new items to the description of https://github.com/coreos/fedora-coreos-tracker/issues/1222
17:10:17 <dustymabe> let's run through them real quick
17:10:27 <dustymabe> subtopic 121. Build all JDKs in Fedora against in-tree libraries and with static stdc++lib
17:10:38 <dustymabe> Skip. FCOS doesn't ship java
17:10:47 <jlebon> +1
17:10:55 <dustymabe> subtopic 122. RPM Macros for Build Flags
17:11:19 <dustymabe> I guess we could look at the packages we own to see if we want to take advantage of this?
17:12:38 <jlebon> i guess in future cleanups. not sure it's worth investigating for f37.
17:12:51 <dustymabe> +1
17:13:02 <dustymabe> skip, nothing for us to to at the distro level
17:13:15 <dustymabe> subtopic 123. Gettext Runtime Subpackage
17:14:27 <dustymabe> nothing for us to do.. I don't guess
17:14:48 <dustymabe> though I guess we could consider if we need full blown gettext or can get by with gettext-runtime
17:14:59 <jlebon> cool, we currently ship gettext so we should benefit from this
17:15:22 <dustymabe> I'm looking at the "Grouping of binaries" section
17:15:27 <dustymabe> gettext-runtime:
17:15:30 <dustymabe> envsubst gettext gettext.sh ngettext
17:15:32 <dustymabe> gettext:
17:15:34 <dustymabe> msgattrib msgcat msgcmp msgcomm msgconv msgen msgexec msgfilter msgfmt msggrep msginit msgmerge msgunfmt msguniq recode-sr-latin xgettext
17:16:02 <jlebon> it's pulled in as a dep, so would be automatic but requires that all the packages we ship that need it move over to requiring gettext-runtime IIUC
17:16:16 <dustymabe> ahh, true
17:16:39 <dustymabe> gettext is needed by (installed) grub2-pc-1:2.06-42.fc36.x86_64
17:16:49 <dustymabe> that's the only one I see, though there may be more
17:16:58 <dustymabe> anywho we don't need to do anything specific for this
17:17:03 <dustymabe> moving on to the next subtopic
17:17:15 <dustymabe> subtopix 124 Golang 1.19
17:17:16 <jlebon> ahh cool. that's a common one so I suspect will be looked at
17:17:43 <dustymabe> anything we need to do in our Go packages for go 1.19?
17:17:59 <jlebon> we should add CI for it in our go-based repos
17:18:30 <dustymabe> i'm not sure exactly what that means :) - can you handle making sure that happens?
17:18:35 <dustymabe> or open a ticket?
17:19:04 * dustymabe assumes the mass rebuild is coming up and our rawhide stream will tell us anyway
17:19:44 <jlebon> i'll chat with other Ignition and Butane maintainers. we might already have a procedure to add CI once new golang minor versions are released
17:19:58 <dustymabe> subtopic 125 Fallback Hostname
17:20:13 <dustymabe> I submitted this proposal along with other reps from different groups :)
17:20:34 <dustymabe> This should be no change for us, other than removing the workaround we've been using.
17:20:47 <jlebon> woohoo!
17:20:51 <jlebon> dustymabe++
17:20:57 <jlebon> thanks a lot for driving that
17:21:02 <dustymabe> subtopic 214 LLVM 15
17:21:15 <dustymabe> skip, I don't think we have anything to do with LLVM ?
17:21:50 <dustymabe> subtopic 215 Supplement of Server distributables by a KVM VM disk image
17:22:00 <dustymabe> skip, limited to server working group/deliverable
17:22:08 <dustymabe> subtopic 216 Erlang 25
17:22:36 <dustymabe> skip, we don't use erlang that I'm aware of
17:22:55 <dustymabe> and that's all of them
17:23:02 <jlebon> +1
17:23:15 <dustymabe> any comments before I move on? will head to open floor since we're almost out of time
17:23:54 <dustymabe> #topic open floor
17:24:32 <dustymabe> actually probably should mention https://github.com/coreos/fedora-coreos-tracker/issues/1234
17:24:39 <dustymabe> anything noteworthy we should mention for june?
17:25:24 <dustymabe> I know there was just a butane release (but that landed in july)
17:25:25 <jlebon> i think there were a couple of releases
17:25:56 <jlebon> also coreos-installer and rpm-ostree
17:26:44 <dustymabe> any other topics for open floor?
17:26:52 <dustymabe> jdoss: jbrooks: anything exciting going on?
17:27:09 <jbrooks> dustymabe, No :)
17:27:25 <lucab> I found one more GID mismatch for "tape" group -> https://github.com/coreos/fedora-coreos-tracker/issues/1201#issuecomment-1172142703
17:27:59 <lucab> how do we feel about fixing that one directly now?
17:29:07 <dustymabe> I don't know if I'm up to speed enough to have an opinion.. maybe jlebon or travier would know better?
17:29:49 <walters> good that this bug was caught on tape
17:29:56 <lucab> lol
17:30:12 * dustymabe closes out the meeting soon
17:30:17 <walters> i know at this moment you're all thinking "glad colin is in this meeting"
17:30:37 <dustymabe> i'm always happy to have you here!
17:31:11 <dustymabe> #endmeeting