fedora_coreos_meeting
LOGS
16:29:34 <nasirhm> #startmeeting fedora_coreos_meeting
16:29:34 <zodbot> Meeting started Wed Oct 21 16:29:34 2020 UTC.
16:29:34 <zodbot> This meeting is logged and archived in a public location.
16:29:34 <zodbot> The chair is nasirhm. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:29:34 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:34 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:54 <nasirhm> #topic roll call
16:30:07 <nasirhm> .hello2
16:30:08 <zodbot> nasirhm: nasirhm 'Nasir Hussain' <nasirhussainm14@gmail.com>
16:30:32 <dustymabe> .hello2
16:30:33 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:35 <slowrie> .hello2
16:30:37 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:30:45 <skunkerk> .hello sohank2602
16:30:48 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:30:59 <jdoss> .hello2
16:31:00 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:31:02 <jdoss> there we go
16:31:14 <jlebon> .hello2
16:31:15 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:22 <red_beard> .hello redbeard
16:31:23 <zodbot> red_beard: redbeard 'Brian 'redbeard' Harrington' <bharring@redhat.com>
16:31:39 <lucab> .hello2
16:31:40 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:32:26 <cyberpear> .hello2
16:32:27 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:32:37 <nasirhm> #chair lucab red_beard jlebon jdoss skunkerk slowrie cyberpear dustymabe
16:32:37 <zodbot> Current chairs: cyberpear dustymabe jdoss jlebon lucab nasirhm red_beard skunkerk slowrie
16:32:53 <nasirhm> #topic Action items from last meeting
16:33:14 <nasirhm> #info dustymabe to send an email to the coreos list (not status) with a PSA about the meeting time sticking with UTC
16:34:11 <dustymabe> #undo
16:34:11 <zodbot> Removing item from minutes: INFO by nasirhm at 16:33:14 : dustymabe to send an email to the coreos list (not status) with a PSA about the meeting time sticking with UTC
16:34:27 <dustymabe> #info dustymabe sent email to the list about meeting sticking to UTC time https://github.com/coreos/fedora-coreos-tracker/issues/650
16:34:30 <dustymabe> #undo
16:34:30 <zodbot> Removing item from minutes: INFO by dustymabe at 16:34:27 : dustymabe sent email to the list about meeting sticking to UTC time https://github.com/coreos/fedora-coreos-tracker/issues/650
16:34:36 <dustymabe> #info dustymabe sent email to the list about meeting sticking to UTC time https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/message/KTEAQSBAZD6LVCKYJKZXB3LEHK723UB2/
16:34:51 <dustymabe> done!
16:34:59 <nasirhm> Thank you dustymabe for working on it.
16:35:31 <nasirhm> #topic How to enable kdump feature on Fedora CoreOS
16:35:43 <nasirhm> #link https://github.com/coreos/fedora-coreos-tracker/issues/622
16:36:14 <dustymabe> jlebon: you added the meeting label, do you want to give a summary to start discussion?
16:36:25 <jlebon> yup sure
16:37:06 <jlebon> there have been a number of requests both internally and externally to have kdump enabled in FCOS/RHCOS
16:37:36 <jlebon> this isn't something that we would enable by default, but we want to make it easy for users to enable it if they want
16:38:12 <jlebon> tech-wise, there isn't much that needs to be done to have kdump work. it mostly Just Works already.
16:38:45 <dustymabe> so what's the open question? missing bits?
16:38:56 <jlebon> so we could in theory leave it as an extension users overlay
16:39:26 <jlebon> however, given that (1) kdump is tiny and (2) we want to reduce friction (possibly adding fcct sugar), there's general consensus that we should just bake it in
16:40:05 <jlebon> this ties into a few things, notably that by baking it in, it's easier to have it from day-1
16:40:26 <dustymabe> i think it makes sense, but i'm always the easy one to convince to include something
16:40:27 <jlebon> we do eventually want to support "day-1 extensions" but there's still a lot of work that needs to happen
16:40:45 <dustymabe> it might be worth at least answering the questions that we came up with recently for requests like this
16:40:55 <jlebon> the other piece of "day-1" is karg support via Ignition which is likely much easier
16:41:14 <jlebon> dustymabe: yeah, that's a good idea
16:41:20 <dustymabe> the list: https://github.com/coreos/fedora-coreos-tracker/issues/641#issue-712286532
16:41:27 <dustymabe> Can the package be run from a container?
16:41:29 <dustymabe> Is it possible to layer the package onto the base OS as a day 2 operation?
16:41:32 <dustymabe> Does the package have additional dependencies? (i.e. does it drag in Python, Perl, etc)
16:41:34 <dustymabe> What is the size of the package?
16:41:35 <dustymabe> Can the packaging be adjusted to just deliver binaries?
16:41:37 <dustymabe> What is the intended use case of the package?
16:41:40 <dustymabe> How can this package be abused as a Turing complete interpreter?
16:41:49 <jlebon> probably should've numbered those :)
16:41:59 <dustymabe> one sec
16:42:25 <dustymabe> 1   Can the package be run from a container?
16:42:27 <dustymabe> 2   Is it possible to layer the package onto the base OS as a day 2 operation?
16:42:29 <dustymabe> 3   Does the package have additional dependencies? (i.e. does it drag in Python, Perl, etc)
16:42:31 <dustymabe> 4   What is the size of the package?
16:42:33 <dustymabe> 5   Can the packaging be adjusted to just deliver binaries?
16:42:35 <dustymabe> 6   What is the intended use case of the package?
16:42:38 <dustymabe> 7   How can this package be abused as a Turing complete interpreter?
16:42:53 <bgilbert> .hello2
16:42:53 <jlebon> 1. probably could be shoehorned into a container... though it's pretty "host-level". e.g. it needs to write to /boot
16:42:53 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:43:12 <nasirhm> #chair bgilbert
16:43:12 <zodbot> Current chairs: bgilbert cyberpear dustymabe jdoss jlebon lucab nasirhm red_beard skunkerk slowrie
16:43:32 <jlebon> 2. it is, we're discussing here reducing friction further
16:44:06 <jlebon> 3. not AFAIK, need to sanity-check that
16:44:11 <red_beard> jlebon: to be fair, it's just "writing to part of the host filesystem", that it's /boot is incidental
16:44:48 <red_beard> citing that as a reason exacerbates the meta-problem of folks thinking content _needs_ to be in the host mount namespace
16:45:41 <walters> i do think the dracut part should be containerized, not necessarily via podman; that's what rpm-ostree does with bwrap when you do `rpm-ostree initramfs --enable`
16:45:53 <red_beard> i'm not disagreeing with the sentiment that kdump is useful or even disagreeing with it's inclusion, just suggesting a bit more rigor with the justification
16:46:03 <jlebon> 4. RPM pkg comes in at 486k, including docs
16:46:20 <dustymabe> here's what I see when I layer it
16:46:22 <dustymabe> Added:
16:46:24 <dustymabe> dracut-squash-050-61.git20200529.fc32.x86_64
16:46:26 <dustymabe> ethtool-2:5.9-1.fc32.x86_64
16:46:28 <dustymabe> kexec-tools-2.0.20-17.fc32.x86_64
16:46:30 <dustymabe> snappy-1.1.8-2.fc32.x86_64
16:46:32 <dustymabe> squashfs-tools-4.3-25.fc32.x86_64
16:47:02 <dustymabe> just a datapoint ^^
16:47:23 <jlebon> red_beard: true, but /boot is pretty special though
16:47:43 <jlebon> it's dropping things directly into the ostree /boot fs layout
16:48:08 <jlebon> so technically it'd be cleaner to have it bound to ostree lifecycle wise
16:48:46 <jlebon> i probably should've added those details earlier :)
16:49:41 <jlebon> dustymabe: thanks. hmm yeah we should probably dig a bit into those
16:49:44 <dustymabe> so what about question 6 and 7
16:50:04 <dustymabe> 6. to support kdump.service ?
16:50:16 <jlebon> 6. intended use case is kdump enablement
16:51:37 <jlebon> 7. i don't know enough about kdump, but unlikely :)
16:52:20 <dustymabe> i'm sure some fun could be had with kexec, but if a bad actor can run kexec (privileged), you're owned already
16:52:33 <jlebon> we should think of this more as "growing the host API" rather than "baking in an extension"
16:52:58 <cyberpear> sounds somewhat related to the bootupd project?
16:53:01 <jlebon> i.e. do we want FCOS/RHCOS to support kdump OOTB?
16:54:06 <dustymabe> i think it makes sense, especially for something so low level, to not run it in a container.
16:54:21 <dustymabe> it also would be nice to be able to support it OOTB
16:54:34 <dustymabe> though I don't know specifically how many people would enable it
16:54:39 <jlebon> cyberpear: hmm, can you expand? they're mostly orthogonal i think (except that both do things in /boot)
16:54:49 <red_beard> cyberpear: that *is* a good point
16:55:14 <nasirhm> Can some please provide some information on OOTB.
16:55:23 <slowrie> out of the box
16:55:24 <dustymabe> out of the box
16:55:27 <cyberpear> jlebon: i figured they might be similar since both muck w/ /boot
16:55:28 <dustymabe> slowrie: youfast
16:55:46 <nasirhm> Ah, Thanks :)
16:55:57 <slowrie> dustymabe: I stopped coughing long enough to type a response :)
16:56:02 <red_beard> "Linux systems handle updates for bootloader data in an inconsistent and ad-hoc way. .... The goal of [Bootupd] project is to be a cross-distribution, OS update system agnostic tool to manage updates for things like /boot/efi...."
16:56:34 <jlebon> cyberpear: right they're similar in that respect :)
16:56:45 <cyberpear> okay, so maybe it's a difference of /boot vs /boot/efi, therefore unrelated to bootupd?
16:58:20 <walters> bootupd only owns the bootloaders, `/boot` in fact doesn't contain any of that although bootupd stores its states there
16:58:37 <red_beard> actually... `The scope is otherwise limited; for example, bootupd will not manage anything related to the kernel such as kernel arguments; that's for tools like grubby and ostree.`
16:58:45 <walters> IOW yes, not related to bootupd
16:58:51 <dustymabe> so is anyone opposed to us trying to support this mechanism (used for crash investigation) more natively by including the kexec-tools package in the base layer?
16:59:15 <jlebon> /boot isn't a single thing. ostree manages /boot/loader and /boot/ostree, bootupd manages the MBR and /boot/efi
16:59:16 * cverna waves
16:59:33 <dustymabe> 👋
16:59:35 <nasirhm> #chair cverna
16:59:35 <zodbot> Current chairs: bgilbert cverna cyberpear dustymabe jdoss jlebon lucab nasirhm red_beard skunkerk slowrie
16:59:41 <red_beard> i'd like to hear more about how this gets configured _consistently_ so that folks aren't YOLOing their way through the setup
17:00:14 <jlebon> red_beard: good point. ideally the FCCT sugar would help a lot with this
17:00:24 <red_beard> otherwise, we're just getting back to creating another general purpose distro
17:00:52 <bgilbert> red_beard: it'd be pretty much what you'd expect: docs with config fragments to copypaste, and/or FCC sugar
17:01:12 <slowrie> I think in an ideal world: Ignition sets the kargs (and reboots during first boot initrd), enables kdump service. On every boot the kdump daemon checks if a crashkernel exists for the running kernel (generates if not present). FCC sugar to generate that Ignition snippet automatically. Did I miss anything?
17:02:05 <red_beard> so, related to the inclusion, we should have tasks to cover the generation of the docs & config fragments and issues filed for the sugar for FCCT, right?
17:02:56 <red_beard> to mark this as "done", that is
17:03:01 <red_beard> if it gets included
17:03:59 <jlebon> slowrie: maybe. i think it depends on how kdump.conf is configured in the wild. we may want sugar for that too, or it might be too broad
17:04:22 <jlebon> red_beard: that makes sense to me
17:05:15 <slowrie> jlebon: yeah; for the FCCT sugar ideally we scope it down to the littlest common set as possible with docs pointing to writing your own kdump.conf if you want more configuration
17:06:02 <jlebon> hmm, looks like kdump.conf doesn't support drop-ins. something to look into
17:06:32 <bgilbert> the sugar could allow you to write arbitrary kdump.conf contents
17:06:54 <lucab> when enabled, this also locks a bunch of memory in kernel-space, so we are not yet sure if people usually turn it on at install time, or just in time when needed
17:07:00 <jlebon> brb
17:08:05 <dustymabe> anyone want to take a stab at closing off this discussion topic?
17:08:15 <jlebon> sorry, back now
17:08:20 <jlebon> dustymabe: i can try a proposa
17:08:21 <jlebon> l
17:10:06 <jlebon> #proposed we will include kexec-tools in FCOS to make crash collection built in. we will add documentation and possibly FCCT sugar to standardize how it's configured. we will investigate dependencies that it pulls in.
17:10:32 <dustymabe> ack
17:10:35 <bgilbert> +1
17:10:46 <nasirhm> +1
17:10:57 <slowrie> +1
17:10:59 <dustymabe> jlebon: if you don't mind can you try to answer 1-7 questions in the ticket, just for the record ?
17:11:06 <skunkerk> +1
17:11:23 <jlebon> #agreed we will include kexec-tools in FCOS to make crash collection built in. we will add documentation and possibly FCCT sugar to standardize how it's configured. we will investigate dependencies that it pulls in.
17:11:27 <jlebon> dustymabe: ack, sure
17:11:53 <dustymabe> red_beard: cyberpear: did that proposal look good?
17:11:57 <jlebon> sorry, maybe i didn't wait long enough.  red_beard, lucab any reservations?
17:12:08 <red_beard> +1
17:12:10 <cyberpear> yeah, seems fine
17:12:40 <dustymabe> cool
17:12:45 <jlebon> cool, thanks!
17:13:15 <nasirhm> Hmm, So let's move to our next ticket for today
17:13:42 <nasirhm> #topic Fedora 33 rebase work
17:13:50 <nasirhm> #link https://github.com/coreos/fedora-coreos-tracker/issues/609
17:14:03 <dustymabe> #info there is a new `next` release coming out today/tomorrow
17:14:07 <nasirhm> dustymabe: would you like to provide some context
17:14:21 <dustymabe> Over the past two weeks we have:
17:14:40 <dustymabe> added the migration to systemd-resolved: https://github.com/coreos/fedora-coreos-tracker/issues/646
17:14:50 <dustymabe> fixed the chrony dhcp propagation test: https://github.com/coreos/fedora-coreos-tracker/issues/643
17:15:08 <dustymabe> fixed the ext.config.root-reprovision* tests: https://github.com/coreos/fedora-coreos-tracker/issues/644
17:16:00 <dustymabe> stopped downgrading the crypto policy for SSH RSA-1 keyex: https://github.com/coreos/fedora-coreos-config/pull/702
17:16:27 <dustymabe> we have a new potential issue that did pop up
17:16:52 <dustymabe> which is that in f33 it seems like systems get the hostname of `fedora` instead of `localhost` if there is no other hostname given: https://github.com/coreos/fedora-coreos-tracker/issues/649
17:17:03 <dustymabe> which seems to be affecting OKD (I still need to look into it)
17:17:26 <nasirhm> Thanks for the info about the highlights dustymabe , ++ to everyone who made it possible :)
17:17:33 <dustymabe> oh also it's worth noting that dnsmasq is in the base layer now https://github.com/coreos/fedora-coreos-tracker/issues/519
17:18:07 <dustymabe> and ICYMI: systemd-networkd can be package layered if you really want to: https://github.com/coreos/fedora-coreos-tracker/issues/610#issuecomment-707809280
17:18:26 * dustymabe done
17:18:34 <dustymabe> oh actually
17:18:47 <dustymabe> we're trying to schedule a test day for the F33 rebase for FCOS
17:18:59 <dustymabe> Nov 6th was the tentative date
17:19:38 <nasirhm> dustymabe: should i assign an action item for you to work on the hostname issue
17:20:05 <dustymabe> and we made a badge for participation in the test day: https://pagure.io/fedora-badges/issue/782
17:20:16 <jlebon> woah nice!
17:20:18 <dustymabe> nasirhm: I think I've got it
17:20:21 <dustymabe> no need for action item
17:20:23 <nasirhm> Do we have our test cases documented for people to follow for the test day ?
17:20:37 <nasirhm> dustymabe: Cool :)
17:21:32 <dustymabe> nasirhm: not yet, there will be a test day page created in the coming week
17:22:34 <nasirhm> great, I would love to work on documenting them for the test case :)
17:24:14 <nasirhm> Time for open floor
17:24:26 <nasirhm> #topic Open Floor
17:25:50 <dustymabe> I've got nothing for open floor I don't think
17:25:54 <nasirhm> Anybody got anything to dicuss for the open floor ?
17:25:59 <jlebon> same here
17:26:35 <cyberpear> docs
17:27:08 <nasirhm> cyberpear: docs ?
17:27:17 <cyberpear> we don't ship docs "because we don't ship info or man" -- would it make sense to ship a "tools" container built-in so that we ship docs (in the container) and a way of  viewing them (also in the container)?
17:28:11 <lucab> ship == build and publish to a registry?
17:28:16 <jlebon> cyberpear: "built-in" as in part of the OS image?
17:28:32 <cyberpear> I was thinking as part of the image, but ship to a registry would be a good start, too
17:28:48 <nasirhm> cyberpear: That sounds straight froward, but I'm wondering what benefit would it provide to add all those docs in the OS (when we're currently not shipping man pages too.)
17:29:10 <red_beard> cyberpear: one thing we had discussed at CoreOS early on was having man pages hosted on a site and then an alternate `man` command which sent the version of the OS in a header and streamed the man page to the user
17:29:14 <cyberpear> the default fedora container doesn't include docs either, and even if it did, it wouldn't have rpm-ostree docs since that's not shipped in the Fedora Base contianer
17:29:27 <dustymabe> red_beard: i've had similar ideas in the past
17:29:39 <red_beard> then, it's also easy to package that _into_ a container for users to ship
17:30:05 <red_beard> specifically, i'd say that idea goes back to March 2014 - https://github.com/girlpages/ :D
17:30:08 <cyberpear> the streamed-man-pages sounds cool, but probably doesn't exist yet?
17:30:15 <dustymabe> definitely doesn't exist
17:30:55 <red_beard> not as of yet, basically it needs CI to take the man pages from the builds and push them into a http location (rendered from nroff/troff)
17:31:38 <cyberpear> what's the biggest reason we don't ship `man`? is it size of the docs?
17:31:45 <red_beard> and translations
17:32:13 <jlebon> it also pulls in perl
17:32:19 <cverna> we have tools container (https://src.fedoraproject.org/container/tools/blob/master/f/Dockerfile) that is not really maintained but if someone is interested, it should be straight forward to rebuild and add the docs
17:32:25 <cyberpear> man doesn't pull in perl, does it?
17:32:44 <nasirhm> cverna: I can take care of it, as I'm currently working on rebuilding some of the containers.
17:33:30 <bgilbert> cyberpear: also, you're not really supposed to be running things from the FCOS command line :-)
17:33:31 <red_beard> cyberpear: yes, it does: https://koji.fedoraproject.org/koji/rpminfo?rpmID=23454403
17:34:21 <cverna> bgilbert: +1 :)
17:34:25 <dustymabe> bgilbert: though.. if you're developing against a target environment, I can see running things on the command line in order to figure out what you need to put in an Ignition config as valid
17:34:36 <cyberpear> red_beard: it doesn't get pulled when I install it on F33
17:34:45 <cyberpear> maybe it's a build requirement?
17:34:45 <bgilbert> dustymabe: yeah, I'm +1 to providing a way to get corresponding docs
17:35:03 <dustymabe> cyberpear: maybe you have perl already installed
17:35:12 <red_beard> i dunno, that "requires: perl-interpreter" seems pretty explicit
17:35:32 <jlebon> that's on the SRPM though. the binary rpm indeed doesn't: https://koji.fedoraproject.org/koji/rpminfo?rpmID=23228846
17:35:46 <cyberpear> not sure how to read the koji page, but rpm -qp --requires shows no perl
17:36:01 <dustymabe> jlebon: +1
17:36:02 <jlebon> though doing `dnf remove man-db` in my pet container does want to remove some perl pkgs, so i suspect it gets pulled in by one of its deps
17:36:20 <dustymabe> one sec
17:36:53 <dustymabe> Added:
17:36:55 <dustymabe> gdbm-libs-1:1.18.1-3.fc32.x86_64
17:36:57 <dustymabe> groff-base-1.22.3-22.fc32.x86_64
17:36:58 <dustymabe> libpipeline-1.5.2-2.fc32.x86_64
17:37:00 <dustymabe> man-db-2.9.0-3.fc32.x86_64
17:37:52 <dustymabe> not sure we'd want to include, but that's the depchain
17:37:54 <red_beard> i'm very pleased to be incorrect then.
17:37:54 <jlebon> oh neat. either this changed recently, or i was plain wrong :)
17:38:14 <dustymabe> we're a bit over, maybe we can continue discussion later?
17:38:38 <nasirhm> +1 to discuss later in the #fedora-coreos channel or in the next meeting.
17:38:48 <jlebon> +1  FWIW, i too would really love easily accessible man pages versioned to the OS
17:38:56 <cyberpear> those 4 packages add 4.1M to the install AFAICT
17:39:03 <cyberpear> happy to end it here, though
17:39:26 <nasirhm> are we good to end ?
17:39:31 <bgilbert> nasirhm: +1
17:39:42 <red_beard> +1
17:39:46 <nasirhm> #endmeeting