fedora_coreos_meeting
LOGS
16:30:53 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:53 <zodbot> Meeting started Wed Sep  5 16:30:53 2018 UTC.
16:30:53 <zodbot> This meeting is logged and archived in a public location.
16:30:53 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:53 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:53 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:58 <dustymabe> #topic roll call
16:31:09 <bhavin192> .hello2
16:31:10 <zodbot> bhavin192: bhavin192 'Bhavin Gandhi' <bhavin7392@gmail.com>
16:31:17 <rfairley> .hello rfairleyredhat
16:31:18 <zodbot> rfairley: rfairleyredhat 'Robert Fairley' <rfairley@redhat.com>
16:31:30 <dustymabe> .hello2
16:31:31 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:45 <bgilbert> .hello2
16:31:46 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:31:50 <ksinny> .hello sinnykumari
16:31:50 <lorbus[m]> .hello lorbus
16:31:50 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com>
16:31:53 <zodbot> lorbus[m]: lorbus 'Christian Glombek' <c@petersen-glombek.de>
16:31:55 <kaeso> .hello lucab
16:32:00 <zodbot> kaeso: lucab 'Luca Bruno' <lucab@redhat.com>
16:32:04 <yzhang> .hello2
16:32:05 <zodbot> yzhang: yzhang 'Yu Qi Zhang' <jzehrarnyg@gmail.com>
16:32:06 <jbrooks> .fas jasonbrooks
16:32:09 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:32:26 <mskarbek> .hello2
16:32:27 <zodbot> mskarbek: mskarbek 'None' <redhat@skarbek.name>
16:32:31 <strigazi> .hello2
16:32:34 <zodbot> strigazi: strigazi 'Spyros Trigazis' <strigazi@gmail.com>
16:32:40 <dustymabe> yay strigazi
16:32:59 <strigazi> :)
16:33:10 <strigazi> I beat the traffic
16:33:13 <dustymabe> #chair bhavin192 rfairley bgilbert ksinny lorbus[m] kaeso yzhang jbrooks mskarbek strigazi
16:33:13 <zodbot> Current chairs: bgilbert bhavin192 dustymabe jbrooks kaeso ksinny lorbus[m] mskarbek rfairley strigazi yzhang
16:33:24 <slowrie> .hello2
16:33:25 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:33:26 <sayan> .hello sayanchowdhury
16:33:28 <zodbot> sayan: sayanchowdhury 'Sayan Chowdhury' <sayan.chowdhury2012@gmail.com>
16:33:57 <mnguyen_> .hello mnguyen
16:33:58 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:34:51 <dustymabe> #chair mnguyen_ slowrie sayan
16:34:51 <zodbot> Current chairs: bgilbert bhavin192 dustymabe jbrooks kaeso ksinny lorbus[m] mnguyen_ mskarbek rfairley sayan slowrie strigazi yzhang
16:35:42 <dustymabe> #topic previous meeting action items
16:35:56 <dustymabe> * dustymabe to create a FCOS tracker ticket for toolbox v2 design
16:35:58 <dustymabe> discussion
16:36:00 <dustymabe> * ajeddeloh to tag closed FCOS issues into design-doc PRs
16:36:02 <dustymabe> * dustymabe to add additional discussion to
16:36:04 <dustymabe> coreos/fedora-coreos-tracker#35
16:36:47 <dustymabe> #info dusty created FCOS tracker for toolbox v2 design https://github.com/coreos/fedora-coreos-tracker/issues/38
16:37:18 <dustymabe> #info dusty added additional discusstion to https://github.com/coreos/fedora-coreos-tracker/issues/35
16:38:42 <dustymabe> I'll re-action andrew for now
16:39:03 <dustymabe> #action ajeddeloh to tag closed FCOS issues into design-doc PRs
16:39:20 <dustymabe> I'll move into meeting items
16:39:36 <ajeddeloh> .hello2
16:39:37 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:40:07 * rishi is lurking
16:40:19 <dustymabe> #topic Equivalent to system containers from Fedora Atomic in Fedora CoreOS
16:40:21 <geoff-> the CL toolbox is just a script to run a container image with systemd-nspawn, but this issue/38 goes into the creation of an image
16:40:33 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/37
16:41:51 <strigazi> I think everyone is positive on portable systemd services, we need some experience with it and
16:42:10 <strigazi> iterate a bit on the delivery method
16:42:31 <geoff-> +1 for that
16:43:01 <ajeddeloh> its also still marked experimental
16:43:32 <dustymabe> ajeddeloh: we could chat with systemd maintainers to see how mature they think it is
16:43:44 <ajeddeloh> the systemd PS + ostree + Ignition idea sounds real slick
16:43:47 <ajeddeloh> yeah
16:43:57 <strigazi> if we put effort it may graguate from experimental
16:44:22 <strigazi> At least I hope it does :)
16:45:04 <strigazi> Should we try the portable cli in the atomic host?
16:45:18 <dustymabe> strigazi: is that what it's called ?
16:45:21 <dustymabe> portable cli ?
16:45:30 <strigazi> portablectl
16:45:41 <dustymabe> strigazi: is that a separate rpm we need to include ?
16:45:47 <dustymabe> or is it there already ?
16:45:51 <ajeddeloh> systemd portable services (not that I'd played with them much) do look like they need an equivilent of "enablement"
16:46:15 <strigazi> dustymabe: I have to check, I don't know
16:46:16 <kaeso> dustymabe: it's a flag on systemd srpm, I don't know if the binary rpm is split
16:46:17 <ajeddeloh> like it looks like right now you need to manually run `portablectl attach`
16:46:36 <dustymabe> kaeso: +1
16:47:05 <dustymabe> strigazi: if we think this is a possible future for us then we can try to get the appropriate bits into f29AH so you can at least try to prove them out
16:48:21 <kaeso> strigazi: to be clear, those wouldn't help with augmenting the non-containerizable bits (docker, kubelet, etc)
16:49:08 <strigazi> kaeso: we couldn't run kubelet as a systemd portable service?
16:50:02 <kaeso> strigazi: the kubelet --containerized flag is being deprecated as it is broken (from a design PoV)
16:50:25 <dustymabe> kaeso: yes, that is true
16:50:31 <ajeddeloh> systemd PS don't have much "containized" bits to them though
16:50:33 <dustymabe> at least I've been told
16:50:48 <ajeddeloh> its not a ton more than a chroot
16:51:02 * ajeddeloh also doesn't know much about the kubelet
16:51:52 <kaeso> strigazi: but that was just an example. In general there are software that assumes running in the host, and those can't be easily swapped via containerized services
16:52:52 <dustymabe> kaeso: I think where he is coming from is "system containers" so I think "some containerization" probably still satisfies his use case
16:52:59 <ajeddeloh> If systemd PS work well for docker, kubelet, etc I wonder if we could just use that and not ship rpm-ostree (just ostree)
16:53:11 <strigazi> kaeso: ok, do you have any pointers on the deprecation of --containerized ?
16:53:31 <dustymabe> strigazi: jbrooks might have a link handy
16:53:43 <dustymabe> but mostly look at upstream kube and I think you should see discussion about it
16:53:48 <kaeso> strigazi: I don't have them at hand, no
16:54:20 <strigazi> as ajeddeloh mentioned systemdPS are just a chroot plus a few more bits. I'll have a closer look.
16:54:32 <kaeso> ajeddeloh: that's the same split between rkt and torcx, I don't think it can go away
16:54:34 <jbrooks> I don't have a link
16:55:15 <dustymabe> does anyone know if we can deliver a PS payload wvia OCI ?
16:55:15 <jbrooks> They were talking about it in https://github.com/kubernetes/kubernetes/issues/43708
16:55:23 <dustymabe> i.e. an OCI registry ?
16:55:34 <ajeddeloh> kaeso: what do you mean?
16:56:47 <strigazi> jbrooks: also relevant https://github.com/kubernetes/kubernetes/issues/61675
16:57:49 <dustymabe> any takeaways from this discussion we should add to the ticket ?
16:58:07 <strigazi> dustymabe: delivering systemdPS with skopeo or similar would be great. it should be possible.
16:58:12 <ajeddeloh> should we investigate running docker, kubelet, etc from a PS?
16:58:18 <kaeso> ajeddeloh: that if you move a piece of software to its own mountns/chroot, you break all its assumptions about files/binaries/services on the host that it gets via runtime probing
16:58:19 <strigazi> ^^ yes
16:58:42 <ajeddeloh> kaeso: ah, otcha
16:58:49 <ajeddeloh> gotcha*
16:59:17 <dustymabe> https://public.etherpad-mozilla.org/p/20180905-fcos
16:59:57 <kaeso> ajeddeloh: we already did and it works at a first approximation, but it breaks its fundamental assumptions
17:01:09 <kaeso> I think that PS (or podman) would be fine for most of our needs
17:01:17 <kaeso> for the rest, we have rpm-ostree
17:01:45 <dustymabe> kaeso: are you saying we might just be able to use podman itself instead of PS?
17:02:07 <kaeso> and we fix software to be container-friendly, we can move more stuff there
17:02:39 <dustymabe> draft of what I'll put in the ticket: https://public.etherpad-mozilla.org/p/20180905-fcos
17:02:56 <dustymabe> strigazi: are there any pieces of that that you can sign up for to investigate ?
17:03:16 <kaeso> dustymabe: it may, like rkt did. It's mostly a matter of patterns in writing systemd service units running podman
17:03:27 <dustymabe> kaeso: +1
17:03:41 <strigazi> dustymabe: I'm a little lost, should we (or I) put effort on investigating kubelet or docker as  a PS?
17:04:27 <dustymabe> strigazi: i think it would defintely be useful to know if it worked or not
17:04:33 <dustymabe> and if you hit any obvious blockers
17:05:08 <strigazi> ok, I can give it a go.
17:05:22 <kaeso> I can also try to collect all the open tickets on containerized docker/kubelet I remember
17:05:40 <dustymabe> cool
17:05:41 <strigazi> kaeso: thanks, that would be great
17:05:42 <dustymabe> I updated the ticket
17:05:49 <dustymabe> https://github.com/coreos/fedora-coreos-tracker/issues/37#issuecomment-418806643
17:05:54 <dustymabe> next topic
17:06:10 <dustymabe> #topic How to ship cloud specific bits
17:06:14 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/41
17:06:43 <dustymabe> ajeddeloh: i'll let you lead
17:07:40 <ajeddeloh> Do every cloud will need a different image. Ideally we want them to be as similar as possible and to localize any differences
17:07:47 <ajeddeloh> on CL we had the OEM partition
17:08:08 <ajeddeloh> we decided against that for FCOS (the OEM partition was a bad idea in many ways)
17:08:36 <ajeddeloh> That ticket is mostly "how do we localize the changes"
17:09:07 <ajeddeloh> things that will probably need to change between clouds: base ignition configs, files for those configs, and grub configs
17:09:28 <ajeddeloh> thoughts?
17:09:44 <dustymabe> if /boot/ is reliable then it seems reasonable to use that
17:10:17 <bgilbert> it doesn't really need to go in /boot, and it feels like /boot should be minimal
17:10:31 <bgilbert> also there's always the possibility that we'll need to ship something large for a cloud
17:10:48 <dustymabe> yeah, i was thinking these would mostly be non-binary files
17:10:53 <bgilbert> probably
17:10:58 <ajeddeloh> *cough* openvmtools *cough*
17:11:01 <bgilbert> so for me, the rootfs seems the natural place
17:11:05 <bgilbert> well, openvmtools would go in the image anyway
17:11:39 <dustymabe> bgilbert: is there a problem with using the rootfs (i.e. the ignition disks "blow away root" case)
17:11:54 <dustymabe> i guess not because we'd replace it with something and then ignition-files stage could read whatever is there?
17:12:09 <bgilbert> well
17:12:21 <bgilbert> there would be Ignition configs in whatever-the-place-is
17:12:31 <kaeso> ajeddeloh: so your question is whether we ship all of them and enable only one based on OEM id, or if we do some advanced ostree trick so that only the right one is in the deployed rootfs?
17:12:34 <bgilbert> so if we blow away the rootfs, then fail, then reboot, we wouldn't be able to reread the base config
17:12:41 <bgilbert> fair point
17:13:01 <ajeddeloh> kaeso: more the former, not sure I follow the latter
17:13:31 <ajeddeloh> for large things for clouds, can we pull those down from somewhere?
17:14:55 <ajeddeloh> (via Ignition configs :D )
17:15:00 <kaeso> ajeddeloh: I don't have enough knowledge for that, but I was jut wondering if your question was more complex than what I thought
17:15:41 <dustymabe> so basically we should "try rootfs" and fallback to /boot if we hit blockers ?
17:16:20 <kaeso> ajeddeloh: rpm-ostree has a feature to rebuild the initramfs on the client, so I think you'll always need to ship those bits somewhere in the rootfs
17:16:24 <bgilbert> if we want to support the reformatting-root case, rootfs seems like more trouble than it's worth
17:16:37 <ajeddeloh> dustymabe: what about the rootfs blown away case
17:16:49 <dustymabe> right
17:16:59 <ajeddeloh> kaeso: I'm firmly of the opinion we should not support regenerating the initramfs on the client
17:17:12 <bgilbert> ajeddeloh++
17:17:12 <zodbot> bgilbert: Karma for ajeddeloh changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:18:11 <kaeso> ajeddeloh: that's probably not a bad idea, but again I don't have the knowledge about usecase it's solving
17:18:20 <kaeso> *which usecase
17:18:26 <dustymabe> ok cool ajeddeloh mind updating the ticket with this discussion ?
17:18:34 <dustymabe> we can move to open floor for last 10 minutes
17:18:40 <ajeddeloh> but regardless of where we put them, do we have opinions on if we ship everything or just 1 clouds worth (assuming no large things)
17:18:47 <ajeddeloh> dustymabe: sure
17:19:08 <dustymabe> ajeddeloh: I prefer everything.. but if we start shipping large files for each cloud I may change my opinion
17:19:14 <bgilbert> +1
17:19:28 * dustymabe wishes all clouds were created equal :)
17:19:36 <ajeddeloh> ultimately the grub config will need to differ to set oemid
17:19:48 <ajeddeloh> but if we can minimize the differences that'd be ideal
17:20:43 <dustymabe> ajeddeloh: right.. i'd honestly like to be able to "detect" and set the oemid too, but I've been told by multiple people that's a bad idea :)
17:21:00 <dustymabe> can't win them all
17:21:07 <dustymabe> #topic open floor
17:21:16 <dustymabe> anyone with anything for open floor ?
17:21:28 <dustymabe> I wanted to highlight some work by walters recently
17:22:06 <kaeso> dustymabe: I think it was me, autodetection was the bad part in that context. Setting it from externally-provided hint is ok.
17:22:53 <dustymabe> walters: created https://github.com/coreos/coreos-assembler - which is kinda a "toolbox" for creating ostrees and image artifacts for fedora coreos
17:23:00 <dustymabe> it's rough around the edges right now
17:23:14 <ksinny> walters++
17:23:16 <dustymabe> https://github.com/coreos/coreos-assembler/pull/29 is a WIP PR which I've been using
17:24:03 <dustymabe> there is also https://github.com/cgwalters/fedora-coreos-config which is where the manifest files are held
17:24:12 <dustymabe> #link https://github.com/coreos/coreos-assembler
17:24:17 <dustymabe> #link https://github.com/cgwalters/fedora-coreos-config
17:24:23 <walters> all of this is open to feedback/change
17:24:30 <dustymabe> geoff-: I think has already opened a PR
17:24:34 <dustymabe> geoff-++
17:25:44 <dustymabe> any other open floor items ?
17:25:44 <ajeddeloh> walters: would be nice to have a rough: run these steps and build an image guide, even if its crude right now
17:26:13 <ajeddeloh> but super cool nonetheless
17:26:13 <dustymabe> ajeddeloh: if you look at the readme in the PR you'll see a very rough guide
17:26:15 <walters> it's in the README.md in the PR
17:26:19 <ajeddeloh> ah +1
17:26:41 <dustymabe> ajeddeloh: of coures there may be some "i'm running fedora so this just works for me" things
17:26:52 <walters> but there's a whole lot of things more; the biggest I'd say is polishing the workflow for "I have a patch/local git for e.g. systemd, use that for build"
17:26:53 <dustymabe> so if you hit trouble just comment in the PR
17:27:08 <ajeddeloh> +1
17:27:37 <dustymabe> any other topics for open floor ?
17:28:12 <geoff-> I'll try to spend some more time with coreos-assembler this week
17:29:23 <dustymabe> cool
17:29:29 <walters> awesome, FWIW I completely agree with https://github.com/coreos/fedora-coreos-tracker/issues/9#issuecomment-404920705
17:29:50 <dustymabe> ending meeting in 30 seconds
17:30:30 <dustymabe> #endmeeting