16:29:39 <dustymabe> #startmeeting fedora_coreos_meeting
16:29:39 <zodbot> Meeting started Wed Sep 12 16:29:39 2018 UTC.
16:29:39 <zodbot> This meeting is logged and archived in a public location.
16:29:39 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:29:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:39 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:42 <dustymabe> #topic roll call
16:29:45 <slowrie> .hello2
16:29:46 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:29:53 <mnguyen_> .hello mnguyen
16:29:54 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:29:55 <ajeddeloh> .hello2
16:29:57 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:30:27 <yzhang> .hello2
16:30:29 <zodbot> yzhang: yzhang 'Yu Qi Zhang' <jzehrarnyg@gmail.com>
16:30:35 <rfairley> .hello rfairleyredhat
16:30:36 <zodbot> rfairley: rfairleyredhat 'Robert Fairley' <rfairley@redhat.com>
16:31:04 <jlebon> .hello
16:31:04 <zodbot> jlebon: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:31:05 <lorbus_> .hello lorbus
16:31:07 <zodbot> lorbus_: lorbus 'Christian Glombek' <c@petersen-glombek.de>
16:31:09 <jlebon> .hello jlebon
16:31:10 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:35 <sayan> .hello sayanchowdhury
16:31:35 <zodbot> sayan: sayanchowdhury 'Sayan Chowdhury' <sayan.chowdhury2012@gmail.com>
16:31:37 <ksinny> .hello sinnykumari
16:31:38 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com>
16:31:40 <geoff-> geoff-: Geoff Levand <geoff@infradead.org>
16:31:54 * lorbus_ isn`t sure why his nick is temporarily unavailable?!
16:32:23 <ashcrow> .hello smilner
16:32:24 <zodbot> ashcrow: smilner 'None' <smilner@redhat.com>
16:32:25 <sanja> .hello2
16:32:26 <zodbot> sanja: sanja 'Sanja Bonic' <sanja@redhat.com>
16:32:34 <miabbott> .hello miabbott
16:32:35 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:33:08 <dustymabe> #chair slowrie mnguyen_ ajeddeloh yzhang rfairley jlebon lorbus_ sayan ksinny geoff- ashcrow sanja miabbott
16:33:08 <zodbot> Current chairs: ajeddeloh ashcrow dustymabe geoff- jlebon ksinny lorbus_ miabbott mnguyen_ rfairley sanja sayan slowrie yzhang
16:33:22 <dustymabe> do we have walters or bgilbert around today?
16:34:12 <jbrooks> .fas jasonbrooks
16:34:13 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:34:22 <dustymabe> will move into previous meeting action items
16:34:42 * cverna wages
16:34:56 <dustymabe> #topic Action items from last meeting
16:35:02 <dustymabe> #chair cverna jbrooks
16:35:02 <zodbot> Current chairs: ajeddeloh ashcrow cverna dustymabe geoff- jbrooks jlebon ksinny lorbus_ miabbott mnguyen_ rfairley sanja sayan slowrie yzhang
16:35:12 <dustymabe> * ajeddeloh to tag closed FCOS issues into design-doc PRs
16:35:37 <dustymabe> ajeddeloh: that was from two weeks ago.. not sure if we should consider it done or not
16:35:52 <ajeddeloh> I think so, the only two that made sense to include are now merged
16:36:45 <ajeddeloh> It'll kinda become an ongoing thing, but that was specifically talking about the backlog
16:36:47 <dustymabe> cool
16:37:10 <dustymabe> #info ajeddeloh closed FCOS issues into design-doc PRs
16:37:13 <dustymabe> thanks
16:37:39 <dustymabe> #topic created a bunch of new issues in FCOS tracker
16:37:51 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues
16:38:27 <dustymabe> I created a bunch of new issues related to either work we need to do or discussions we need to have in order to hammer down our strategy for many different parts of Fedora CoreOS
16:39:23 <dustymabe> I also added some priority labels `high` `medium` etc.. so we can try to prioritize ones we think are most important and/or ones that we think will take a long time to complete
16:39:54 <dustymabe> any questions about that before we move into meeting tickets ?
16:40:24 <geoff-> looks good
16:40:46 <ajeddeloh> lgtm
16:41:07 <bhavin192> .hello2
16:41:08 <zodbot> bhavin192: bhavin192 'Bhavin Gandhi' <bhavin7392@gmail.com>
16:41:24 <dustymabe> k. some background, we have a lot of meeting tickets today so we may not get to them all
16:41:32 <dustymabe> https://github.com/coreos/fedora-coreos-tracker/issues?q=is%3Aissue+is%3Aopen+label%3Ameeting
16:41:39 <dustymabe> #chair bhavin192
16:41:39 <zodbot> Current chairs: ajeddeloh ashcrow bhavin192 cverna dustymabe geoff- jbrooks jlebon ksinny lorbus_ miabbott mnguyen_ rfairley sanja sayan slowrie yzhang
16:41:49 <dustymabe> #topic ostree delivery model
16:41:59 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/23
16:43:06 <dustymabe> There is some discussion in the ticket about this, but basically we have a choice on how we choose to deliver our OSTree payloads to clients
16:44:00 <dustymabe> one is by using the OSTree repository (like Git repo) model using an HTTPS server (this is what Atomic Host, Silverblue and Fedora IoT use today)
16:44:14 <ajeddeloh> I'm in favor of the vanilla https payloads. KISS. https servers are pretty well figured out by now.
16:44:16 <dustymabe> the other is a new delivery model using RPMs
16:44:45 <dustymabe> which would allow us to leverage our existing Yum repo mirror network, but would require some integration with build tools
16:44:58 <ajeddeloh> using rpm's feels like we're shoehorning it into that because we have them, not because they're the best choice
16:45:14 <ajeddeloh> https servers can also be done with s3, gce cloud storage, etc
16:45:48 <dustymabe> I don't disagree, thanks for the input ajeddeloh
16:46:00 <dustymabe> anyone else have any thoughts on this topic ?
16:46:07 <ashcrow> ajeddeloh: One thing to think about though is that it's not easy to teach different delivery systems about ostree, but many already understand rpms. But it I don't disagree really.
16:46:34 <ajeddeloh> From a different perspective, if I saw an RPM I'd wonder if it's just the fs tree that's being distributed or if there's other "rpm magic" going on
16:46:39 <jbrooks> It's shoehorning it in not just because we have them, but because of what others have -- like the mirror infra, or infra w/in orgs to deal w/ rpms
16:46:53 <jbrooks> With that said, I'm fine w/ the current status quo
16:47:03 <ajeddeloh> ashcrow: what do you mean by "different delivery systems"
16:47:38 <miabbott> if we can reliably mirror the ostree content, i'm in favor of the https/KISS approach.  for example, even though we are fronting the ostree bits for AH/Silverblue via CloudFront, it still seems slow
16:47:44 <jlebon> the alternatives mentioned in the ticket are: ostree, rojig, container; only that last one is not httpd friendly
16:47:48 <dustymabe> one pain point I should also mention is that when someone package layers something now there can be a mismatch between what is in the yum repo and what is in the base tree on the system that causes some conflicts, I think colin's idea was that rojig would solve this somehow, but not sure on that
16:48:20 <ashcrow> ajeddeloh: similar to what jbrooks noted. As an example, if someone wants to mirror ostree content that isn't easy to do with many open or closed products. However, many udnerstand how to mirror rpm content. True, rsync is always an option, but usually people are looking for something smarter. It's not a requirement though.
16:48:45 <ashcrow> walters: ^^
16:49:14 <ajeddeloh> Is there a reason we can't do both?
16:49:49 <dustymabe> to know how big of a problem this is (for FCOS) we should ask ourselves how often we think people are going to try to store our ostree content locally and redistribute vs just pulling from our mirrors
16:49:49 <ajeddeloh> i.e. start with the https KISS method and later add support for rpm if we determine we want it
16:50:05 <ashcrow> Nothing that is a show stopper. The normal development/maint overhead.
16:50:11 <ashcrow> I'm cool with that
16:50:30 <dustymabe> ajeddeloh: +1 we can always pivot later if we want to change our strategy
16:50:32 <ashcrow> Start with KISS, iterate if/as needed
16:50:47 <ajeddeloh> That'd be good because we can keep the common case simple but still support the other cases if we need to
16:51:00 <dustymabe> I should also add that we should pull in Iot/Silverblue because it would be great if we did the same thing for all distributed OSTree content
16:51:25 <ashcrow> +1
16:52:50 <dustymabe> #proposed The general sentiment is that we should target plain OSTree repo + enhanced mirroring support (not implemented today) and then augment with rojig in the future if needed. We will try to get buy in from IoT/Silverblue teams so that we can have a common strategy for all of Fedora delivered OSTree content.
16:53:13 <jlebon> though note transitioning later on would be much more painful than starting in rojig mode right off the bat if that's where we want to go
16:53:49 <dustymabe> jlebon: I agree. but I'm not sure what the benefits are exactly, other than taking advantage of the mirror network
16:54:07 <dustymabe> do you have any off the top of your head?
16:54:22 <bgilbert> .hello2
16:54:23 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:54:30 <dustymabe> #chair bgilbert
16:54:30 <zodbot> Current chairs: ajeddeloh ashcrow bgilbert bhavin192 cverna dustymabe geoff- jbrooks jlebon ksinny lorbus_ miabbott mnguyen_ rfairley sanja sayan slowrie yzhang
16:54:58 <miabbott> if we chose rojig, wouldn't we also have to change the retention of the RPMs needed for each commit?  i.e. to preserve commit history, we would need all those rpm versions
16:54:58 <jlebon> yeah, mirroring is definitely the big one. as miabbott mentioned above, ostree updates are still pretty slow even today
16:55:36 <ajeddeloh> why can we not use <insert cloud> for mirroring. It's just a bunch of files served over https?
16:55:45 <dustymabe> yep. I think we've been putting off "mirroring efforts" because we weren't sure between that or rojig
16:55:59 <dustymabe> ajeddeloh: that's an option, yes
16:56:10 <miabbott> ajeddeloh: it's possible.  we use cloudfront now for atomic host/workstation
16:56:11 <dustymabe> and amazon has offered us credits for stuff like this
16:56:23 <dustymabe> so we just need to implement it
16:56:33 <ajeddeloh> We've been using google cloud storage for CL and it's worked well
16:56:38 <dustymabe> and I think once we make this decision work can start on that front and we can prioritize it with fedora infra
16:56:43 <miabbott> s/workstation/silverblue/  :P
16:56:56 <jlebon> the main idea with rojig is that is allows users to reuse that whole ecosystem of tools that know how to handle RPMs
16:57:09 <dustymabe> anyone want to ack/nack the #proposed statement ?
16:57:48 <ajeddeloh> ack
16:58:06 <miabbott> ack
16:58:10 <ksinny> +1
16:58:14 <lorbus_> ack
16:58:14 <mnguyen_> ack
16:58:17 <ashcrow> ack
16:58:20 <geoff-> ack
16:58:26 <dustymabe> \o/
16:58:43 <dustymabe> #agreed The general sentiment is that we should target plain OSTree repo + enhanced mirroring support (not implemented today) and then augment with rojig in the future if needed. We will try to get buy in from IoT/Silverblue teams so that we can have a common strategy for all of Fedora delivered OSTree content.
16:58:56 <bgilbert> ack
16:58:58 <dustymabe> moving on to next topic
16:59:10 <dustymabe> #topic Allow binaries written in Python via a "platform python" style approach
16:59:17 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/32
16:59:41 <dustymabe> I placed this one as high priority because other discussions hinge on it (like the firewall discussion)
17:00:09 <dustymabe> I laid out 4 goals related to python in FEdora CoreOS in the ticket
17:00:36 <dustymabe> 1. prevent users from running scripts directly on the host
17:00:38 <dustymabe> 2. prevent shipping/maintaining python
17:00:40 <dustymabe> 3. prevent issues where user's python script needs library X that isn't installed
17:00:42 <dustymabe> 4. prevent security issues in python requiring a respin
17:01:03 <dustymabe> are there any others that I missed?
17:01:37 <ajeddeloh> There's a (tiny) 5th: less space used on disk + less data transmitted for updates
17:01:50 <dustymabe> +1
17:02:03 <ajeddeloh> plus an even smaller 6th: better perception we're minimal
17:02:19 <dustymabe> :)
17:02:21 <ajeddeloh> (which is something I've seen as a selling point for CL)
17:02:43 <ashcrow> +1
17:02:44 <dustymabe> the two options being discussed are
17:02:51 <dustymabe> - no python at all
17:03:13 <dustymabe> - a shipped python that is not accessible to users (an attempted goal)
17:03:40 <ajeddeloh> If we have nothing that _hard_ requires python (i.e. firewalld) then imo its worth breaking up the packages that think they require it but dont
17:03:44 <dustymabe> out of the ~6 goals we laid out above, which are the most important and which ones are not addressed by the 2nd option ?
17:03:53 <ashcrow> I think "no python at all" is the ideal. But I think that is a large mountain to climb given a specific time frame.
17:04:02 <ashcrow> Note: I'm a big fan of python as a developer :-)
17:04:12 <ajeddeloh> haha ashcrow same :)
17:04:27 <cverna> +1 to no python at all
17:04:28 <ajeddeloh> 2,4,5,6
17:04:58 <ajeddeloh> can we run firewalld in a container or systemd PS?
17:05:15 <dustymabe> ok let's discuss 2,4,5,6 real quick
17:05:31 <ajeddeloh> I feel like this is becoming a catchphrase but "this seems like a job for systemd portable services"
17:05:43 <dustymabe> 2. i'm not too worried about because it's already being maintained in fedora
17:05:50 <dustymabe> does anyone agree/disagree with that ?
17:06:06 <jlebon> yeah, agreed
17:06:08 <ashcrow> I don't disagree.
17:06:13 <miabbott> agreed
17:06:41 <bgilbert> I do, to some extent
17:06:45 <rfairley> +1
17:06:53 <bgilbert> do disagree, sorry
17:07:01 <dustymabe> bgilbert: :) I was going t oask
17:07:03 <bgilbert> we still have to be concerned about the package set
17:07:15 <dustymabe> bgilbert: i.e. security issues ?
17:07:43 <bgilbert> most of the work is done elsewhere, yes, but there's always a development cost to carrying things
17:08:17 <dustymabe> bgilbert: is that  concern that the rest of fedora would drop it and we'd be maintaining it ourselves?
17:08:43 <ashcrow> I think it's more so we have to keep an eye on new deps and what changes. Sizes may go up and down into what is pulled in or dropped by upstream.
17:08:50 <ajeddeloh> bgilbert: you're talking about the cost of splitting packages?
17:09:18 <bgilbert> not really.  but for any package we carry, there's always the possibility of having to dig into it to root-cause an issue.
17:09:28 * ashcrow nods
17:09:33 <bgilbert> because we tickle some case that upstream Fedora doesn't
17:09:48 <bgilbert> so, the fewer large things we carry, the better. /shrug
17:10:00 <dustymabe> +1 I agree with that
17:10:22 <dustymabe> so caveat there that just because the work is being done somewhere else, doesn't mean it won't ever cause us pain
17:10:56 <dustymabe> i.e. firewalld (example) doesn't work in FCOS because of the way we have the system set up and it's a bug in python-xyz package that doesn't show up on vanilla Fedora
17:11:11 <dustymabe> bgilbert: ^^ accurate ?
17:11:45 <bgilbert> yep
17:11:54 <dustymabe> ok so I think we are done with 2.
17:12:02 <walters> .hello2
17:12:03 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
17:12:06 <dustymabe> 4. prevent security issues in python requiring a respin
17:12:15 <dustymabe> this is a legitimate concern
17:12:57 <dustymabe> hopefully our tooling/processes allow us to respin things easily, but there is always costs associated and disruptions as a result
17:13:23 <dustymabe> if we ship python I don't think we'll get away from #4, but do wonder how often python itself has security issues
17:13:27 <dustymabe> anyone know ^^
17:13:32 <jlebon> just as a rough measure, I was scanning https://bodhi.fedoraproject.org/updates/?packages=python3 to see how many updates are marked as security
17:13:46 <jlebon> there was one 5 months ago, another 9 months ago
17:14:32 <dustymabe> jlebon: and depending on the severity of those two, maybe wouldn't have even caused us to respin
17:14:34 <jlebon> the next one after that was in f23
17:14:57 <jlebon> dustymabe: true
17:15:09 <ajeddeloh> well it's not just python we have to worry about. If firewalld depends on X Y and Z python packages we need to worry about them as well
17:15:09 <dustymabe> that's probably not the full picture, though. as related libraries could have had security issues and needed updating
17:15:17 <dustymabe> so we'd probably need to do a little more investigation
17:15:23 <jlebon> though OTOH, this is *only* the main python3 package. including python3 means it's easier to include other packages which in turn can have sec erratas
17:15:39 <misc> unlike the kernel, who do not have CVE per dozen every month :p
17:16:14 <dustymabe> i think if we do go the "system python" route we still want to be very careful about anything that we include that is python based, we only include what we really really want OR need
17:17:14 <dustymabe> @bgilbert should we try to get more information about security issues in python3 and related libraries to get more info for this point?
17:17:40 <bgilbert> wouldn't hurt.
17:17:51 <dustymabe> ok I can try to do that or ask someone else to
17:17:58 <bgilbert> there are certainly other packages that we have to respin for much more often
17:18:20 <dustymabe> #action dustymabe to gather more information on frequency of python (and related library) security issues in Fedora
17:18:38 <dustymabe> ok points 5/6 (which I'll group together as smaller/minimal footprint)
17:18:49 <dustymabe> shipping python will make us less minimal, yes
17:19:34 <dustymabe> alternatively though, rewriting the few things that we do need in python in Go (large binary) might not actually save that much disk space ??
17:19:40 <ashcrow> Somewhat OT: (I don't think it should change the discussion) IF using ansible to do things is planned you will have to have quite a few libraries added in to system python and have instructions for using ansible with the distribution.
17:19:51 <ashcrow> dustymabe: true. Until it's looked at there is no way to really know.
17:20:26 <ajeddeloh> afaik we're not supporting ansible, right?
17:20:35 <dustymabe> ashcrow: maybe naive of me but I don't think we should consider ansible at this time (or maybe force that into a containerized route)
17:20:40 <ajeddeloh> (at least on host)
17:20:40 <ashcrow> +1
17:20:57 <dustymabe> ajeddeloh: agreed, mostly
17:21:33 <misc> well, not only that, I wonder if there is some selinux tooling in python too :/
17:21:34 <ashcrow> ajeddeloh: FWIW ansible needs the libs on the host when run from an external system. But +1 that we punt on that.
17:21:40 <dustymabe> so after discussion points 1-6 - do we have a feel for the ones that are most important to us?
17:21:45 <dustymabe> misc: there definitely is
17:22:16 <ashcrow> For me 1 is the most important :-) But I'll defer to others.
17:22:33 <dustymabe> the way I see it is 1,3 are most important to us
17:23:02 <ajeddeloh> iirc (and very much may be wrong on this) the selinux python tools are higher level and you can get by with the ones not in python, but it's been a while since I looked at that
17:23:02 <dustymabe> 2 I think we can rely on the rest of Fedora for that, but note bgilbert's caveat that says there won't be zero pain
17:23:42 <dustymabe> 4 we have an action item to gather more information, but it doesn't look like there have been my primary python package security issues in the past year
17:24:00 <misc> ajeddeloh: well, the experience of debugging selinux issue would be likely more annoying and push users to disable it. That's kinda not something we want
17:24:08 <walters> i feel like at some point ansible probably will find it beneficial in general to support a containerized mode
17:24:24 <dustymabe> 5/6 - minimal is good, but people who choose not to use Fedora CoreOS specifically because we chose to ship system python probably want more control over their package set anyway and will build their own
17:25:01 <ajeddeloh> dustymabe: I don't think we're going to get a lot of "build it on your own" people
17:25:18 <ajeddeloh> the whole point of a *coreos is that you don't need to do anything for updates
17:25:28 <dustymabe> ajeddeloh: so if they choose "not FCOS" because of this, you think they'd choose another offering ?
17:26:01 <ajeddeloh> yeah, you'd have to run your own build infra for that
17:26:12 <dustymabe> right
17:26:33 <dustymabe> with coreos-assembler that will be easier :), but I agree, it would be work they'd have to do
17:26:46 <dustymabe> ok I'll try to take all of these points and put them in the ticket
17:27:02 <ajeddeloh> thats not something I expect many people to do, and they'd either have to automate looking for our updates or manually kick off builds
17:27:07 <dustymabe> after I gather more information maybe we can get together next week and agree on a strategy
17:27:31 <dustymabe> ok, look at the time
17:27:31 <bgilbert> probably worth noting that the inaccessible-Python approach means that we don't need to make a permanent decision up front.
17:27:52 <bgilbert> we can try it, explicitly document that users shouldn't use it, and revisit down the road.
17:27:58 <ajeddeloh> bgilbert: I disagree
17:28:00 <dustymabe> yes, the decision here is between "no python" and "inaccessible-Python"
17:28:13 <ajeddeloh> that python is in service of _something_ that is user facing
17:28:25 <bgilbert> ajeddeloh: concerned about compat constraints?
17:28:26 <ajeddeloh> presumably removing it means removing the user facing thing
17:28:34 <ajeddeloh> yeah
17:28:41 <bgilbert> fair
17:29:04 <dustymabe> ok i'll open up for open floor now
17:29:06 * bgilbert creates github/coreos/coreos-firewalld
17:29:12 <dustymabe> #topic open floor
17:29:32 <dustymabe> this is a topic i wanted to get to because I think it will be short but I'll just ask people to weigh in on the ticket so we can close it out:
17:29:49 <dustymabe> Default filesystem choices for FCOS https://github.com/coreos/fedora-coreos-tracker/issues/33
17:30:07 <dustymabe> basically it seems like we are leaning toward XFS ^^ please weigh in in the ticket with +1 or -1
17:30:16 <dustymabe> anyone else with anything for open floor ?
17:30:27 <jlebon> i'm not sure if this was brought up in a previous meeting, though https://github.com/coreos/fedora-coreos-config is worth a look if you haven't already
17:30:40 <jlebon> along with https://github.com/coreos/coreos-assembler
17:30:50 <dustymabe> jlebon: +1
17:31:01 <jlebon> i guess one could call it "proto-proto-FCOS"
17:31:03 <ajeddeloh> re fs choice: can I +0? I don't care much tbh
17:31:20 <dustymabe> there was also a long/detailed question written on the discourse last night that I tried to answer with a lot of good information
17:31:27 <dustymabe> ajeddeloh: sure that's still good information to have
17:31:42 <dustymabe> re: discourse see https://discussion.fedoraproject.org/t/getting-involved/386/2
17:32:05 <dustymabe> anyone with anything else for open floor?
17:32:50 <ashcrow> Nothing from me
17:33:37 <dustymabe> ok closing out
17:33:39 <dustymabe> #endmeeting