server_working_group_weekly_meeting_(2016-11-01)
LOGS
20:00:07 <sgallagh> #startmeeting Server Working Group Weekly Meeting (2016-11-01)
20:00:07 <zodbot> Meeting started Tue Nov  1 20:00:07 2016 UTC.  The chair is sgallagh. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:07 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:07 <zodbot> The meeting name has been set to 'server_working_group_weekly_meeting_(2016-11-01)'
20:00:07 <sgallagh> #chair nirik sgallagh mhayden dperpeet smooge jds2001 vvaldez adamw mjwolf
20:00:07 <zodbot> Current chairs: adamw dperpeet jds2001 mhayden mjwolf nirik sgallagh smooge vvaldez
20:00:07 <sgallagh> #topic roll call
20:00:07 <sgallagh> .hello sgallagh
20:00:09 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
20:00:14 <vvaldez> .hello vvaldez
20:00:15 <zodbot> vvaldez: vvaldez 'Vinny Valdez' <vvaldez@redhat.com>
20:00:16 <adamw> .hello adamwill
20:00:16 <smooge> here
20:00:20 <mjwolf> .hello mjwolf
20:00:21 <zodbot> adamw: adamwill 'Adam Williamson' <awilliam@redhat.com>
20:00:24 <zodbot> mjwolf: mjwolf 'Michael Wolf' <mjwolf@us.ibm.com>
20:01:47 <nirik> morning
20:01:48 <mhayden> .hello mhayden
20:01:52 <zodbot> mhayden: mhayden 'Major Hayden' <major@mhtx.net>
20:01:52 <nirik> .hello kevin
20:01:55 <zodbot> nirik: kevin 'Kevin Fenzi' <kevin@scrye.com>
20:02:02 <jds2001> .hello jstanley
20:02:03 <zodbot> jds2001: jstanley 'Jon Stanley' <jonstanley@gmail.com>
20:02:24 <sgallagh> OK, I think that's everyone I expected.
20:02:29 <sgallagh> (I think dperpeet is on PTO)
20:02:37 <sgallagh> #topic Agenda
20:02:54 <sgallagh> #info Agenda Item: Node.js for alternative architectures
20:03:08 <sgallagh> #info Agenda Item: Common requirements for server roles
20:03:29 <sgallagh> Are there any other urgent or important topics for this week? (I expect these to fill the hour)
20:04:02 <smooge> nothing from me
20:04:36 <sgallagh> OK, save it for Open Floor or take it to the list if anyone comes up with something else.
20:04:36 <mhayden> nada
20:04:49 <sgallagh> #topic Node.js for alternative architectures
20:05:37 <sgallagh> Dan Horak asked us to figure out how to deal with building Node.js on the new arches that were imported into prod Koji
20:05:54 <sgallagh> (I assume as a precedent for other similar packages)
20:05:57 <smooge> does upstream support them? because I thought support was pretty limited
20:06:12 <mjwolf> upstream support should be there
20:06:52 <sgallagh> smooge: "upstream" supports it, though I think that really means "IBM contributes patches"
20:07:28 <sgallagh> (not to denigrate that, just that I suspect it's one of those cases where support will last as long as the vendor of the arch cares about it)
20:07:28 <jds2001> whoever they come from is fine with me :)
20:08:05 <jds2001> i thought that the base problem was the ExlusiveArch in node packages uses a macro that we do not control in RHEL.
20:08:15 <sgallagh> The specific Node.js situation is somewhat complicated because of how it's built.
20:08:18 <jds2001> so this is more an EPEL problem than a Fedora proper problem.
20:08:21 <sgallagh> jds2001: That's a problem for EPEL
20:08:42 <sgallagh> In Fedora, we have more leeway, but we still have a couple choices to make.
20:09:27 <mhayden> so are there different upstream sources depending on the arch?
20:09:30 <sgallagh> Because of the way that Node.js packages are built, we'd have to arrange this for a mass-rebuild
20:09:38 <sgallagh> mhayden: No, it's one tarball
20:10:03 <langdon> .hello langdon
20:10:04 <mhayden> okay
20:10:04 <zodbot> langdon: langdon 'Langdon White' <langdon@fishjump.com>
20:10:15 <sgallagh> Because as of today, all Node.js packages are built with an ExclusiveArch: %{nodejs_arches} macro
20:10:20 * smooge is in listen mode so is following along
20:11:21 <sgallagh> The trivial answer is to just edit the %{nodejs_arches} macro to include s390x (I'm unclear about s390?) and force a rebuild of all Node.js packages
20:11:48 <mhayden> that sounds reasonable -- is there a catch?
20:11:57 * mhayden is a nodejs newbie for sure
20:12:31 <sgallagh> mhayden: There's been some question as to whether we should try to switch to ExcludeArch rather than ExclusiveArch, but I'll admit that I don't really see the advantage.
20:12:46 <sharkcz> sgallagh: IIRC pbrobinson's idea was to drop the Exlusivearch completely, rel-eng can do the mass editing
20:12:50 <sgallagh> pbrobinson: You were the one that suggested that. Do you happen to be around?
20:13:15 <smooge> i think it is still night for him
20:13:32 <smooge> though he may be in EU
20:13:33 <mhayden> if we drop ExclusiveArch, it will build on all available architectures, correct?
20:13:35 <langdon> IIRC the problem was when adding a new arch you had to change a lot of stuff instead of just getting builds happening automatically
20:13:47 <sgallagh> sharkcz: The only problem I see there is that we get into situations where if the node.js interpreter isn't present, builds will fail if they end up on an incompatible builder.
20:14:14 <sgallagh> Which is a long-standing issue with noarch packages in Koji
20:14:18 <jds2001> sgallagh: are there still arches where it wont work?
20:14:23 <sharkcz> sgallagh: it shouldn't be the case if we don't add mips(?) and riscv
20:14:23 <jds2001> i.e. the interpreter?
20:14:24 <smooge> so thi si more about dealing with a long standing issue with koji
20:14:42 <sharkcz> but dunno about i686 ...
20:15:11 <sgallagh> We support i686 downstream, at least.
20:15:26 <sgallagh> But it will still introduce issues whenever a new arch appears.
20:15:48 <sharkcz> ppc64/ppc64le support is there as is s390x, but you are right with any new arch
20:16:23 <smooge> arm/aarch?
20:16:39 <mhayden> why not edit the macro and specify the architectures where we are 100% sure if will compile and work?
20:16:42 <sharkcz> I think the main use case for dropping ExlcusiveArch is to support epel on all arches without depending on redhat-rpm-confiog update from rhel
20:16:57 <sgallagh> mhayden: Right, that's the trivial solution I mentioned above
20:17:03 <sharkcz> but for me (aka fedora/s390x) any variant works :-)
20:17:17 <sgallagh> sharkcz: Yeah, that's perhaps more relevant for EPEL
20:17:38 <sgallagh> /me isn't sure why redhat-rpm-config provides that macro, save as an accidental merge from Fedora
20:17:44 <sgallagh> Since RHEL doesn't ship Node.js
20:18:06 <jds2001> sgallagh: i was wondering the same. There are some layered products that do.
20:18:20 <sharkcz> yep, looks like fedora heritage
20:18:22 <jds2001> sgallagh: but they can take care of that themselves i think....
20:18:34 <langdon> rhel ships nodejs.. but as an scl
20:18:50 <jds2001> langdon: openshift ships a non-scl version
20:19:06 <jds2001> but that's neither here nor there i think......
20:19:08 <langdon> jds2001, do they now? they used to use all the scl versions
20:19:44 <sgallagh> Proposal: In Fedora, let's just stick with whitelisting and add s390x to the list
20:19:58 <sgallagh> sharkcz: What about s390 without x?
20:19:58 <nirik> sounds fine to me. +1
20:19:58 <jds2001> langdon: im reasonably sure i found it in the package search i did this morning while i was scratching my head about this.......
20:20:19 <sharkcz> sgallagh: we have killed 32-bit s390 before f24 GA
20:20:27 <sgallagh> OK
20:20:33 <sgallagh> /me didn't see that happen
20:20:51 <sharkcz> sgallagh: for epel I think it might be possible to override the macros via /etc/rpm/...
20:21:06 <sgallagh> (As for EPEL, let's track that as a bug to be opened with RHEL. Maybe we can ask them to add a nested macro expansion for us.)
20:21:24 <sgallagh> sharkcz: I tried that. Panu tells me it's not possible
20:21:31 <sharkcz> sgallagh: ah, ok
20:22:14 <sgallagh> Anyone want to strongly disagree with the above?
20:22:58 <mhayden> sgallagh: i'm +1 on thw whitelisting
20:23:26 <jds2001> +1 to whitelisting s390x and ppc64/ppc64le
20:23:27 <smooge> +1 with whitelisting
20:24:01 <sgallagh> jds2001: ppc is already included
20:24:15 <jds2001> sgallagh: oh, then ignore that part :D
20:24:27 <sharkcz> but is missing for epel :-)
20:24:36 <sgallagh> nirik: We have a mass rebuild scheduled for F26, right?
20:24:36 <sharkcz> afaik
20:24:49 <nirik> yes.
20:24:57 <sgallagh> sharkcz: I'm not recommending that we backport this change to F24 or F25 at this point.
20:25:08 <sharkcz> sgallagh: yep, f26+ is ok
20:25:20 <jds2001> sgallagh: yeah, that would be too invasive at this point.
20:25:23 <sgallagh> Or at least, I'm not willing to expend the effort on the side-tag dance
20:25:53 <sgallagh> Of course, if someone wants this in EPEL, they're going to be stuck doing that dance over there.
20:27:39 <sgallagh> #agreed We will add s390x to the %{nodejs_arches} macro in time for the F26 mass rebuild. We are abstaining from a decision on EPEL.
20:28:01 <sgallagh> #topic Common requirements for server roles
20:28:13 <sharkcz> sgallagh: is any bootstrap procedure needed?
20:28:23 <sgallagh> OK, this is a complicated topic. I've got some starter questions.
20:28:57 <sgallagh> sharkcz: No, the Node.js interpreter is written in C/C++
20:29:04 <sgallagh> It doesn't need to build itself or anything like that
20:29:26 <sharkcz> sgallagh: ok, so no circular deps?
20:30:06 <vvaldez> sgallagh: I just sent my draft to the list for my 2 roles (I didn't include the domain controller role we are co-owning yet): https://docs.google.com/document/d/1jyOEarqPw51rHETC7N-bEtILoLiQ2yMBEKbTcVbFQVw/edit#
20:30:22 <sgallagh> vvaldez: Yes, I saw it but didn't have a chance to review it yet.
20:30:34 <vvaldez> I followed your original example, I don't have user stories in there which would probably be helpful
20:30:50 <sgallagh> Let's start with a high-level architecture question though:
20:32:12 <sgallagh> Where do we draw the configuration line? Using DNS as an example: Does the role deployment have to deploy the complete DNS configuration, or does it deploy and start the services and then provide ongoing UI for updating the configuration?
20:33:03 <jds2001> sgallagh: interesting question - are we going to be in the market of providing control panel type functionality here?
20:33:09 <sgallagh> Put another way, when we do the deployment, are we being fully declarative like an Ansible playbook or Puppet recipe, or are we running an installer that will produce a role that is reconfigurable later?
20:33:41 <vvaldez> sgallagh: well, specific to deployment role, I was assuming we would use either cobbler or Foreman, which automate the DNS setup, should we consider a fully independant setup that doesn't use cobbler or Foreman and manually configures each service (DNS/DHCP/TFTP/HTTP/etc)?
20:33:41 <sgallagh> (This gets fuzzy when you start talking about more complicated services like FreeIPA; in that case I think it's clear that we wouldn't want to try to set up all the users in the deployment!)
20:33:43 <jds2001> i.e. if I think of the Microsoft DNS Server, they do provide UI without the need to edit any files (heck, im not even sure if you *can* edit files, but let's assume you can)
20:34:23 <jds2001> is that something that we want to do???
20:34:24 <sgallagh> vvaldez: I think that's *too* specific. I used DNS as an example in the "master DNS for your company" context. Not really the "subcomponent of another role" context.
20:34:33 <vvaldez> that's what I figured
20:35:16 <vvaldez> for a first pass I think it's Ok to implement something very constrained that will help the inexperienced admin. More advanced admins might not need to use the role, but I don't think we are targetting them with these roles, right?
20:35:30 <vvaldez> yet
20:35:33 <sgallagh> /me nods
20:35:34 <jds2001> vvaldez: explictly not targeting them
20:35:40 <langdon> personally, i lean toward "fully configured" ... if we think in terms of "modules" we want the "thing" to offer some sort of api.. and that is hard with different configs per user.. however, there may be a number of "install profiles" which provide different apis.. (where api loosely means "provides some service(s)")
20:35:52 <jds2001> vvaldez: at the same time, not getting in their way
20:36:55 <vvaldez> ok, so with that in mind going back to DNS, if we constrain it to say, 1 domain only and 1 subnet, we should be able to gather variables to implement that pretty consistently. Future iterations can expand on that
20:37:17 <sgallagh> I think we need to define a line between "service-level configuration" and runtime data.
20:38:15 <sgallagh> Maybe in the DNS case, everything is "service-level configuration" and we just redeploy for changes. Ditto NFS/Samba shares.
20:38:27 <sgallagh> But for a database or FreeIPA, I think the distinction becomes important
20:39:54 <smooge> From experience DNS goes in two sets.. very small simple, complicated enterprise setup
20:40:09 <sgallagh> Perhaps we draw the line at "if the service itself provides an interface to manage it ongoing, we don't try to take over that"?
20:40:45 <smooge> so is this a question about cockpit development or our role as server committee group?
20:43:14 * smooge wonders if he got netsplit
20:43:16 <vvaldez> it's gone now, but I did some work with Dell about 8 years ago to do something very similar: provide a GUI front-end to enable clustered services like NFS/Samba with very constrained parameters. I found the source rpm via wayback machine: http://web.archive.org/web/20101102211213/http://linux.dell.com/files/ha_linux/
20:43:28 * vvaldez pokes smooge
20:43:36 <sgallagh> smooge: role development in specific (which will hopefully involve Cockpit)
20:44:48 <sgallagh> smooge: It's about where we draw the line when building roles. Hypothetically, if BIND 9 had a RESTful interface for making DNS modifications, would it change where we drew that line?
20:45:11 <sgallagh> Would we still do a full redeploy of flat files if a mechanism for live modification was present?
20:46:24 <smooge> well even if it had the interface it would need a tool to work the interface
20:46:26 <langdon> well.. adding to that.. is this a "golden image world"? in other words, does a server get "updated" or just "redeployed"
20:48:02 <sgallagh> langdon: Our stance on Server thus far has been that its primary niche is dealing with long-running services that aren't likely to become microservices this decade.
20:48:20 <sgallagh> So your storage servers, your domain controllers...
20:48:26 <smooge> so to me a role can be 'committed' for what we have tools in hand to deploy the role with
20:48:42 <sgallagh> smooge: Sorry, could you rephrase that?
20:49:08 <dperpeet> sorry, I forgot that our time change two days ago didn't change UTC
20:49:58 <langdon> i dont mean microservices.. i mean the scenario where you have ansible playbooks (or whatnot) which create your golden images and then you deploy x number of them in to your datacenter.. but, no server is managed directly.. only the goldens
20:49:59 <sgallagh> dperpeet: We actually track US time, simply because everyone on the WG but you is in the US.
20:51:13 <smooge> sgallagh, I am trying to answer : <sgallagh> Put another way, when we do the deployment, are we being fully declarative like an Ansible playbook or Puppet recipe, or are we running an installer that will produce a role that is reconfigurable later?
20:51:46 <sgallagh> langdon: I'm not sure how that differs here? I mean, let's again take the FreeIPA case. There's no way you can realistically redeploy its database. You need to carry that forward.
20:52:13 <langdon> sgallagh, the data itself, yes, the database-software? sure..
20:52:26 <smooge> so what I was trying to say was that I believe that we are relying on other tools that we are just an installer for
20:52:30 <sgallagh> langdon: This particular app is very problematic in that regard.
20:52:37 <langdon> ha
20:52:41 <sgallagh> Last I checked, there still wasn't a proper export/import process.
20:52:52 <langdon> so.. a fine answer is rat hole.. worry about it later..
20:52:58 <sgallagh> You would just create a new replica, let it sync, then decommission the original.
20:53:20 <sgallagh> And you wouldn't attempt to provision new users with something like ansibl,e
20:53:29 <langdon> last i knew, server-wg was more about "pet systems" than "cattle systems" be they phys, vm, or containers..  so .. maybe table it for another time
20:53:31 <sgallagh> You would use the ipa tools
20:54:02 <sgallagh> langdon: I'm trying to migrate more to a beehive analogy, because I think it works better and describes modern datacenters more realistically.
20:54:14 <vvaldez> right smooge, I see a role as simply an automed series of commands that could be done in the shell manually, but we've collected them in a convenient playbook
20:54:25 <sgallagh> You've got your worker bees as you new, microservice systems. If one dies, the hive continues to survive.
20:54:40 <sgallagh> Then you've got your Queen Bee. It's death is catastrophic.
20:54:46 <sgallagh> s/It's/Its/
20:55:18 <sgallagh> They have to work together, but some systems (queen bees) require more care and protection.
20:55:57 <sgallagh> vvaldez: Well, it gets murkier because I want to also lob the grenade of "should all roles be required to run inside a container?"
20:56:06 <sgallagh> (Fire in the hole!)
20:56:16 <vvaldez> sgallagh: so our users are flowers and we're trying to pollinate them?
20:56:38 * vvaldez wants some honey now
20:56:48 * jds2001 tosses sgallagh's grenade back at him :D
20:56:51 <sgallagh> vvaldez: http://img.memecdn.com/speechless_o_2899449.jpg
20:57:02 <vvaldez> :)
20:57:20 <vvaldez> sgallagh: ideally it would be nice if the cockpit UI had a "containerize?" toggle button
20:57:29 <sgallagh> vvaldez: No, I disagree
20:57:33 <langdon> so.. i would make a bunch of comments about that but 1) i think it is a rathole 2) need to run off to manage a kid crisis ...
20:57:43 <sgallagh> That's giving users a question they're not adequately capable of making.
20:57:54 <sgallagh> (Or rather, if they *are* capable of making it, they'd do it themselves)
20:57:57 <vvaldez> I could see that
20:58:06 <smooge> sgallagh, should the role that installs the containers run inside a container?
20:58:13 <dperpeet> containerizing shouldn't be done blindly
20:58:39 <sgallagh> dperpeet: Well, there are real advantages to containerizing *services*, like these roles.
20:58:48 <adamw> jeez, i step out for ten minutes to run a fedora 13 install and come back and you're all talking about flowers?
20:59:00 <sgallagh> Not least of which being that they can tolerate an upgrade of the host without themselves having to be updated on the same schedule
20:59:01 <smooge> adamw, we set it up for you
20:59:07 <dperpeet> of course, but I'm having trouble envisioning a "containerize" button in cockpit
20:59:48 <sgallagh> dperpeet: Yeah, I don't think that is a good idea.
20:59:51 <vvaldez> dperpeet: you can ignore that
21:00:06 <dperpeet> sgallagh, from a user story point of view, most should be solvable with containers
21:00:07 <smooge> I am waiting to see if containers to go out of fashion like thin computing does
21:00:16 <dperpeet> that doesn't mean in a single one
21:00:18 <sgallagh> I think we should provide the roles in whatever way delivers the best experience for the user and not burden them with knowing how we did it
21:01:10 <sgallagh> dperpeet: I don't think that's what vvaldez was suggesting
21:01:27 <dperpeet> although I would prefer to have a thoughtful design that lends itself to containerization, and not producing a monolithic container that does the job just to say "there's a container"
21:01:31 <sgallagh> My read on that was basically a toggle "this is done on bare metal" vs "this is done through containers"
21:01:46 <dperpeet> ok
21:01:57 <sgallagh> dperpeet: I'm reasonably certain you and I are saying exactly the same thing in different word
21:01:59 <sgallagh> *words
21:02:03 <vvaldez> sgallagh: yes exactly, but I wasn't remembering the target audience
21:03:22 <sgallagh> But my original question was: do we want to require that all roles should be a pod of containers? (I'll use that terminology to avoid confusion)
21:03:34 <sgallagh> Or all bare-metal, or do we allow either?
21:03:46 <sgallagh> I'm going to recommend that "both" is not an option for any one role.
21:04:08 <dperpeet> depends on the role, I'd say, with the default containers
21:04:28 <vvaldez> I like the idea of conainerizing them for the sheer fact of upgrade simplicity
21:04:40 <vvaldez> I agree, containerize by default where possible
21:04:48 <sgallagh> Rephrasing, would that mean "You can use either, but if you choose not to containerize, you'll have to justify it strongly"?
21:05:01 <dperpeet> yes
21:05:13 <vvaldez> +1
21:05:14 <sgallagh> I'm good with that.
21:05:37 <sgallagh> Anyone opposed to that?
21:05:51 <mjwolf> I'm fine with that
21:06:07 <sgallagh> /me notes that this carries an implication that Server will need to ship with a container management technology by default.
21:06:33 <sgallagh> docker, rkt, systemd-nspawn, openshift...
21:06:41 <dperpeet> sgallagh, or the ability to pull one if you want to enact roles
21:07:53 <sgallagh> hmm
21:08:02 <sgallagh> I can see arguments for both choices.
21:08:12 <vvaldez> don't forget cri-o
21:08:55 <sgallagh> dperpeet: On a related note: has PackageKit support landed in Cockpit?
21:09:16 <sgallagh> Can we stop shipping all the subpackages and just pull them in the first time we go to a tab, yet?
21:09:36 <dperpeet> sgallagh, nope
21:09:39 <sgallagh> /me sighs
21:09:48 <sgallagh> That would make it a lot easier to minimize the default package set.
21:10:00 <dperpeet> if this is a priority I'm sure it could be done
21:10:20 <dperpeet> software installation is currently in the design phase
21:10:28 <dperpeet> I can get packagekit back on the map
21:10:32 <sgallagh> It would be great if we could pull in storaged or the selinux troubleshooter on demand rather than install them by default
21:10:50 <sgallagh> dperpeet: I don't know if we need general purpose installation at this point.
21:11:03 <sgallagh> Opportunistic installation of cockpit components would be a good start.
21:11:10 <dperpeet> right, but this is part of the design
21:11:39 <dperpeet> I'll look into it
21:12:04 <dperpeet> "opportunistic installation" is definitely a goal
21:12:13 <dperpeet> part of making a system discoverable
21:12:14 <sgallagh> OK, I'm going to call rathole on myself here
21:13:00 <sgallagh> #info After much discussion, we largely agreed on "You can use either containers or bare-metal for a role, but if you choose not to containerize, you'll have to justify it strongly"
21:13:27 <sgallagh> We're over time and I need to go cook dinner. Shall we call it a day for now?
21:13:39 <dperpeet> sure
21:13:44 <smooge> sounds good
21:13:46 <vvaldez> sgallagh: sounds godo
21:13:52 <smooge> and I am ok with the agreed statement
21:14:01 <sgallagh> Thanks for coming folks (and a lively discussion it has been)
21:14:13 <dperpeet> thanks!
21:14:17 <vvaldez> thanks sgallagh and all
21:15:06 <sgallagh> #endmeeting