14:34:09 <jzb> #startmeeting CentOS Atomic SIG
14:34:09 <centbot> Meeting started Tue Oct  7 14:34:06 2014 UTC.  The chair is jzb. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:34:09 <centbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:34:09 <zodbot> Meeting started Tue Oct  7 14:34:09 2014 UTC.  The chair is jzb. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:34:09 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:34:24 * Evolution here
14:34:30 <jzb> #chair quaid imcleod jbrooks Evolution gregdek walters
14:34:30 <zodbot> Current chairs: Evolution gregdek imcleod jbrooks jzb quaid walters
14:34:30 <centbot> Current chairs: Evolution gregdek imcleod jbrooks jzb quaid walters
14:34:38 <imcleod> .hellomynameis imcleod
14:34:39 <zodbot> imcleod: imcleod 'Ian McLeod' <imcleod@redhat.com>
14:34:49 <jzb> .hellomynameis jzb
14:34:51 <zodbot> jzb: jzb 'Joe Brockmeier' <jzb@redhat.com>
14:35:20 <jbrooks> .hellomynameis jasonbrooks
14:35:21 <zodbot> jbrooks: jasonbrooks 'None' <JBROOKS@REDHAT.COM>
14:35:32 <jzb> #topic Hardware for bare metal work
14:35:54 <jzb> quaid: I think this is still in process, but the request has been filed, yeah?
14:36:11 <jzb> kbsingh: you had some questions on this?
14:36:24 <Evolution> jzb: does it need to be bare metal, or is a vm acceptable?
14:36:50 <jzb> Evolution: this was for Anaconda work, IIRC
14:36:59 <dwalsh> Hello
14:37:05 <jzb> Evolution: walters was saying he needed bare metal
14:37:19 <jzb> dwalsh: welcome
14:37:24 <walters> Evolution, the compose scripts use (libguestfs | ImageFactory) which both spawn VMs, so if the compose server is a VM, you're in vm-in-vm land which is slow.
14:37:26 <imcleod> walters, jzb: FWIW, we have been able to use VMs with nested-virt enabled for ImageFactory and Oz development.  Perf is slightly degraded but not horrible.  Far better than full emulation.
14:37:36 <Evolution> okay. that should be doable. a vm is easiest to provision, but we should be able to allocate bare metal as well.
14:37:44 <imcleod> walters: I can point you at an internal host to fiddle on if you'd like to try.
14:38:08 <imcleod> OTOH-I fully support HW purchases.  Always.....
14:38:40 <jzb> Evolution: I think you're looped in on the email thread about this, but if not I will loop you in.
14:39:07 <Evolution> jzb: you and I each filed a bug on this, so mostly we just need to make sure we're saying the right things. I linked mine to yours.
14:39:08 <walters> imcleod, last i tried it was ~10x perf hit
14:39:20 <walters> though things are worse with libguestfs
14:39:24 <kbsingh> hi guys
14:39:38 <jzb> kbsingh: we're on the hardware topic
14:39:51 <jzb> kbsingh: maybe we can get that hammered out now?
14:39:59 <kbsingh> sounds good
14:40:23 <jzb> kbsingh: what do we need to settle to get that provisioned ASAP?
14:40:29 <kbsingh> so what do we need the hardware machine for - the anaconda dev team has a rather large lab of their own
14:40:40 <kbsingh> which also includes a mainframe iirc
14:41:12 <walters> to be clear if a VM is easiest to quickstart i'm fine with that
14:41:30 <walters> the treecompose part *doesn't* need baremetal, and that's most critical to run often, the images are a periodic thing
14:41:55 <kbsingh> I am still struggling to figure out what the plan or intention is
14:42:05 <jbrooks> kbsingh, building trees and images
14:42:23 <kbsingh> jbrooks: so you need access to a git repo ?
14:42:42 <jbrooks> kbsingh, a machine, preferably bare metal
14:42:47 <jbrooks> is what Colin needs
14:43:00 <walters> (and docker base and prototyping out layered images are an important part of this)
14:43:36 <imcleod> walters, kbsingh: Ideally, we'd get some additional baremetal available to the CBS koji instance and start building there, in a tracked and shareable way.
14:43:44 <imcleod> Similar to what we are doing for the Fedora builds.
14:43:47 <imcleod> That's my thought anyway.
14:44:38 <kbsingh> imcleod: thats my understanding as well, its what i sort of mentioned in the cbs meeting on Monday - if were building official images, and if ImgFac is able to do that now, then the builders should come up in cbs
14:44:51 <kbsingh> which is also why i am struggling to workout what this baremetal machine is needed for.
14:45:23 <kbsingh> we have a few machine ( 6 to be exact ) that can be used for SIG work, and i can likely find resources on a machine - eg the one that virt sig is using for some of their stuff )
14:45:29 <quaid> I see walters saying enormous perf problems with vm-on-vm
14:46:08 <kbsingh> but at the moment, i dont think anyone actually has either a plan or a direction they want to go into mapped out
14:46:15 <kbsingh> so maybe first step might be to work that out
14:46:27 <jbrooks> We talked about this last meeting
14:46:37 <kbsingh> ( either that, or i am missing something... in which case, tell me.. and we can just go find a machine for 45 days, and revisit then )
14:47:17 <jbrooks> walters requested a machine, and getting that machine is what we're talking about -- maybe Colin can provide more detail?
14:47:37 * jbrooks has done all my stuff in VMs so far
14:48:19 <kbsingh> lets start at the top
14:48:30 <kbsingh> we have an atomic build out there, should we get that updated real quick ?
14:48:35 <quaid> kbsingh: is a plan as simple as, "We need a machine for 60 days to do development work on X, with a goal of picking out the best pieces of that infra and putting them in cbs.centos.org."
14:48:35 <kbsingh> I've taken down the AMI already
14:48:50 <jbrooks> kbsingh, If that's going to be the build we'll use, we really need the json
14:49:00 <kbsingh> jbrooks: right, so lets work on that
14:49:05 <jbrooks> Like, in an email, or anything
14:49:28 <kbsingh> then, in the longer term ( ie. a week from now ) - how are we going to be building images ? so far, I hvent seen an authoritative answer on use-anaconda or use-toolbox
14:49:39 <jzb> kbsingh: is the system you've used to generate the existing image somewhere we can turn over to Ian?
14:50:07 <kbsingh> jzb: i think its just the jason
14:50:07 <kbsingh> erm
14:50:20 <kbsingh> json that needed, i can put that into a git repo and setup a pull from there to build
14:50:36 <jbrooks> That would be good
14:50:43 <kbsingh> if you guys want to skip to next point, i can go do the git thing now, and we can cover it before EOF
14:51:57 <jbrooks> That's fine w/ me
14:52:02 <walters> sorry, got pulled
14:52:07 <jbrooks> ah, cool
14:52:09 <walters> basically we can skip the baremetal if necessary
14:52:17 <walters> most important thing is to get the sources in centos
14:52:18 <jzb> walters: what do we lose if we skip that?
14:53:47 <jbrooks> speed, up to 10x, anecdotally, right?
14:54:40 <walters> yes just speed
14:55:08 <jzb> walters: "just speed" - is this going to inhibit the ability to get things done, though?
14:55:39 <walters> probably not
14:55:39 <jzb> walters: e.g., if you can do builds at regular speed will we gain days getting this out the door?
14:56:14 <jbrooks> And we can expect bare metal on the horizon when we mash this into koji
14:56:20 <jbrooks> cbs
14:56:38 <jzb> imcleod: anything to add / any thoughts here since this is also going to be on your plate?
14:57:06 <imcleod> jzb: Just that I think Colin should get whatever HW he wants, but I'll try to help speed up his VM performance in the meantime ;-).
14:57:17 <jzb> imcleod: +1
14:57:51 <jbrooks> excellent
14:58:07 <maxamillion> welp, I'm apparently terrible at telling UTC time and totally missed this one ... sorry all :(
14:58:14 <jzb> kbsingh: here's my suggestion, if we have the ability to provision this for a short term (e.g. 45 days)
14:58:20 <jzb> kbsingh: we should do so.
14:58:32 <jzb> kbsingh: if we can't, then let us know today and we'll move forward in VMs
14:58:39 <jzb> but I'd like to get this item decided
14:59:24 * dustymabe is just as bad as maxamillion
14:59:27 <kbsingh> jzb: so here is what i think we can do
15:00:01 <kbsingh> lets get the json files up, lets do a build or a few, lets get a good image we can promote at people in a way that they can use it - that should be fairly straightforward with whats already in place
15:00:33 <kbsingh> once we have an image set we are happy with, we can reprovision the box that is being used as the builderbox now, i can remove all non atomic stuff from there, and that can be the dev machine
15:00:57 <jzb> kbsingh: builderbox for Atomic?
15:01:03 <kbsingh> that also gives us the room to keep doing side builds, which dont need to make it to the release / dev cloud.centos.org setup
15:01:05 <jzb> kbsingh: does imcleod have access to that?
15:01:54 <kbsingh> no, noone has access to that apart from me and johnny, its part of the centos buildservices ( the private one )
15:02:00 <maxamillion> jzb: what's builderbox?
15:02:14 <jzb> maxamillion: ask kbsingh  :-)
15:02:26 <maxamillion> kbsingh: what's builderbox?
15:02:43 <kbsingh> arbitary container that runs builds :)
15:02:47 <maxamillion> (not trying to derail the meeting, just curious ... I can follow up later if that's preferred)
15:02:54 <kbsingh> can be baremetal or vm or cluster of vm's
15:03:03 <maxamillion> kbsingh: ah
15:03:05 <kbsingh> in this case, its a blade in a bladecenter
15:03:20 <jzb> kbsingh: to be clear, my understanding is that imcleod will be taking on getting the Atomic builds done.
15:03:36 <jzb> kbsingh: but it's sounding like this setup will still depend on you?
15:03:52 <kbsingh> hence my instance on a plan for it to not keep depending on me
15:04:22 <jzb> imcleod: can you and kbsingh get together and set up a timeline for transitioning this?
15:04:23 <kbsingh> where 'it' => the atomic image we ship for centos atomic
15:04:43 <kbsingh> transition to what ?
15:05:20 <jzb> kbsingh: where imcleod is getting the builds done
15:05:21 <imcleod> kbsingh: Transition to me being responsible for building it, and having access to the needed resources to do so.
15:05:26 <imcleod> jzb: ^^^ - correct?
15:05:43 <jzb> imcleod: yes
15:05:52 <imcleod> kbsingh: Sound good?
15:07:14 <kbsingh> to me this is more of a tooling thing than a resource thing, but i seem to be missing something here, so lets move on.
15:07:38 <kbsingh> is anaconda able to build atomic images at this point ?
15:07:54 <jzb> kbsingh: I'm not sure that's the case.
15:08:14 <jzb> kbsingh: what I'm trying to establish is at what point will imcleod be able to generate a build.
15:08:31 <imcleod> walters: Correct me if I'm wrong.  The answer on Anaconda is "yes in F21, yes in RHEL, not in CentOS at the moment"
15:09:16 <walters> right
15:09:29 <imcleod> kbsingh: This will get at another agenda item for my discussion with you.  Namely, assuming we have Anaconda patches to create an Atomic-capable CentOS 7 installer, I'm not sure how we would turn that into a composed CentOS 7 tree.
15:09:34 <jbrooks> Is this rhel 7.1's anaconda that we need?
15:10:01 <imcleod> kbsingh: The CBS is very clearly for generating additional packages.  The core build and compose are done on a different system in a non-public way, correct?
15:10:01 <walters> kbsingh, https://copr.fedoraproject.org/coprs/rvykydal/anaconda-el7-atomic/
15:10:05 <kbsingh> so then the 'transition' that is needed is to get the toolbox stuff deprecated out and replaced with an anaconda build able to generate the images we need, right /
15:10:36 <jbrooks> I think the transition is that the current test image can't be reproduced by anyone else
15:10:41 <kbsingh> imcleod: right, the distro is still built with the older system, as are the isos and images - eventually we will end up with one system
15:10:49 <jbrooks> And we need to transition away from that
15:11:20 <kbsingh> jbrooks: hopefully that will happen in a few, i just need to relocate to a diff machine and tunnel into machine to get content
15:11:33 <jbrooks> kbsingh, sweet
15:12:03 <jbrooks> Then, we can build images in the deprecated way. But if we're to start building the new way, we need documentation on that
15:12:16 <jbrooks> Is that all documented by the Fedora peeps now?
15:12:57 <walters> kbsingh, only kind of...the toolbox code now contains a wrapper for imagefactory in addition to the old libguestfs way
15:13:11 <imcleod> jbrooks: It's as documented as any of the fairly deep components of the build system, which is to say "slightly, and at least a bit by oral tradition".
15:13:35 <imcleod> jbrooks: But trying to duplicate the process in a parallel CentOS koji instance will certainly help identify doc gaps.
15:13:47 <jbrooks> imcleod, cool
15:13:57 <jbrooks> So for now, we proceed in the old way?
15:14:00 <jbrooks> Or both
15:14:20 <jbrooks> I want to see updated images and updates avail
15:14:43 <walters> me too =)
15:15:28 <jbrooks> We can take the draft atomic definition that jzb put together and use it to update the json
15:15:29 <imcleod> jbrooks: FWIW-I'm prepared to move forward either way, or both ways in parallel.  Right now the images are ad-hoc, by one person.  I don't think we need the next formal step to be "fully automated inside of a merged CentOS koji/core build system"
15:15:39 <jbrooks> Agreed
15:15:51 <imcleod> jbrooks: "Partially automated on a sidecar build host" sounds just fine.
15:16:03 <walters> i'm going back to step 0: figure out where to build packages for centos, and is it OK if they override (e.g. systemd)
15:16:20 <imcleod> walters: I believe we can do that in the CBS koji instance with appropriate tags.
15:16:27 <jbrooks> walters, I think that it's ok, by definition (of a SIG) that pkgs overlap
15:16:32 <jbrooks> Where to build is a good Q
15:16:44 <imcleod> walters: With a potential backup of continuing with COPR.
15:16:56 <jbrooks> COPR now, CBS soon?
15:17:07 <jzb> at this point I'm happy to just put down my CC for an AWS instance or instances to set up a build system
15:17:09 <jbrooks> The current test image uses copr pkgs
15:17:30 <jzb> if it's what it takes to get moving.
15:17:31 <imcleod> jzb "CC" ?
15:17:37 <jzb> imcleod: credit card
15:17:56 <imcleod> jzb: Unfortunately, the one place I can guarantee poor "VM in VM" perf is inside of any public cloud.
15:18:13 <jbrooks> At least one important pkg for us, docker 1.2, is currently built in CBS
15:18:25 <jzb> imcleod: just expressing my impatience
15:18:27 <imcleod> jzb: Now, if you're willing to put your CC down for ebay, I have some ideas.
15:18:28 <walters> yes, it's not just about the ostree part of atomic - need to figure out the docker interlock
15:18:48 <jzb> I feel like we've still not reached a plan here.
15:18:56 <walters> also we are trying to do geard -> kubernetes, which is now built in fedora
15:19:33 <jbrooks> OK, kbsingh is putting the json in a git repo, and we can use that to update the image
15:19:45 <jbrooks> W/ the update in the deprecated way to start
15:20:02 <imcleod> jbrooks: The deprecated way meaning toolbox, yes?
15:20:05 <jbrooks> new pkgs can come from a combo of copr and cbs, w/ the intention to move to all cbs soon
15:20:10 <jbrooks> imcleod, Yep
15:20:14 <kbsingh> https://github.com/CentOS/sig-atomic-buildscripts coming here
15:20:35 <kbsingh> we can also setup a sig-atomic team there to run that, otherwise Evolution and I can do pull req's in the interim
15:20:48 <kbsingh> jbrooks: ^
15:20:50 <jbrooks> thanks, kbsingh
15:20:51 <jzb> kbsingh: please add me and jbrooks
15:20:56 <jzb> and imcleod
15:21:01 <jzb> imcleod: what's your GH username?
15:21:21 <jzb> kbsingh: (I'm jzb)
15:21:22 <walters> i'm cgwalters
15:21:26 <jbrooks> mine is jasonbrooks
15:21:28 <imcleod> jzb: I am imcleod
15:21:33 <kbsingh> Evolution: ^
15:21:45 <jzb> jbrooks: I always look for "jbrooks" first.
15:22:00 * jbrooks not into the whole brevity thing, on GH
15:22:34 * number80 suggests that we add github handles in FAS3
15:23:01 <jbrooks> good idea
15:23:10 <jzb> OK, once we have the json - do we have a plan for next steps?
15:23:16 <jzb> imcleod: ^^
15:23:40 <Evolution> kbsingh: jperrin --^ gh username
15:23:41 <walters> is sig-atomic-buildscripts the equivalent of https://fedorahosted.org/fedora-atomic/ ?  ie the definition of the host?
15:23:58 <kbsingh> Evolution: erm, i meant could you please do the team thing :)
15:24:21 <Evolution> oh, certainly. wasn't paying attention and missed the context in scrollback
15:24:30 <kbsingh> thanks
15:24:40 <kbsingh> I'm not sure how user teams work without ading them to all of /CentOS
15:24:41 <imcleod> jzb: I will attempt to build with the JSON on HW I already have access to.
15:25:11 <imcleod> jzb: Then formulate a plan for getting the Docker and Atomic updates into either COPR or CBS on an ongoing basis, in collaboration with walters and dan.
15:25:29 <imcleod> jzb: Then formulate a plan for building regularly using that update stream.
15:25:51 <imcleod> jzb, kbsingh: Do we have AWS credentials for the CentOS project proper?  I can look at auto-updating AMIs in that case.
15:26:04 <jzb> #action imcleod Attempt to set up private builds, then a plan for getting Docker + Atomic updates into COPR or CBS
15:26:32 <jzb> #action imcleod work on plan to create regular Atomic builds
15:26:35 <kbsingh> imcleod: you'll need that from the project atomic side of things
15:26:47 <jzb> #action Evolution add SIG members to github repo
15:27:01 <jzb> #action kbsingh upload JSON files for Atomic to https://github.com/CentOS/sig-atomic-buildscripts
15:27:05 <jzb> did I miss anything?
15:27:34 <imcleod> kbsingh: By that you mean that project Atomic has its own credentials?  (Or needs them?)
15:28:19 <Evolution> okay. invites sent.
15:28:48 <kbsingh> imcleod: i'd guess so
15:28:58 <jzb> we're approaching the 1 hour mark, and I know several folks have a meeting
15:29:00 <kbsingh> although we have a marketplace setup etc, there is no 'centos' account in AWS
15:29:15 <imcleod> kbsingh: How have you uploaded AMIs?  PErsonal credentials?
15:29:19 <jzb> kbsingh: how do you upload the CentOS images?
15:29:22 <jzb> what imcleod  said.
15:29:37 <kbsingh> you dont need to upload AMI's in the AMP
15:30:03 <kbsingh> this is the decade of the cloud, these things happen over shared excel spreadsheets and raw backing files over http
15:30:21 <kbsingh> ( not kidding.. )
15:30:23 <jzb> o.O
15:31:04 <imcleod> kbsingh: I've never touched the marketplace stuff, only the public API to make public AMIs.  Anyhew, I guess we can pick this up on the next call, by voice.
15:31:07 <walters> overall, I would love to have publicly available CentOS Atomic hosts with online updates by AWS re:invent (Nov 11)
15:31:50 <kbsingh> walters: thats a good target
15:32:08 <jzb> walters: just for completeness I'll give you that AI
15:32:08 <jzb> :-)
15:32:35 <jzb> #action walters work with imcleod to see if we can get CentOS Atomic AMIs by AWS re:Invent.
15:32:37 <walters> the online updates is a very important aspect of this discussion - it's not just about uploading an image once, we need a flow of trees (and ideally, regularly updated docker images built from centos content too)
15:32:44 <jzb> walters: +1
15:33:30 <jzb> OK - any last items before we adjorn?
15:33:50 <walters> in rpm-ostree-toolbox we've been prototyping picking up package build notifications from a QPID instance after Koji builds
15:34:26 <walters> but if we just get the packages built that's a good first start
15:34:35 <walters> can someone link me to the docker CBS builds?
15:34:50 <walters> jzb, nothing else from me beyond that last q
15:35:01 <jzb> walters: OK, thanks@
15:35:03 <jzb> er, thanks!
15:35:11 <jzb> #action jzb get link to Docker CBS builds for walters
15:35:23 <jzb> OK, winding it up -see you all on the mailing lists.
15:35:26 <jzb> #endmeeting