15:03:13 <jzb> #startmeeting Atomic infrastructure
15:03:13 <zodbot> Meeting started Mon Jul 21 15:03:13 2014 UTC.  The chair is jzb. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:04:18 <jzb> #topic Minimum for Alpha
15:04:29 <jzb> OK, I sent out a quick agenda - everybody OK with that?
15:04:32 <mattdm> walters: I understand the urge to panic as the calendar pages keep turning. But I think we have the right people lined up to make it really work.
15:04:39 <mattdm> jzb: +1 agenda
15:04:48 * stickster looks at mail for agenda
15:04:49 <walters> ok, ~2 weeks to alpha right?
15:04:54 <nirik> walters: no worries at all. :)
15:04:55 <jzb> walters: can you give a quick summary of what we neeed for Alpha?
15:05:06 <jzb> (or others)
15:05:16 <jzb> let's figure out the MVP and then roll from there
15:05:18 <walters> i think there are two threads here; infrastructure and build/composing that are related
15:05:29 <stickster> That would be super helpful, since (not to step on nirik toes) I get the feeling the Fedora infra guys want to help but aren't sure what is needed/proposed.
15:05:57 * nirik nods. It seems like a lot of ?s are around releng/producing things more than distributing them.
15:06:14 <stickster> And from rel-eng perspective, oddshocks is vict^Wvolunteered to help but he says discussions he was invited to thus far were general so he may be in same boat
15:06:17 * stickster shuts up
15:06:29 <walters> on friday we were moving forward with mirroring the existing composes, and dgilmore was trying initial rpm-ostree composes inside the daily, but not sure how far he got
15:07:19 <misc> I did also ask to bochecha if he could help for the rel-eng part
15:07:35 <stickster> The more the merrier :-)
15:07:40 <walters> ok so minimum for alpha i think is having something consumable with updates i'd say; i.e. cloud qcow2 image, with tree available (and ideally mirrored) for receiving updates
15:08:05 <walters> this would be what we've had for months, just hopefully on better infrastructure
15:08:21 <walters> like, not a one off box in the infra scratch cloud with no ssl hosting the repo
15:08:22 <jzb> walters: so we're pretty much "green" if infra can help us with mirroring
15:08:24 <jzb> ?
15:08:50 <jzb> (on something with a valid SSL cert)
15:08:58 <nirik> so, one wrinkle on that... if we produce something at alpha and it's under the releases/testing/21/ tree... most mirrors assume anything under there is static. They can sync it once and be done.
15:09:00 <mattdm> walters: does this need to be a single-name, consistent mirror or will mirrormanager/redirects work?
15:09:19 <walters> jzb, yes...ish, if we move forward with the current tooling, which is distinct from the koji/imagefactory thread
15:09:45 <misc> jzb: well, we do not control the mirrors network, so i am not sure they can offer a valid ssl cert
15:09:45 <dgilmore> mattdm: we likely need mirrormanager work
15:09:50 <mattdm> From conversation with dgilmore that I remember, the suggestion was a new top-level (or close to it?), and then asking mirrors to opt in
15:10:02 <dgilmore> most mirrors do not offer https at all
15:10:12 <misc> and if they do, we cannot control the ssl cert
15:10:19 <dgilmore> misc: right
15:10:19 <misc> ( ie, some are likely self signed )
15:10:27 * nirik nods. Master mirrors do have https and a valid cert.
15:10:28 <walters> mattdm, ostree doesn't do the mirrorlist part yet; are we going straight to that instead of the dl.fp.org approach?
15:10:29 <mattdm> misc, jzb: and ssl does not give any meaningful assurance once it's off our network anyway
15:10:47 <dgilmore> walters: we have to make the mirror bit work
15:10:52 <walters> dgilmore, for alpha?
15:11:15 <misc> mattdm: well, if ostree verify the ssl cert to be one specific cert, it would, but yeah, outside of that, it doesn't help
15:11:18 <dgilmore> walters: I think so yeah. though i dont expect we will get too many users
15:11:37 <dgilmore> walters: we do have to assume the content is downloaded via http
15:11:46 <mattdm> walters, dgilmore is that entirely ostree coding or does it require mirrormanager changes?
15:11:52 <stickster> dgilmore: What's the reason not to do this in stages, then? Meaning, I don't think we want to "conveniently forget" to get mirrors working, but with a low number of users, why not plan to start with dl.fp.o?
15:12:10 <dgilmore> mattdm: i suspect both
15:12:10 * stickster might not fully grok the implications though
15:12:30 <dgilmore> stickster: I dont want to overload a host/hosts and have no way to fix it
15:12:31 * stickster didn't see oddshocks here but raises his hand in absentia ;-)
15:13:00 <mattdm> stickster right I was going to suggest oddshocks could at least tackle the mirrormanager side
15:13:34 <stickster> dgilmore: Do you think that's a high risk?  I mean, if we don't expect many users...
15:13:39 <mattdm> nirik can we get an infrastructure view on whether dl.fp.o might be overloaded?
15:13:42 <dgilmore> the closer we make things look like a yum repo the less work thats likely needed on the mm side
15:13:47 <stickster> mattdm++
15:13:55 <dgilmore> stickster: completely unknown
15:14:05 <nirik> well, hard to say... ;) I mean we could expect not many users and get a massive pile of them. Which would be good and all... but...
15:14:10 <misc> stickster: we can have someone trying to deploy it on 50 system, since atomic is for cluster and so have 50 time more the user than planned
15:14:24 <stickster> True.
15:14:34 <nirik> we could setup a mirror in another datacenter I suppose...
15:14:38 <misc> I also expect people to update a whole cluster at the same time :)
15:14:51 <stickster> #info Mirroring approach probably needed for scale, even if we don't project a high number of users.
15:15:09 <dgilmore> stickster: alpha might be okay without it
15:15:15 <dgilmore> Beta it will be essential
15:15:36 <nirik> yeah, I think I would be ok with just doing dl for alpha... but we should have a plan in place for beta, etc
15:15:44 <jzb> dgilmore, walters so can we tag that as "stage two"?
15:15:51 <jzb> nirik: +1
15:15:55 <mattdm> I agree -- in general alpha is usually our internal testing and beta gets the real interest (in my impression at least -- I don't have numbers)
15:16:24 <stickster> #info Volunteer oddshocks to work on MirrorManager code fixes needed (need to get this more granular, work with dgilmore/nirik)
15:16:34 <jzb> mattdm: I hope that Alpha will generate more interest than usual, but still shouldn't be overwhelming.
15:16:35 <nirik> also, if we put it under alt for alpha that makes it less... widespread than if it was in with the f21 alpha release.
15:16:51 <walters> ok, beta is september?  I'd be comfortable with that
15:17:01 <dgilmore> stickster: need to work with walters really
15:17:06 <stickster> #undo
15:17:15 <dgilmore> stickster: as we need mirrormanager to identiry an ostree repo
15:17:18 <stickster> #info Volunteer oddshocks to work on MirrorManager code fixes needed (need to get this more granular, work with dgilmore/nirik/walters as needed)
15:17:20 <mattdm> walters: beta change detadline is August 26th
15:17:22 <dgilmore> identify
15:17:28 <walters> mattdm, ah, right
15:17:40 <mattdm> alpha change deadline is, nominally, _tomorrow_.
15:17:50 <nirik> is there some central thing we can checksum to know what exactly is there in the tree?
15:18:17 <walters> nirik, ostree is all about content-addressed storage; however a single repository can contain multiple branches, deduplicating among them
15:19:09 <nirik> yeah... not sure how mm can deal with that. I guess if each branch has a serial or checksum or 'latest' is passed it could send back a mirror that has that branch?
15:19:42 <walters> http://rpm-ostree.cloud.fedoraproject.org/repo/refs/heads/fedora-atomic/rawhide/x86_64/server/docker-host
15:19:57 <walters> that branch name is just a file that contains a sha256 that points into the repo
15:20:09 * stickster recalled this was pretty much git-ish, yeah
15:20:38 <nirik> cool. Might be workable then...
15:21:04 <nirik> I'll note as a side note that pingou has been re-writing mm in flask... dunno how far along it is though.
15:21:26 <walters> i'll look at mirrormanager and the ostree side of parsing the data
15:21:30 <stickster> walters: nirik: I'm also interested in what we perceive as short/medium/long term storage needs -- but speak up if you think I'm jumping topics and I'll hold off for a minute.
15:21:40 * mattdm googles and is relieved to learn that flask is a python framework, not yet another new language :)
15:21:57 <stickster> mattdm: +1, we're moving more and more apps to that and off TG
15:22:11 <nirik> stickster: good question yeah.
15:22:16 <KidProton> mattdm: Slacker :)
15:22:21 <mattdm> walters: please call for help if you need another coder
15:22:39 * mattdm looks meaningfully over at oddshocks
15:23:08 <walters> stickster, basically strongly variable based on how often the RPMs change and how often we compose the tree, plus the obvious one of the RPM sizes
15:23:49 <jzb> walters: any best guess?
15:24:02 <mattdm> we could probably estimate this closely based on update history for f19 and f20 cloud images
15:24:27 <stickster> walters: I'd guess since the tree contains binary data that it's significant, though... mattdm++
15:25:20 <mattdm> The base size is around 0.5GB, plus, I guess, the size of all the deltarpms that have ever been generated against RPMs that are in those images
15:25:25 <stickster> Is there someone here who'd be able to take on the task of that estimate?  It's not critical path but I think it's information we'd REALLY like to have.
15:25:26 <walters> some current stats: http://fpaste.org/119559/95631914/
15:26:04 <stickster> walters: That's over what period?
15:26:09 <mattdm> walters ooh that's quite a bit bigger than I was thinking. but presumably that contains a lot of rawhide churn?
15:26:16 <nirik> 121 days?
15:26:26 <walters> 121 commits
15:26:37 <walters> Date:  2014-06-21 13:55:42 +0000 to Date:  2014-07-18 17:02:35 +0000
15:26:48 <walters> yep, constantly recomposing from rawhide
15:26:52 <stickster> Wow, so that's only a month
15:26:53 <nirik> so about a month?
15:27:03 <walters> yeah
15:27:05 <mattdm> hmmmm.
15:27:43 <stickster> This sounds like we'd need to think even in medium term about a pretty hefty storage increase.
15:28:00 <nirik> rawhide of course has a lot more churn than release trees, but yeah
15:28:16 <walters> seeing how big a single tree is
15:28:34 <walters> latest tree is 375MB compressed
15:28:38 <stickster> jzb: Heads up, that's something we should be addressing internally. Think $$. (Though not astronomical)
15:28:40 <walters> one commit
15:28:47 <jzb> stickster: ack
15:29:01 <walters> note the package set used to be a lot larger too
15:29:14 <walters> there was some depchain from cloud-init that pulled in wayland
15:29:30 <walters> or actually, it was abrt i think
15:29:37 <mattdm> walters in fact much larger for _most_ of that. not just wayland -- X11 too, and a ton of graphics libs
15:30:01 <mattdm> plus you went down from @standard to something more like the cloud image
15:30:14 <walters> right
15:30:41 <mattdm> So as we were discussing before, there is the option to keep less history
15:31:02 <nirik> I think some of this we are just going to need to pick defaults and adjust as we go forward...
15:31:03 <walters> yep, i fixed some bugs in prune over the weekend
15:31:05 <mattdm> proposal: reset tree _now_, keep track of how it grows over the course of alpha, make decision on what to do about that at beta.
15:31:21 <KidProton> mattdm +1
15:31:37 <stickster> mattdm: +1
15:31:41 <jzb> mattdm: +1
15:31:52 <nirik> sure, seems reasonable. ;) +1
15:31:52 * stickster would rather not hold things up trying to engineer perfectly, rather than iterate and improve
15:31:58 <jzb> stickster: +1
15:32:29 <walters> right now atomic01 isn't composing f21 branch even, so i have no stats on that, and starting would be a new history anyways
15:32:43 <jzb> we're 30-past, are we still on alpha or have we sort of drifted into blockers for 21?
15:32:49 <nirik> can we/should we mirror that to alt also starting now?
15:33:12 <stickster> jzb: +1, I can go longer if needed but I want to make sure we know who's tasked with what.
15:33:27 <walters> nirik, i'd like to mirror yes as an initial stopgap, but i'd like to know where we are on plans for official tree composes and mirroring those
15:33:47 <nirik> walters: sure. understood.
15:33:58 <jzb> I will have to duck out shortly - at OSCON running the Open Cloud Day event
15:34:04 <walters> dgilmore, did you have any more luck with that?
15:34:11 <jzb> but don't let that hold y'all up :-)
15:35:02 <stickster> jzb: Is this slot one that generally would work for you in future, e.g. get back together same bat-time, bat-channel next week?
15:35:09 <dgilmore> walters: with what sorry, releng meeting has just started so im not following
15:35:12 <jzb> stickster: yeah, just travel making it crappy
15:35:18 <stickster> jzb: +1, thx
15:35:40 <stickster> dgilmore: 11:33 < walters> nirik, i'd like to mirror yes as an initial stopgap, but i'd like to know where we are on plans for official tree composes and mirroring those
15:35:43 <bochecha> releng meeting is starting right now, though, maybe push this meeting a bit earlier?
15:35:45 <walters> dgilmore, running rpm-ostree as part of your compose script and having that content written to a place where koji can read, and is mirrored externally
15:36:18 <stickster> bochecha: Sure, we can fix time for next one after we get done with main topics here
15:36:25 <dgilmore> walters: i have it working for the TC/RC process. I need to change up a few things. not yet looked at doing it for rawhide/branched yet
15:36:32 <walters> dgilmore, we can break this topic into a ML thread too somewhere
15:36:35 <jzb> bochecha: I'm not opposed, though it will make things a bit ugh for people on the West coast if we have anyone there.
15:36:50 * stickster has a *lot* more open slots now that RHEL 7 is out ;-)
15:37:00 * nirik wouldn't be all that happy with a 8am monday meeting, but do what you must.
15:37:06 * jzb notes that stickster is bored and needs more to do... ;-)
15:37:08 <stickster> nirik: +10
15:37:32 <jzb> walters: can we give you the action to start the thread?
15:37:39 <walters> yep
15:37:39 <nirik> dgilmore: if we can tie into rawhide/branched, that would be great, because it would then mirror and we could figure out mm based on that
15:37:41 <jzb> walters: maybe infra is best
15:37:54 <stickster> #action stickster Send out whenisgood to figure out a slot for next week
15:37:59 <stickster> jzb: ^^ cool?
15:38:03 <jzb> #action walters will start a thread on official tree composes and mirroring..
15:38:17 <oddshocks> hi
15:38:17 <jzb> stickster: +1 thanks
15:38:21 <oddshocks> sorry >.<
15:38:28 <stickster> no worries oddshocks, we just gave you a bunch of TODOs
15:38:29 <mattdm> oddshocks np. we signed you up for all kinds of things.
15:38:36 <stickster> haha, the goto joke.
15:38:46 <walters> heh
15:38:48 <oddshocks> ok cool.
15:38:56 <jzb> Let's move to other immediate help items?
15:39:01 <jzb> any objections?
15:39:14 <stickster> oddshocks: j/k, but we did sign you up to work with walters & whoever else on MirrorManager bits that might need fixing.
15:39:41 <dgilmore> nirik: its doable, and really shouldnt take much effort
15:39:42 <stickster> (no surprises, iow)
15:40:03 <nirik> dgilmore: excellent. ;) So would it make a new tree every day then/ or have some way to update a tree's history with the new days changes?
15:40:16 <oddshocks> stickster: sounds good.
15:40:30 <stickster> dgilmore: boffo
15:40:53 <jzb> #topic Where do we need help immediately, and who can step up?
15:41:04 <jzb> anything else we're missing urgently right now?
15:41:09 <jzb> walters: ^^ ?
15:41:09 <walters> nirik, by default if you run compose tree with an existing commit, it will grow history.  If you run it on an empty repo, it starts a new history
15:41:43 <stickster> jzb: nirik: One thing to note for colo visit -- assess open space in case we need to consider equipment adds later to support atomic/ostree
15:42:13 <stickster> nirik: Maybe you guys know that like the back of your hands already... just thinking out loud
15:42:20 <nirik> walters: yeah, but not sure the logistics there.... I guess it could access the tree with history on the master mirror, but we dont' want to update it until we are ready. But thats all logistics.
15:42:32 <nirik> stickster: yeah. We should indeed check that...
15:42:48 <nirik> stickster: although netapps aren't in our racks if we get more netapp space.
15:43:12 <misc> getting more space on the netapp is gonna be complex IMHO
15:43:23 <misc> wouldn't a set of gluster server be more suitable ?
15:43:27 <jzb> I'm wondering if we can use cloud storage of some kind instead
15:43:42 <nirik> misc: not sure gluster is a good use here... lots of small files?
15:43:49 <misc> nirik: indeed :/
15:44:01 <walters> i was originally thinking ostree would be best backed by an object store like swift/s3
15:44:04 <KidProton> misc: Is block storage better? Ceph?
15:44:07 <misc> ( for some reason, i wasthinking it was the contrary )
15:44:10 <walters> or ceph's object storage
15:44:15 <misc> KidProton: yeah
15:44:28 <walters> but it would require custom code to interact with each object store
15:44:45 <nirik> well, I don't think we need to decide anything today.... lets see the delta over time in alpha.
15:44:45 <misc> for now, we can use regular posix FS
15:45:21 <walters> jzb, as far as missing, I think we should look from the Alpha goals; need regular cloud image composes, and regular tree updates mirrored.  Both of those can be done from atomic01.qa for now.  Then after that testing of the actual *content* =)
15:45:46 <dgilmore> nirik: to be decided
15:46:13 <jzb> walters: we're currently working on composes + mirroring, yes?
15:46:20 <mattdm> cloud image compose _in general_ still seems broken, let alone ostree....
15:46:22 <smooge> sorry guys all the calender entries said 10 am MT for me
15:46:27 <stickster> #info We need to consider where object stores might be backed in the future, based on the observed delta from F21 Alpha
15:46:38 * mattdm didn't dig into it but this morning's images still came out red
15:46:43 <jzb> so is there another item we're missing for MVP?
15:46:54 <misc> smooge: have no fear, oddshocks was also late so he was the one that got volunteered :p
15:48:06 <walters> jzb, i can't think of anything at the moment, there are various other threads but not critpath
15:48:07 <stickster> walters: As far as testing content... not sure what exactly you propose, is this a combination of automated testing and community pigpile?
15:48:13 <jzb> walters: yay!
15:49:19 <KidProton> walters: Can you point me to the alpha goals when you get a chance?
15:49:21 <walters> stickster, ah...i think we'll look at whatever mainline cloud is doing
15:49:41 <walters> KidProton, my goal is cloud image available with tree for updates
15:50:42 <mattdm> +1 to that goal
15:50:48 * oddshocks is caught up on backlog
15:50:54 <mattdm> for beta, "cloud image that works" :)
15:51:29 <jzb> hey - I am being distracted by the real world stuff, I will be logged in but a bit distracted for the next 15 minutes
15:51:59 <stickster> jzb: It seems like we're mainly done here. walters, is there any task we didn't put someone on, that you're worried is still floating?
15:52:04 <walters> btw, do we have any updates on https://fedoraproject.org/wiki/Changes/Docker_Container_Image ?  is that planned to be part of the compose too?
15:52:13 <stickster> s/task/critical path task/
15:52:22 <mattdm> walters yes, but separately.
15:52:40 <mattdm> (non-blocking parallel problem track)
15:52:42 <walters> stickster, i'm OK with the current state, i have some action items on my side and want to try mirroring current content to dl.fp.org
15:52:43 <jzb> stickster: thanks
15:52:49 <bochecha> just for the record, I've just recently started working in the RCM team at Red Hat, and I'm supposed to be given time to help on Fedora releng (which is pretty much all I'm doing at the moment, as the internal stuff hasn't really started taking my time)
15:52:57 <jzb> and thanks to everybody for all the work. This is going to be awesomsauce.
15:53:08 <walters> bochecha, cool =)
15:53:16 <KidProton> We've been generally posting action items, and stickster is going to work on the next meeting time, so those items are done.
15:53:17 <mattdm> bochecha++
15:53:20 <stickster> walters: If you feel like anything's at a standstill from infra side, let nirik, smooge, and me know
15:53:22 <walters> thanks all!  i'll trim the repo and mirror it right after this
15:54:11 <stickster> jzb: You may need to #endmeeting
15:54:12 <nirik> yeah, I'm super busy, but happy to move things along where I can.
15:54:14 <mattdm> If anyone would like to volunteer to help with the docker image creation (bochecha?) I'm sure dgilmore and lsm5 could use some help there too.
15:54:33 <jzb> #chair KidProton
15:54:33 <zodbot> Current chairs: KidProton jzb
15:54:50 <KidProton> If we are done, I'll close it out.
15:54:53 <bochecha> mattdm, we can talk about that after the meeting maybe?
15:54:59 <mattdm> bochecha yep!
15:55:05 <KidProton> Thanks all!
15:55:09 <KidProton> #endmeeting