19:00:01 <nirik> #startmeeting Infrastructure (2013-04-04) 19:00:01 <zodbot> Meeting started Thu Apr 4 19:00:01 2013 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:01 <nirik> #meetingname infrastructure 19:00:01 <zodbot> The meeting name has been set to 'infrastructure' 19:00:02 <nirik> #topic welcome y'all 19:00:02 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean 19:00:02 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean 19:00:19 * skvidal is here 19:00:25 * pingou here 19:00:26 * lmacken here for 15m 19:00:33 * relrod here 19:00:37 * threebean is here 19:00:52 * fchiulli is listening 19:01:18 <nirik> #topic New folks introductions and Apprentice tasks. 19:01:19 <smooge> is here 19:01:29 <nirik> any new folks or apprentices with questions or comments? 19:01:41 <nirik> oddshocks: you wanted to introduce yourself? ;) 19:01:44 * oddshocks is new! 19:01:52 <threebean> :P 19:02:09 <oddshocks> Hey everyone, I just sent out a mailing list introduction, I'm an IT student and free software hacker from RIT hoping to get involved with things around here :) 19:02:17 <oddshocks> potential Red Hat intern for the Summer 19:02:19 <skvidal> oddshocks: welcome 19:02:21 <oddshocks> nice to meet you all :) 19:02:23 <lmacken> welcome :) 19:02:28 <rossdylan> hye oddshocks :p 19:02:28 <nirik> welcome oddshocks 19:02:30 <pingou> nice, welcome :) 19:02:38 <nirik> you more interested in sysadmin or software devel? 19:03:35 <oddshocks> I'm still kinda on the fence, but systems administration catches my eye a bit more 19:03:51 <nirik> sure, you could also help out in both time and focus permitting. ;) 19:04:16 <oddshocks> I like writing code and I also like working with systems, so yeah 19:04:20 <nirik> anyhow, welcome and do see us after the meeting in #fedora-admin / #fedora-apps and we can point you at where to get started. 19:04:27 <oddshocks> Perfect, thanks! 19:04:33 <nirik> #topic Applications status / discussion 19:04:46 <nirik> any applications news? Note that we are in alpha freeze right now. 19:04:55 <abadger1999> wanna talk about gsoc here or open floor? 19:05:18 * pingou worked on pkgdb2 still 19:05:21 <nirik> either way. ;) 19:05:24 <pingou> I let threebean mention tagger :) 19:05:32 * threebean mentions tagger 19:05:47 * nirik will have a seperate topic about freeze discussions after apps/sysadmin ones. 19:06:01 <threebean> :P 19:06:11 <pingou> that was easy ;) 19:06:28 <abadger1999> GSoC is Google Summer of Code. It's coming up soon and Fedora has applied. We've already had students interested in working on infra come to find us in irc. What we really need to pull this off, though, is mentors. 19:06:30 <threebean> tagger dev work is getting close to something we could put in staging. early next week at the earliest. 19:06:52 <nirik> cool. 19:07:04 <nirik> abadger1999: yeah, most of us are pretty strapped for time tho. ;( 19:07:13 <abadger1999> Most of the usual suspects are busy this year (lmacken, myself, nirik, skvidal, pingou, etc). 19:07:15 <pingou> threebean: btw we should update the dev instance with the python-fedora from abadger1999 so we can start checking it 19:07:27 <skvidal> abadger1999: additionally i'm a terrible mentor 19:07:33 <threebean> pingou: oh yeah, let's set that up in #fedora-apps post meeting 19:07:45 <pingou> threebean: cool 19:07:49 <pingou> skvidal: really? :) 19:07:59 <nirik> abadger1999: some of the items are not really directly needing mentors from us... the gitlab thing could well be mostly packaging. 19:08:01 <abadger1999> I sent a request to the infra list for mentors to raise their hands if they want to work on something but so far no replies. 19:08:02 <skvidal> pingou: no need for you to mock 19:08:07 <abadger1999> <nod> 19:08:18 <threebean> ...chain.py! 19:08:47 <abadger1999> nirik: although... being useful to infra would make it more likely to be accepted. 19:08:51 <pingou> wrt GSoC: tbh, there has been some people that looked very motivated, it kind of makes me want to help them 19:08:58 <abadger1999> <nod> 19:09:01 <nirik> it's a bit difficult in that ideas on the page are submitted by anyone without infra really looking or saying they would like them. Perhaps next year we could be more proactive and prune ideas to ones we really would like and that might get more mentors involved. 19:09:21 <abadger1999> Yeah, biker has showed up a number of times and been pretty enthusiastic. 19:09:40 <pingou> skvidal: honestly, I wasn't 19:09:46 <nirik> yeah, agreed. 19:09:49 <abadger1999> He wondered about doing a unittesting GSoC... I'd be willing to partially mentor that. 19:10:03 <nirik> biker was also interested in any other ideas too... wasn't set on the ideas page ones. 19:10:07 <skvidal> pingou: SSSSSSSSSSSSSURRE you weren't :) 19:10:08 <abadger1999> <nod> 19:10:13 <pingou> skvidal: no really :) 19:10:32 <pingou> GSoC: pkgdb2? Elections? 19:10:32 <nirik> I've seen a lot of interest in gitlab and unit testing for fedpkg... 19:10:48 <pingou> start porting to ansible? 19:11:02 <pingou> MirrorManager? 19:11:02 <abadger1999> elections, fchiulli is managing... pretty far along -- but from the hours that he's available I don't htink he can mentor. 19:11:31 <nirik> I don't think mdomsch has time to mentor for mm... and not sure anyone else is up to speed enough yet. 19:11:33 * fchiulli is not available much during the day. Nights vary 19:11:49 <abadger1999> mirrormanager needs a new driver since mdomsch has less time. Without a driver, gsoc contributions are likely to not get merged. 19:11:58 <abadger1999> <nod> 19:12:19 <nirik> would anyone be willing to co-mentor with abadger1999 on unittesting? 19:12:21 <abadger1999> ansible would be good -- but that would need a sysadmin mentor... and you guys are all pretty tapped, right? 19:12:23 <nirik> perhaps thats one we could do? 19:13:21 <pingou> abadger1999: well I did work quite a bit trying to port MM to TG2 19:13:23 <nirik> the problem with the ansible side is that a lot of it is just deciding how we want to organize things... I'm not sure what GSoC folks could do there until we have more things in place and then not sure how exciting it would be. and yeah, time... 19:13:30 <abadger1999> <<nod> 19:13:40 <abadger1999> pingou: Want to take ownership? 19:14:01 <nirik> we still need to get mm1.4 into production. 19:14:06 <nirik> and fix issues with it. 19:14:06 <abadger1999> pingou: heh -- if you aren't careful, you'll end up with ownership of pkgdb as well ;-) 19:14:18 <pingou> abadger1999: I can deal w/ pkgdb more than MM :) 19:14:36 <pingou> abadger1999: but don't think you're out of the loop ;-) 19:14:50 <abadger1999> darn! ;-) 19:15:25 <nirik> so, where does that leave us? ;) 19:15:30 <abadger1999> anyhow -- if anyone wants to step up to mentor, I can co-mentor unittesting python-fedora and some other things. 19:15:50 <abadger1999> can't do it alone. 19:15:57 <pingou> pkgdb2 is an option? 19:16:08 <dgilmore> nirik: it would be awesome for some mm love 19:16:31 <nirik> sure, but without a driver on it, not sure GSoC would help us any 19:16:46 <dgilmore> yeah 19:16:53 <abadger1999> pingou: yes and no. If you want to mentor that I can help you --- but I wouldn't want to be primary mentor for that this summer. 19:16:54 * Smoother1rOgZ is around 19:17:26 <pingou> abadger1999: and I shouldn't (not that I wouldn't) 19:17:30 <abadger1999> hehe :-) 19:17:34 <abadger1999> understood :-) 19:17:50 <nirik> ok, well, lets keep discussing and see if we can offer anything... 19:17:54 <nirik> any other application news? 19:18:10 <abadger1999> python-fedora alpha is in infra; surprisingly stable. 19:18:21 <threebean> cool :) stg or prod? 19:18:22 <abadger1999> I just sent a change request to fix a few bugs. 19:18:28 <abadger1999> threebean: prod 19:18:30 <threebean> awesome 19:18:50 <abadger1999> working on a pkgdb update (pingou's slimming of the api and db) 19:18:51 <nirik> cool. 19:18:58 <abadger1999> will go to stg next week I hope 19:19:00 <nirik> that would be after alpha? 19:19:04 <nirik> (for prod 19:19:05 <nirik> ) 19:19:16 <abadger1999> and figuring out how to make otp fit into the python-fedora api 19:19:22 <abadger1999> nirik: correct. 19:19:25 <nirik> fun times. ;) 19:19:41 <threebean> good news: I've heard no negative feedback after tuesday's freeze break request - http://lists.fedoraproject.org/pipermail/infrastructure/2013-April/012724.html 19:19:42 <nirik> ok, moving along to sysadmin... 19:19:53 <nirik> threebean: yeah, seems ok, but the None's are weird. 19:19:57 * threebean nods 19:20:14 <threebean> those are omitted from the ircbot messages in git... coming down the pipe. 19:20:15 <nirik> #topic Sysadmin status / discussion 19:20:35 <nirik> threebean: yeah 19:20:41 <nirik> so, on the sysadmin side of things... 19:21:08 <nirik> we have 3/4ths of our arm SOC's on line and I will be redoing the other one soon when switch ports are changed on it. 19:21:18 <nirik> I created a sop for dealing with the arm socs: 19:21:27 <nirik> http://infrastructure.fedoraproject.org/infra/docs/arm.txt 19:21:56 <dgilmore> nirik: :) many thanks 19:22:11 <nirik> no problem. Nice to see them finally cranking away. 19:22:36 <nirik> I will likely request a freeze break in a bit for updating postgresql... might need a short outage for it. 19:23:21 <nirik> bnfs01 (our backup storage box) is hosed again. smooge was looking into it. 19:23:41 <nirik> we are finalizing our new netapp storage now, hopefully will have some in a while. 19:23:54 <smooge> it is not looking good 19:24:01 <nirik> smooge: ;( 19:24:22 <nirik> any other sysadmin stuff? 19:24:29 * skvidal has been working on ansible-y stuff 19:24:38 <skvidal> the cron-runner 19:24:40 <abadger1999> postgres update -- no dump and reload needed right? 19:24:41 <nirik> oh, I was thinking we should add monitoring for the arm boxes at some point. I might file a ticket on that and let someone else do it. ;) 19:24:46 <skvidal> the freezes thing (which uncovered a bug) 19:24:51 <nirik> abadger1999: correct. just update and restart 19:24:56 <abadger1999> cool. 19:25:34 <nirik> skvidal: so, were we going to do cron there? or triggered by commit? (or jury still out)? 19:25:45 <skvidal> nirik: well the script is the same, ultimately 19:25:52 <skvidal> so... 19:25:58 <nirik> yeah. 19:25:59 <skvidal> one way or the other we need to be able to say 19:26:09 <skvidal> 'these hosts or groups' - run the playbook for them and don't overlap 19:26:14 <nirik> I don't know if we can get ansible to say 'this commit affects these hosts' 19:26:17 <skvidal> run them forked off - multiple in parallel at a time 19:26:25 <nirik> yeah. 19:26:26 <skvidal> nirik: we can for playbooks, we can't, easily, for files 19:26:46 <nirik> ah. ok. 19:26:48 <skvidal> nirik: for tasks it is.... theoretically... possible - but there will be a lot of grepping involved 19:26:56 <pingou> speaking of ansible, do we finally have an idea how we want to handle stg/prod? 19:27:08 <nirik> yeah, we will just need to look at where we want to draw the line probibly. 19:27:36 <nirik> pingou: we did come up with a plan. ;) But it's not sure yet that it will work, so we need to test it... 19:27:41 <skvidal> :) 19:27:50 <nirik> basically similar to how we do it in puppet... a path for things. 19:28:05 <skvidal> that plugin should work - right now the issue is this 19:28:16 <nirik> skvidal: that was pulled into 1.1? or no? 19:28:20 <skvidal> nirik: pretty sure 19:28:22 <skvidal> lemme verify 19:28:37 <skvidal> yep 19:28:40 <skvidal> first_found lookup plugin 19:28:58 <skvidal> so - there are 2 things here, really 19:29:08 * nirik did update lockbox01 to 1.1 19:29:23 <skvidal> the playlets idea that I came up with is.... not very attractive unless we can make it easy to use in an existing playbook 19:29:42 <skvidal> in many case we can make-do w/o needing those playlets - just with included tasks 19:29:55 <skvidal> but I'm trying to figure out which cases where tasks will make it more confusing to use 19:29:56 <nirik> yeah. 19:30:11 <skvidal> in most of the cases I can think of 19:30:11 <nirik> yeah, I figure we will need to try things and adjust. 19:30:20 <skvidal> it's just where the variable includes would be ugly in every playbook 19:30:24 <skvidal> and you really just want to say 19:30:37 <skvidal> "you know that last group of hosts you ran against?" 19:30:48 <skvidal> "run this playbook over here, against them and only them" 19:31:38 <skvidal> if anyone wants to talk about this with me sometime 19:31:47 <skvidal> I think I could really use someone to bounce this back and forth with 19:32:04 <skvidal> http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/playlets/README 19:32:06 <skvidal> this is the idea 19:32:18 <threebean> cool -- I'll queue it up for reading 19:32:24 <skvidal> and then 19:32:41 <nirik> yeah, lots of ways to do things... ;) 19:33:20 <skvidal> http://paste.fedoraproject.org/6712/10399613/ 19:33:36 <nirik> ok, lets keep poking at it. ;) I'd like to add arm-releng to ansible repo soon so I can try and work out in reality how things will look 19:33:45 <skvidal> yah 19:33:59 <skvidal> did we already do cloud stuff? 19:34:00 <nirik> any other sysadmin stuff anyone has? 19:34:03 <nirik> not yet. ;) 19:34:05 <skvidal> ok 19:34:10 <nirik> #topic Private Cloud status update / discussion 19:34:14 <nirik> now. go! :) 19:34:18 <skvidal> heh 19:34:28 <skvidal> so I'd like to add a new ip range to the openstack cloudlet 19:34:42 <nirik> sounds good. are we ready to add more compute nodes too? 19:34:42 <skvidal> I'm almost positive that the command I want to run won't break the existing network ip range ;) 19:34:47 <skvidal> nirik: we are 19:35:02 <nirik> if it does... oh well, we do the best we can. ;) 19:35:02 <skvidal> fed-cloud05 and 07 need to be reformatted 19:35:16 <skvidal> but they are not doing anything now 19:35:33 <skvidal> so anyone in sysadmin-main who would like to kick them should feel free 19:35:47 <nirik> cool. 19:35:59 <skvidal> also 19:36:02 <nirik> adding compute nodes is super easy too... nova.conf and a few packages and away you go. 19:36:16 <skvidal> 01 and 03 should be ready to be turned over to grizzly if anyone would like to do that 19:36:29 <nirik> excellent. 19:36:32 <skvidal> this is sorta-cloud 19:36:52 <skvidal> I played with systemd-nspawn a bit and what it can do 19:37:12 <nirik> oh yeah. f18? 19:37:16 <skvidal> and I think if anyone would like to tinker with infrastructure/scripting/etc for that on a big fedora 18 cloud instance 19:37:27 <skvidal> it would be something that will eventually be useful to us 19:37:31 <skvidal> us == fedora infra 19:37:37 <nirik> yeah, I agree. 19:37:45 <skvidal> it is also completely bluesky afaict 19:37:51 <skvidal> so if you want to noodle around with it 19:37:54 * nirik nods. 19:37:58 <threebean> bluesky? 19:38:07 <skvidal> threebean: go, play, doesn't matter if nothing comes of it 19:38:10 <skvidal> threebean: no constraints 19:38:15 <nirik> not ready for real use right now, by someday... perhaps... 19:38:24 <threebean> ok, thanks 19:38:29 <skvidal> so a nspawn is a container 19:38:35 <skvidal> forming one up is easy 19:38:47 <nirik> I think the big things around it would be: managing them, and networking. 19:38:51 <skvidal> yum --installroot=/some/path install @core 19:39:09 <skvidal> systemd-nspawn -D /some/path -b 19:39:17 <skvidal> and then you're in the chroot, lickety-split 19:39:51 <nirik> yep. 19:40:00 <skvidal> anyway - if anyone wants to do that 19:40:04 <skvidal> yell at one of us 19:40:10 <skvidal> and we can spin up an f18 instance quickly 19:40:20 <nirik> sounds good. 19:40:26 <nirik> #topic Freezes 19:40:29 <skvidal> brrr 19:40:35 <nirik> so, we are in a freeze for alpha now... 19:40:43 <nirik> and the topic came up how we determine whats frozen, etc. 19:40:47 <nirik> (see post on list) 19:41:29 <nirik> would anyone like to make a case for more hosts NOT being frozen than dev/stg ? or some way to decide those? 19:41:48 <skvidal> nirik: there's one example of a host-specifically marked as freezes:false 19:41:51 <skvidal> people03.fedoraproject.org 19:42:01 <skvidal> adding additional ones like that to the inventory is really easy 19:42:22 <skvidal> nirik: actually - this brings up a question 19:42:32 <skvidal> nirik: right now we have the inventory inside the public ansible repo 19:42:33 <nirik> sure... but I was meaning from a standpoint of critera. 19:42:38 <abadger1999> for alpha/beta or final or both? 19:42:45 <skvidal> abadger1999: I think for all 19:42:50 <nirik> abadger1999: both, either... ;) 19:43:07 <nirik> I'd like to have a critera we can hold a host up to and see if it should be frozen or not. 19:43:26 <nirik> a very simple one would be: is the host *stg* or *dev* ? 19:43:35 <nirik> but that means a lot more things are frozen than were before. 19:43:55 <abadger1999> yeah... I don't really like mor ethings frozen. 19:44:10 <skvidal> nirik: I think there is room for more non-distro-releasing-impacting things being unfrozen 19:44:29 <skvidal> people, hosted, hosted-lists, 19:44:31 <nirik> yeah, but what critera can we use. ;) 19:44:34 <skvidal> the items you mentioned in your email 19:44:38 <nirik> I posted some other attempts... 19:44:42 <skvidal> nirik: 'does it impact the ability to release the distro' 19:44:54 <nirik> well, hosted actually does impact the distro. ;) 19:45:07 <skvidal> nirik: how? 19:45:07 <nirik> comps is pulled from there. 19:45:40 <skvidal> sigh 19:45:43 <abadger1999> comps isn't mirrored into a package? 19:45:50 <nirik> anyhow, we could continue discussion on list too. :) 19:45:57 <abadger1999> i guess its repodata so no.. :-( 19:46:02 <nirik> abadger1999: no, it's pulled from git by composes... 19:46:09 <nirik> also spin-kickstarts is there. 19:46:25 <nirik> now, we could look at moving those, but right now they are there. ;0 19:46:42 <nirik> brb 19:47:22 <skvidal> yah.... I see 19:47:31 * skvidal grumbles... 19:47:34 <skvidal> okay fine 19:47:40 <skvidal> hosted-lists! 19:47:42 <dgilmore> nirik: and branched is pulled from git on fedorahosted nightly 19:48:06 <nirik> yeah, rel-eng repo too... 19:48:27 <skvidal> so - back to my earlier question 19:48:35 <nirik> anyhow, lets continue that discussion on list... for purposes of this alpha freeze, we just keep using the picture. ;) 19:48:37 <skvidal> do we want to split out the inventory subdir of the ansible repo - into its own repo 19:48:47 <nirik> skvidal: what does that get us? 19:48:51 <skvidal> so we can have people editing the inventory who might not be able to commit to ansible itself 19:49:13 <nirik> not sure thats a very common thing is it? 19:49:18 <skvidal> true enough 19:49:28 <skvidal> this is why I ask these things ;) 19:49:30 <nirik> I'd prefer to keep number of repos down if we can. ;) 19:50:05 <nirik> commit on ansible is going to be the same as commit on puppet right? 19:50:15 <nirik> which is what... sysadmin? 19:50:20 <skvidal> lemme look 19:50:34 <skvidal> yes 19:50:34 <skvidal> sysadmin 19:50:36 <skvidal> yah 19:50:38 <skvidal> you're right 19:50:43 <skvidal> I withdraw the suggestions 19:51:12 <nirik> :) of course they must be in a group that has shell on lockbox01 too... 19:51:21 <nirik> #topic Upcoming Tasks/Items 19:51:33 <nirik> here's what I have for upcoming: 19:51:35 <nirik> #info 2013-04-02 to 2013-04-16 ALPHA infrastructure freeze 19:51:36 <nirik> #info 2013-04-15 - remove legacy-openid 19:51:36 <nirik> #info 2013-04-15 - announce openid change on hosted. 19:51:36 <nirik> #info 2013-04-16 F19 alpha release 19:51:36 <nirik> #info 2013-04-17 - switch to new sigul servers. 19:51:37 <nirik> #info 2013-04-19 - switch hosted to openid 19:51:38 <nirik> #info 2013-05-07 to 2013-05-21 BETA infrastructure freeze 19:51:40 <nirik> #info 2013-05-21 F19 beta release 19:51:42 <nirik> #info 2013-05-31 end of 1st quarter 19:51:44 <nirik> #info 2013-06-11 to 2013-06-25 FINAL infrastructure freeze. 19:51:46 <nirik> #info 2013-06-25 F19 FINAL release 19:51:48 <nirik> any other items people would like to schedule or note? 19:52:34 <nirik> #topic Open Floor 19:52:41 <nirik> anyone have items for open floor? 19:52:58 <skvidal> nirik: one question 19:53:24 <skvidal> nirik: what were the future points for determining what to do with hosted 19:53:31 <skvidal> nirik: and making it more-better 19:53:44 <skvidal> as in - it's running right now 19:53:46 <nirik> yeah... so I think we decided on at least a short term plan... 19:53:51 <skvidal> do we just sit on it? 19:53:57 <skvidal> ah - okay - 19:54:02 <nirik> well, we could do that too I guess... 19:54:10 <nirik> I was thinking we could move it to osuosl02... 19:54:28 <nirik> (or at least all the git ones) 19:54:31 <skvidal> okay - so that's something we do plan to do. 19:54:40 <skvidal> did you decide if you wanted osuosl02 to be cloud-y or not? 19:55:15 <nirik> I was really on the fence there. I think cloud would be nice from a 'figure more stuff out' with it, but on the other hand, I don't know how much time we want to spend on hosted really... 19:55:40 <nirik> Ideally we can get it to a stablish thing and not have to deal with it again for a while. 19:56:11 <nirik> let me try and write up a plan and we can tweak it and see what all of it we want to do. 19:56:11 <skvidal> nirik: so... what does that mean? 19:56:13 <skvidal> ok 19:56:18 <skvidal> sounds round 19:56:19 <nirik> I can put it on that hosted wiki page with all the ideas. 19:56:21 <skvidal> thank you 19:56:45 <nirik> hosted02 is running ok now, but I think I'd like to move it anyhow, even if we do nothing else but move it. 19:57:19 <skvidal> cool 19:57:32 * nirik adds to his list. ;) 19:57:37 <nirik> ok, anything else ? 19:58:08 <nirik> ok, thanks for coming everyone! 19:58:09 <threebean> not from me :) 19:58:14 <nirik> #endmeeting