20:00:06 <nirik> #startmeeting Infrastructure (2012-03-29)
20:00:06 <nirik> #meetingname infrastructure
20:00:06 <nirik> #topic Robot Roll Call
20:00:06 <nirik> #chair smooge skvidal Codeblock ricky nirik abadger1999 lmacken dgilmore mdomsch
20:00:06 <zodbot> Meeting started Thu Mar 29 20:00:06 2012 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:06 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:06 <zodbot> The meeting name has been set to 'infrastructure'
20:00:06 <zodbot> Current chairs: Codeblock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge
20:00:09 <CodeBlock> here
20:00:16 * wolfkit is here
20:00:17 * skvidal is here
20:00:18 <nirik> who all is around for a infra meeting? fun and exciting! :)
20:00:46 * kwame here
20:01:02 * mdomsch_ 
20:01:07 * adrianhannah is here
20:01:23 <smooge> is here
20:02:12 <nirik> ok, lets dive on in...
20:02:23 <nirik> #topic New folks introductions and Apprentice tasks.
20:02:23 <nirik> If any new folks want to give a quick one line bio or any apprentices
20:02:23 <nirik> would like to ask general questions, they can do so now. Anyone?
20:02:33 <dgilmore> hey yall
20:02:54 <nirik> morning dgilmore
20:03:13 * nirik will wait a few on this then move on...
20:03:34 * abadger1999 here
20:03:34 * pingou here
20:03:54 <nirik> #topic two factor auth status
20:04:05 <nirik> anything new to report here? wolfkit ? skvidal ?
20:04:09 <skvidal> yes
20:04:17 <skvidal> bressers has passed me a sheaf of patches
20:04:22 <skvidal> fixing a number of things in pam_url
20:04:24 <nirik> awesome.
20:04:36 <wolfkit> skvidal: great, is there a place I can get these patches too?
20:04:37 <skvidal> I'm going to roll those in and let mr wolfkit take a look and mricon@kernel.org
20:04:42 <skvidal> wolfkit: there will be shortly :)
20:04:47 <wolfkit> cool :)
20:04:55 <skvidal> also I spoke with mricon today about the totp.cgi
20:05:07 <skvidal> he has some folks testing it out and pam_url out in "production"
20:05:15 <skvidal> so not _Really_ production but in real life
20:05:21 <nirik> nice.
20:05:24 <skvidal> and there are some issues - but perhaps solveable ones
20:05:37 <skvidal> I think we're moving in on a solution
20:05:43 <nirik> ok. If it would help we could setup a test instance somewhere too...
20:05:49 <skvidal> I think it will
20:05:58 <skvidal> but I am glad to have responses back from bressers
20:06:13 <skvidal> b/c it feels like we're doing the right "due diligence" on something so security sensitive as this
20:06:21 <skvidal> kwim?
20:06:29 <nirik> yes. We don't want to deploy something that messy/insecure for sure.
20:06:45 <skvidal> I have a test setup on my local system
20:06:50 <skvidal> that is just a couple of vm's
20:06:59 <skvidal> I may see about doing the same on a junk box
20:07:06 <nirik> sounds good.
20:07:10 <skvidal> also
20:07:18 <nirik> #info narrowing in on deployable solution.
20:07:19 <skvidal> icon has made the totp.cgi so it takes an optional pin
20:07:26 <skvidal> so instead of just being the otp
20:07:31 <skvidal> it can be pin + otp
20:07:49 <nirik> good.
20:07:57 <skvidal> so a couple of things
20:08:03 <skvidal> if we want to make 2fa for the world
20:08:10 <skvidal> then, obviously, we're going to need to tie this into fas
20:08:25 <skvidal> but as a deployable/functional prototype for sysadmin-* or whatnot
20:08:34 <skvidal> I think we can probably get a way without the fas integration
20:08:47 <skvidal> and I'm sure it won't break abadger1999's heart to not have to deal with a fas rfe :)
20:08:52 <nirik> yep. Just a lookup of pin and secret...
20:08:55 <abadger1999> <nod>
20:08:56 <skvidal> until we know it's what we want
20:09:06 <skvidal> does anyone object to that?
20:09:21 <nirik> I think thats fine for a first cut...
20:09:40 <nirik> if it all runs nicely and we are ready to expand, then we can look at how best to add to fas.
20:09:42 <wolfkit> yeah, no reason to bother integrating into FAS until it's ready for 'primetime' :)
20:10:04 <skvidal> nod
20:10:08 <skvidal> for fas
20:10:15 <skvidal> it should be relatively straightforward
20:10:21 <skvidal> except we'd be storing actual seeds _somewhere_
20:10:25 <skvidal> which is the sensitive part
20:10:28 <nirik> #info will not integrate with fas for first cut, but after proof of concept is working.
20:10:41 <skvidal> abadger1999: hmmm
20:10:44 <nirik> yeah, can see how best to store those...
20:10:47 <skvidal> abadger1999: could we encode the seed in fas with the pin?
20:11:06 <skvidal> abadger1999: so if the pin is wrong or the otp is wrong you get rejected?
20:11:29 <skvidal> abadger1999: it wouldn't be A LOT of security if our db was compromised
20:11:32 <skvidal> but it might slow someone down
20:12:14 <skvidal> anyway
20:12:16 <skvidal> that's off in the weeds
20:12:33 <nirik> yeah, we can figure that out more when we get there. ;)
20:12:36 * smooge gets the weedwhacker
20:12:47 <nirik> thanks for the news and work on this skvidal and icon and wolfkit.
20:12:48 <skvidal> heh
20:12:55 <skvidal> nirik: and bressers
20:12:59 <skvidal> he's been very helpful
20:13:09 <nirik> yes. agreed.
20:13:12 <abadger1999> I guess we could but it would be trivial to reverse
20:14:30 <skvidal> well only if the pin is kept in the clear
20:14:37 <skvidal> anyway
20:14:42 <nirik> right, moving along.
20:14:55 <nirik> #topic Staging re-work status
20:15:11 <nirik> I've done some docs on this, still need to work on it more. and will do it after freeze is over.
20:15:17 <nirik> #topic Applications status / discussion
20:15:22 <nirik> any application news?
20:15:36 <nirik> abadger1999 / lmacken / pingou / threebean / CodeBlock
20:16:14 <wolfkit> haven't been around much the past few days, did anyone figure out what happened with mirrormanager wednesday morning?
20:16:41 * nirik isn't sure.
20:16:42 <mdomsch_> wolfkit, logs were blissfully unenlightening
20:17:00 <abadger1999> pingou has been doing some *great* work on the mailman3 archiver instance
20:17:11 <abadger1999> http://mm3test.fedoraproject.org/
20:17:27 <nirik> oh, I have one bit of news... I went and triaged all the bodhi bugs. Closed dups, closed things that were done, etc... cut them down some. I also added easyfix to some tickets that sound like they would be easy to fix for new folks.
20:17:43 <nirik> nice!
20:17:50 <CodeBlock> I have nothing new for dpsearch. I tried pinging damaestro the other day but he was at dayjob and I didn't have a chance to ping him later that evening. I will try to catch up with him either tonight after class or tomorrow evening (tomorrow is probably better).
20:17:52 <abadger1999> We may be getting a GSoC student to work on that -- I'm trying to start discussion with terri of the mailman devs about who's going to mentor and which org is going to sponsor and such
20:18:23 <pingou> I hope to be able to couple HK with mm3 this week-end (if abadger1999 has some time as well)
20:18:38 <abadger1999> pingou's latest addition to the archiver is to start implementing a REST API: http://mm3test.fedoraproject.org/api/
20:19:22 <nirik> pingou: HK?
20:19:29 <pingou> HyperKitty :)
20:19:44 <nirik> ah, right
20:20:37 <skvidal> so
20:20:42 <skvidal> mm3test has been useful
20:20:51 <skvidal> do we want to leave it as is
20:20:56 <skvidal> or setup another instance
20:21:40 <skvidal> ?
20:21:49 <nirik> oh, also we have some smolt news: npmccallum has setup a new project called 'census' that will replace much of smolt. https://fedorahosted.org/census/browser
20:22:04 <abadger1999> skvidal: at some point I think we want another instance but -- things are still very very developy atm
20:22:07 * nirik is fine on mm3test with whatever works for the people working on it.
20:22:09 <skvidal> abadger1999: if you'd like to leave mm3test as is b/c it lets you move more fluid that's fine
20:22:15 <skvidal> abadger1999: gotcha
20:22:16 <skvidal> absolutely
20:22:23 <skvidal> the machine is automatically applying patches/rpms
20:22:29 <abadger1999> we're doing development in a virtualenv -- so some things aren't packaged and such.
20:22:31 <skvidal> and it is pretty isolated
20:22:37 <skvidal> abadger1999: I completely understand
20:22:42 <skvidal> and am supportive of that
20:22:47 <skvidal> I just wanted to help if you needed help
20:22:51 <abadger1999> <nod>
20:22:57 <abadger1999> Thanks
20:23:03 <skvidal> cool
20:23:56 <pingou> it don't think it will be too hard to get packaged, the dependency list isn't too high I think
20:23:57 <nirik> ok, any other appy news?
20:25:38 <smooge> hey I wonder if we could make the hosted mail mm3?
20:25:44 <smooge> it is the smaller of the listsets
20:25:50 <skvidal> smooge: I wouldn't want mm3 in 'production' yet
20:25:53 <nirik> once it's usable, yeah...
20:26:10 <smooge> i am not talking production.. I am talking hosted :)
20:26:21 <smooge> but yeah I understand
20:26:30 <pingou> smooge: mm3 core is just beta - the admin web ui alpha
20:26:55 <smooge> ah I thought it might be beta through and through
20:26:59 <smooge> okie dokie
20:27:25 <pingou> and the archiver pre-alpha-rc
20:27:28 <nirik> ok, moving along then...
20:27:37 <nirik> #topic Upcoming Tasks/Items
20:28:00 <nirik> ok, we have slipped a week on beta, which slips the entire schedule a week.
20:28:03 <nirik> #info 2012-03-20 to 2012-04-10 - F17 Beta Freeze
20:28:03 <nirik> #info 2012-03-29 - take internetx01 out of rotation and power off
20:28:03 <nirik> #info 2012-03-30 - 1:30am - run diag on internetx01.
20:28:03 <nirik> #info 2012-04-01 - nag fi-apprentices.
20:28:03 <nirik> #info 2011-04-03 - gitweb-cache removal day.
20:28:03 <nirik> #info 2012-04-10 - drop inactive fi-apprentices
20:28:07 <nirik> #info 2012-04-10 - F17Beta release day
20:28:09 <nirik> #info 2012-05-01 to 2012-05-15 - F17 Final Freeze.
20:28:11 <nirik> #info 2012-05-01 - nag fi-apprentices.
20:28:15 <nirik> #info 2012-05-15 - F17 release
20:28:15 <nirik> also, we have some upcoming maint on internetx01.
20:28:24 <nirik> It's been locking up, so we want to take it down and run a hardware test on it.
20:29:40 <nirik> I am going to take it's proxy out and power it off for that later tonight.
20:29:52 <nirik> anything else anyone would like to schedule or note?
20:30:02 <skvidal> nothing leaps to mind
20:30:15 <skvidal> I've been working more with ansible and its playbooks
20:30:26 <skvidal> and I have the basis for something pretty workable for the builders
20:30:32 <skvidal> if anyone wants to get involved and check it out
20:30:52 <skvidal> ping at me or check out ansible on github or #ansible
20:31:20 <nirik> sounds good.
20:31:47 <nirik> #topic Tickets from Ages past
20:32:04 <nirik> anyone want to bring up some ancient tickets we can discuss and dispatch?
20:33:02 <nirik> https://fedorahosted.org/fedora-infrastructure/report/1?sort=created&asc=1 is our full list by filed time.
20:33:08 * skvidal looks at the ground and whistles
20:33:09 <nirik> oh, I also filed the F17 beta tickets.
20:33:53 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/1783 looks like something we could close.
20:34:08 <skvidal> +1
20:34:13 <kwame> :fit
20:35:01 <nirik> #topic Open Floor
20:35:08 <nirik> anyone have any general items for open floor?
20:35:29 <kwame> o/
20:35:44 <skvidal> kwame: is that your hand up?
20:35:50 <kwame> so, I've been in the #fedora-admin channel for some week snow
20:35:55 <kwame> skvidal: yes :)
20:35:58 <skvidal> :)
20:36:14 <kwame> and already hace access to some boxes
20:36:34 <kwame> last week I spoke with nirik about a ticket that I thought I could work on
20:36:40 <smooge> I changed people's email passwords and am working on a blog about
20:36:44 <smooge> its slow going.
20:36:56 <smooge> oh sorry. shuts up for kwame.
20:37:41 <kwame> sorry
20:37:59 <kwame> so, my question is basically, how can we know who are the new commers in here (like me)
20:38:29 <nirik> good question... we don't keep a very good list or whatnot of new folks...
20:38:59 <nirik> we do have the fi-apprentice group. You can mostly be sure anyone in the group as a user is new, if they are a sponsor or admin they have been around a while...
20:39:10 <nirik> but that doesn't cover everyone.
20:39:16 <nirik> .members fi-apprentice
20:39:16 <zodbot> nirik: Members of fi-apprentice: adrianhannah aphukan arielb +codeblock ctria davidvz fortu icon informatux jacibato jsh @kevin kubo mangas marcelk +nb @skvidal smillie @smooge +toshio
20:39:56 <nirik> Open to ideas on improving things...
20:40:14 <kwame> nirik: one more question, the ticket I tried to work on was very simple, just install nrpe and get a new nrpe.cfg in place
20:40:40 <kwame> now, who should I ask about access, and just for +1 or -1 on how to access and do that kind of work? is the mailing list the proper place to do it?
20:40:43 <kwame> or just irc?
20:41:36 <nirik> which ticket? you're welcome to ask on irc... usually for apprentices we ask that you attach to the ticket a patch against our puppet repo that makes the change or fix. Then someone will apply it if all looks well.
20:42:31 <nirik> we could definitely clarify that workflow tho.
20:42:39 <kwame> ok
20:43:08 <nirik> #action nirik to clarify ticket and commits workflow for apprentices on wiki.
20:44:14 <kwame> nirik: so there's already a wiki for commits workflow for apprentices ?
20:44:14 <nirik> kwame: let me make some changes and you can review them to see if they explain things or make sense. ;)
20:44:41 <mdomsch_> how much do we care about s3-mirror logs?
20:45:00 <mdomsch_> I've started the module to collect, process, and analyze them in awstats
20:45:08 <mdomsch_> on log02
20:45:25 <mdomsch_> but they're kind of large, and growing at a couple hundred MB a week
20:45:31 <nirik> cool. I'm wondering if we shouldn't pick something better than awstats... I'm not all that impressed with it.
20:45:54 <mdomsch_> open to suggestions
20:45:56 <nirik> mdomsch_: even compressed?
20:45:59 <smooge> what else is there these days?
20:46:02 <mdomsch_> a perfect apprentice project
20:46:13 <smooge> I can look at other stats issues or help an apprentice
20:46:14 <kwame> lol
20:46:18 <mdomsch_> nirik, it's thousands of very small files, with a few hits per file
20:46:55 <nirik> weird. can we just cat them and compress that. ;)
20:47:57 <mdomsch_> nirik, yes...
20:47:58 <nirik> ideally we just want the data in some nice format and can toss the little files after a while.
20:48:13 <mdomsch_> I have S3 automatically nuking its copy of them after 60 days
20:48:24 <nirik> smooge: might want to see if ianweller's stats thing could take over awstats functions too...
20:48:27 <mdomsch_> can adjust as necessary
20:48:38 <smooge> ok will do so
20:48:46 <ianweller> it could
20:48:48 <mdomsch_> the most interesting bit of the stats is "what content is being requested"
20:48:57 <mdomsch_> e.g. Fedora X, EPEL Y
20:49:02 <mdomsch_> the IPs aren't
20:49:04 <nirik> yeah.
20:49:18 <mdomsch_> so lightweight, lighter than awstats, would be fine
20:49:29 * mdomsch_ has been using cat/awk/sort/grep/uniq
20:49:47 <nirik> yeah. It would be nice to see things like which fedorapeople repos are most popular, etc too...
20:50:05 <skvidal> oh
20:50:07 <skvidal> speaking of that
20:50:07 <nirik> but I guess we can see whats out and easy to deploy
20:50:17 <skvidal> do we want to move the /srv/repos dir on fedorapeople over to /projects
20:50:23 <skvidal> so people aren't quota-limited there?
20:50:53 <skvidal> it was something I didn't think of during the people migration
20:50:58 <skvidal> but after looking at it
20:51:01 <skvidal> I realized it kinda makes sense
20:52:03 <nirik> how big is that?
20:52:06 <nirik> might be nice...
20:52:09 * skvidal looks
20:52:21 <skvidal> 46G
20:52:24 <skvidal> s... biggish
20:53:07 * skvidal looks
20:53:17 <skvidal> most of it is spot
20:53:19 <skvidal> :)
20:53:28 <nirik> :)
20:53:34 <skvidal> seriously
20:53:42 <skvidal> 19G of the 46G is his repos
20:53:52 <skvidal> b/c he's captain bad-ass builder of chromium, etc
20:54:01 * nirik keeps pruning his calibre repo or it would grow pretty badly.
20:54:12 <skvidal> well this would help remove that need
20:54:28 <nirik> yeah, I'd be ok with moving it...
20:54:38 <nirik> a question tho: what do we want to back up there/
20:54:51 <skvidal> that is a good question
20:55:02 <skvidal> so wehave a fair amount of stuff on fedorapeople
20:55:08 <skvidal> b/c it can be very convenient
20:55:16 <nirik> yep. The first full backup hasn't finished since the move. ;(
20:55:24 <skvidal> nirik: I liked your idea of keeping a semi-hot backup somewhere
20:55:34 <skvidal> hmmm
20:55:39 <skvidal> so maybe an interesting test
20:55:45 <skvidal> use a junk box
20:55:54 <skvidal> and test rsync time to copy off all of people03
20:56:05 <skvidal> root+ssh+rsync
20:56:22 <skvidal> or use the vpn
20:57:02 * nirik isn't sure. wants to ponder on that some more.
20:57:59 <skvidal> nirik: it's a shame we don't have another place with enough disk at ibiblio or even at another i2 locaton
20:58:21 <nirik> yeah. it would be nice ot have a people spare thats somewhere where it could go live if needed.
20:58:52 <nirik> however, once we have cloud, we can move the dev/publictest stuff from osuosl and possibly do it there.
20:59:03 <skvidal> nod
20:59:05 <skvidal> I agree
20:59:22 <skvidal> I assume we've got nothing on an eta on anything cloud related or I would have already heard?
20:59:47 <mdomsch_> speaking of people and rsync, there was a request from someone who wanted to mirror fedorapeople.org
20:59:49 <mdomsch_> repos
20:59:56 <skvidal> mdomsch_: nirik replied iirc
21:00:01 <nirik> not really... need to finalize hardware and order... then wait for that to show up, then we need netapp stuff to happy to get us storage there...
21:00:12 <skvidal> mdomsch_: it's not a terrible idea to make it rsyncable - a little worried what that might mean load-wise
21:00:15 <nirik> mdomsch_: yeah, saw that. We could look at a rsync module I guess...
21:00:55 <nirik> ok, we are over time... shall we call it a meeting?
21:01:04 <smooge> okie dokie
21:01:51 <nirik> thanks for coming everyone. Continue on #fedora-admin and on the list!
21:01:53 <nirik> #endmeeting