19:00:04 <nirik> #startmeeting Infrastructure (2013-01-24)
19:00:04 <zodbot> Meeting started Thu Jan 24 19:00:04 2013 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:04 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:04 <nirik> #meetingname infrastructure
19:00:04 <zodbot> The meeting name has been set to 'infrastructure'
19:00:04 <nirik> #topic welcome y'all
19:00:04 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
19:00:04 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
19:00:11 <nirik> who all is around for meeting?
19:00:15 <relrod> here
19:00:17 * skvidal is here
19:00:28 * puiterwijk is here (finally)
19:00:31 * lmacken 
19:00:37 * maayke is here
19:00:58 * pingou 
19:01:08 <dgilmore> hola
19:01:14 * abadger1999 here
19:01:57 <nirik> welcome everyone. ;)
19:02:04 <smooge> hgere
19:02:06 <nirik> #topic New folks introductions and Apprentice tasks.
19:02:15 <nirik> any new folks? or apprentice questions?
19:02:54 <nirik> going once...
19:03:08 <nirik> #topic Applications status / discussion
19:03:23 <nirik> tons of app stuff from fudcon. :) I tried to summarize some of it on the list.
19:03:30 <pingou> and more to discuss
19:03:35 <nirik> if I left out something someone worked on, please chime in on list.
19:03:44 <nirik> yep. so, fire away...
19:04:07 <puiterwijk> hey,I have something
19:04:21 <puiterwijk> since FUDCon, I have been busy with a new OpenID provider for FAS
19:04:23 <puiterwijk> FAS-OpenID
19:05:01 <puiterwijk> At this moment, it's stable and compatible with every OpenID consumer I have found so far, so if anyone wants to test, please talk to me :)
19:06:09 <nirik> puiterwijk: I'll try and get a openid.stg instance setup soon... we can ask more widespread testing then too.
19:06:15 <nirik> this _does_ have the groups handling?
19:06:24 <puiterwijk> yes, this DOES have groups extension
19:07:17 <nirik> ok.
19:07:45 <puiterwijk> this is also the basis for my 2factor authentication policy suggestion
19:08:09 <pingou> which basically brings the question of moving all our apps over to OpenID
19:08:23 <puiterwijk> in short: my idea (and Toshio's) was to make all of our applications sign on using OpenID
19:08:30 * nirik nods.
19:09:05 <puiterwijk> applications are able to require the user to use 2nd factor to access that app
19:09:14 <abadger1999> one ramification of this is that we'd be losing shared-cookie SSO.  But logging in to other apps would be trivial (wouldn't have to retype password as the openid server would have a session cookie to authenticate you to it)
19:09:24 <dgilmore> one thing that would be nice is yubikey support in fedorahosted
19:09:49 <nirik> and getting away from mod_auth_pg
19:09:51 <nirik> would be very nice.
19:10:10 <puiterwijk> yes, the user wouldn't notice that he would be sent to OpenID, as the provider returns him immediately if he sees the user is already logged on
19:10:49 <abadger1999> puiterwijk: We wouldn't have a "confirm you want to send this set of information to this other application" anymore?
19:10:54 <pingou> the current implementation uses a time-out after 15min, the new for the moment is valid for the time the browser is running
19:11:15 <puiterwijk> abadger1999: no, because we have a set of trusted roots (our apps mostly) for which the user won't be asked to confirm. I already have that sytem in place
19:11:22 <abadger1999> we'd probably need to bump that up although unlimited is too much.
19:11:32 <mjg59> /win 153
19:11:39 <dgilmore> mjg59: fail
19:11:40 <pingou> mjg59: oups :)
19:11:50 <relrod> and I thought 80 was a lot of windows. :(
19:11:56 <abadger1999> puiterwijk: excellent.  The plan is fFor other apps (like pypi), to still have that confirmation page, then?
19:11:59 <mjg59> Oh that's only halfway through.
19:12:05 <mjg59> (Sorry, laggy connection failure)
19:12:28 <puiterwijk> abadger1999: yes, so the skipping is only done for apps that in the openid's trusted_root list
19:12:30 * nirik is ok with this plan in general. I want to ponder on it a bit more and see if there's things I am not thinking of that would trip it up
19:12:31 <puiterwijk> (which we can configure)
19:12:31 <abadger1999> <nod>
19:12:41 <pingou> abadger1999: the 'white list' would be configure in the configuration file of the application (iiuc)
19:12:45 <abadger1999> puiterwijk: what about CLI?
19:13:14 <puiterwijk> abadger1999: that would be harder, but I guess we'd have to extend it
19:13:22 <pingou> puiterwijk: any idea how?
19:13:26 <abadger1999> Have to move to a model where we make a separate request to login followed by requests for the information we want?
19:13:30 <dgilmore> puiterwijk: a lot of our apps have clis
19:13:41 <pingou> and we should keep them
19:13:46 <abadger1999> (using the session cookie we get back to auth those followup requests)
19:13:49 <dgilmore> we have to keep them
19:13:56 <puiterwijk> abadger1999: yeah, something like that. maybe with a return value saying "you need 2nd fa"
19:15:53 <abadger1999> k.  I think we can make CLI work then... might just be slightly less efficient under the hood.
19:16:05 <nirik> so, we need mod_auth_openid packaged again (for hosted, etc).
19:16:15 <puiterwijk> nirik: yeah, already busy on that one
19:16:21 <pingou> abadger1999: or via fedora-python?
19:16:28 <nirik> puiterwijk: my last attempt should still be around if you want it.
19:16:38 <puiterwijk> nirik: okay, will check that one
19:16:41 <nirik> also, can wiki use openid?
19:16:53 <nirik> I guess we can move things as we get them sorted out...
19:16:58 <puiterwijk> nirik: http://www.mediawiki.org/wiki/Extension:OpenID
19:17:24 <puiterwijk> yeah, I guess we can do that. that way, if we need more OpenID extensions for some things, I can always add them
19:17:30 <skvidal> stupid question about openid
19:17:40 <skvidal> in the cases where we need to check for a single group
19:17:40 <puiterwijk> skvidal: yes?
19:17:55 <skvidal> is it possible to provide an openid url that ONLY works for that subset of users in that group?
19:18:11 <pingou> like skvidal would like to do with copr atm :)
19:18:23 <skvidal> pingou: indeed
19:18:27 <puiterwijk> skvidal: I guess I could make that yeah
19:18:38 <pingou> and we will want that for cla+1 in most of our apps
19:18:39 <skvidal> then the user doesn't see the openid bits for fas-only auth
19:18:44 <skvidal> it just works
19:18:55 <pingou> unless we did as we do now and provide openid only for cla+1
19:19:01 <puiterwijk> skvidal: but on the other hand: you get group information, so you might just say in your application to filter on that group
19:19:17 <skvidal> does that even make sense?
19:19:23 <puiterwijk> or I could develop an OpenID extension to say you require a special group
19:19:26 <skvidal> if there a nicer/better/more standards compliant way/ of doing that?
19:19:47 <skvidal> puiterwijk: I was thinking that a lot of apps might not be able to do that w/o a lot of local hacking for groups
19:19:48 <nirik> and with badges and such we may be moving to providing openid/alias to cla...
19:20:05 <skvidal> puiterwijk: since most apps just think to get a yes or no from openid
19:20:09 <pingou> true there nirik
19:20:26 <puiterwijk> skvidal: as I said: I could just specify a new extension to require a specific group
19:20:30 <skvidal> I was trying to avoid us having to write custom patches to apps we don't maintain (like the wiki, for example)
19:20:32 <puiterwijk> skvidal: wouldn't be too hard
19:20:34 <nirik> if we do that we also need to be ready for people to use openid for other things... like login to their local fedora machine. ;)
19:20:58 <pingou> \ó/
19:21:45 <puiterwijk> but maybe I could indeed define a new url for <group>.id.fp.o
19:22:08 <puiterwijk> or <group>.group.id.fp.o, we shoud discuss that sometime
19:22:12 <nirik> anyhow, it looks like we all kinda like the idea... needs more work and testing, but might end up saving us fas and application maint in the long run.
19:22:41 <abadger1999> <nod>
19:22:49 <skvidal> anyway - just a wacky thought
19:22:49 <skvidal> puiterwijk: just so we're clear - I think you're efforts here look pretty damn good - I wasn't trying to nitpick with that question
19:23:04 <puiterwijk> skvidal: it's no problem :)
19:23:31 <puiterwijk> I understand :)
19:23:33 <pingou> puiterwijk: eta on 0.1.0 ?
19:23:49 <skvidal> well okay then
19:23:52 <puiterwijk> pingou: I guess today? maybe tomorrow?
19:23:57 * threebean is late, but likes this idea a lot
19:24:04 <puiterwijk> the only thing still needed is theming, and that's just a bunch of HTML
19:24:05 <pingou> abadger1999: when is rolling the next fas release ?
19:24:21 * pingou proposes the FAN to puiterwijk
19:24:30 <abadger1999> pingou: I'm rolling a new python-fedora today.  so fas will probably get put off until tomorrow.
19:24:31 <pingou> Fedora Activity Night :p
19:24:39 <nirik> Smoother1rOgZ wasn't able to make it today but:
19:24:42 <nirik> Smoother1rOgZ> nirik: hey, I might be on my way home during infra meeting so here's my input for apps update: we should have by now a FAS release ready to update. toshio has howver some changes in stg which should bump fas to 0.8.17 as release ready-to-go.
19:24:44 <abadger1999> If only threebean would stop pushing new nad better code ;-)
19:24:54 <pingou> annoying threebean !
19:25:03 <nirik> heh.
19:25:29 <pingou> abadger1999: might be nice to roll another one after w/o the openid part
19:25:38 <pingou> Smoother1rOgZ: ^
19:26:33 <nirik> puiterwijk: another dumb question: right now the openid provider redirects to fas to authenticate right? Is there anything very fas specific in that?
19:27:02 <pingou> nirik: flask-fas under the hood
19:27:02 <puiterwijk> nirik: no, with about 10 lines changed, you can use any other auth system that supports groups I guess
19:27:13 <threebean> :)
19:27:28 <abadger1999> pingou: <nod>  -- once we have fas-openid deployed in production we can remove the openid stuff from fas itself.
19:27:36 <nirik> ok, cool.
19:27:39 <pingou> abadger1999: nice
19:27:53 <nirik> Just thinking thats making us less 'fas specific' if we decided to replace fas someday with something else.
19:28:12 <nirik> anyhow.
19:28:24 <nirik> any other apps news folks have? upcoming releases? cool things?
19:28:27 <abadger1999> flask seems pretty easy to code authentication methods for so should be easier than currently.
19:28:45 <abadger1999> python-fedora release, fas release coming up soonest.
19:28:53 <abadger1999> packagedb release after a waiting period.
19:28:54 <pingou> pkgdb release in a bit :)
19:29:11 <threebean> /packages has been flipping out since saturday; the fix should be ready to go out soon.  hopefully tomorrow?
19:29:17 <abadger1999> nirik: I think we said two weeks from the time I announced that features were going away.
19:29:21 <nirik> yep
19:29:25 <abadger1999> Cool.
19:29:36 <threebean> dgilmore and I are talking about hooking koji up to fedmsg, again hopefully tomorrow
19:29:45 <abadger1999> threebean: yeah -- Does that only need the pyhton-fedora update or something more aswell?
19:30:15 <threebean> abadger1999: just that update, but I also have to write an init file and do the release dance
19:30:28 <abadger1999> k
19:31:01 <nirik> awesome. :)
19:31:26 * nirik will move on then if nothing else pressing...
19:31:35 <nirik> #topic Sysadmin status / discussion
19:31:47 <nirik> I'm going to schedule some outages next week... tuesday and wed.
19:32:00 <nirik> going to do updates/mass reboots of class a / b machines.
19:32:15 <nirik> and also move our iscsi storage. (That will be so fun!)
19:32:30 <mdomsch> made good progress on MM at FUDCon.  Looking forward to taking more patches from abadger1999, pingou, and smooge
19:32:30 <mdomsch> :-)
19:32:33 <mdomsch> want to get that into staging soon and find out what else I broke
19:32:50 <pingou> mdomsch: awesome!
19:33:00 <smooge> I am looking at taking the syslog patch we did and add it to mirrormanager_crawler
19:33:25 <abadger1999> Yay!
19:33:39 <smooge> so I can track when servers are out of order so we can see it sooner.
19:34:07 <mdomsch> snapmirror to download-i2 running again
19:34:31 <mdomsch> smooge yea!
19:34:47 <nirik> yeah, I moved download-i2 to point back to download-ib01 for now. will change it back when the rdu ones are up to date.
19:35:15 <smooge> and galgoci said we needed to change the IPs anyway on the RDU boxes because he gave out the wrong ones
19:35:17 <mdomsch> nirik: care to comment on the new hotfix policy?
19:35:22 <mdomsch> I think it's a great idea
19:36:07 <skvidal> smooge: nice
19:36:15 <nirik> oh yeah...
19:36:18 <mdomsch> check in full files, then check in the patch
19:36:26 <mdomsch> so we can easily see the patches
19:36:28 <nirik> made a sop: http://infrastructure.fedoraproject.org/infra/docs/hotfix.txt
19:39:03 * nirik looks at other sysadmin stuff...
19:39:12 <puiterwijk> nirik: maybe discuss my suggestion?
19:39:33 <puiterwijk> the suggestion to make noc able to use run-puppet nowait on all servers they have access to?
19:39:36 * nirik is somewhat distracted by ongoing dns outage issue.
19:39:55 <nirik> yeah, I'm not opposed to that
19:39:56 <puiterwijk> as right now, I have to wait half an hour for puppet to run on some servers
19:40:23 <puiterwijk> (like last weekend when I was watching the servers, I had to wait a lot for memcached ...)
19:40:47 <abadger1999> +1
19:40:56 * threebean likes it too
19:40:59 <nirik> when we go to ansible we were looking at making it run only on changes.
19:41:13 <skvidal> nirik: hmmm - maybe a sudo entry on lockbox01
19:41:21 <dgilmore> im ok with that as long as sysadmin-noc has no access to any of the build/releng boxes
19:41:43 <skvidal> nm
19:41:43 <skvidal> sorry
19:41:53 <nirik> dgilmore: I don't think they do.
19:42:00 <nirik> can doublecheck.
19:42:02 <puiterwijk> nope, we don't
19:42:06 <puiterwijk> (we as in noc)
19:42:29 <puiterwijk> noc doesn't have access to those. but I suggested only sudo run-puppet nowait access to noc)
19:42:38 <nirik> ok, I'll make that sudoers change after the meeting. ;)
19:42:45 <puiterwijk> ok, thanks :)
19:42:47 <nirik> anything else sysadmin wise?
19:43:19 * threebean has one
19:43:21 <threebean> kinda
19:43:37 <threebean> misc re-raised the idea of sending nagios notifications out over fedmsg
19:43:39 <nirik> sure, fire away
19:43:53 <threebean> we discussed it over the summer and decided that it was unwise to attach nagios in any way to it.
19:44:34 <threebean> now that we've seen fedmsg working (most of the time), is anyone interested in talking about it again or trying it out?
19:44:53 <nirik> well, not sure...
19:45:05 <threebean> it would only make sense in a supplementary capacity.  notifs via email, sms, and fedmsg.
19:45:26 <nirik> I'd say we should look at doing that as part of a larger effort to rework nagios, which I intend to work on
19:45:38 <abadger1999> Hard deps would definitely still be out.  But we do have code to add fedmsg such that it doesn't fail if fedmsg is not available
19:45:56 * abadger1999 remembers to add that to the AppBestPractics page
19:46:00 <nirik> right, it would be informational, not us acting on it.
19:46:18 <nirik> and if we decided to act on it, we would query nagios directly to make sure that was the case.
19:46:25 * threebean nods
19:46:39 <skvidal> a couple of minor things
19:47:16 <nirik> trust, but verify. or... don't trust, and verify. ;)
19:47:42 <skvidal> threebean: we've still not seen fedmsg under load, have we?
19:47:56 <skvidal> ah
19:48:18 <threebean> skvidal: not real load.  I've done some dummy tests.
19:48:48 * Adran wanders in. [sorry about being late.]
19:49:16 <skvidal> okay
19:49:30 <nirik> ok, any other sysadmin stuff? shall we move on?
19:49:37 <nirik> #topic Private Cloud status update / discussion
19:49:39 <skvidal> I'm a LITTLE concerned about our nagios info being public
19:49:44 <nirik> any cloudy stuf?
19:49:53 <skvidal> since putting it on fedmsg would make it available to everyone
19:49:53 <skvidal> but not massively so
19:50:02 <nirik> skvidal: true. it's sysadmin now right?
19:50:09 <threebean> yeah.. I have to auth for https://admin.fedoraproject.org/nagios/cgi-bin//status.cgi?host=all
19:50:17 <skvidal> nirik: sysadmin-noc, isn't it?
19:50:28 <pingou> skvidal: sysadmin
19:50:32 <pingou> I can log in
19:50:32 <skvidal> okay
19:50:53 <misc> maybe having it public would permit to external people to say "we know, this is broken", instead of having people asking over on irc to the few one able to fix the issue :)
19:50:54 <skvidal> so it's not a high threshhold - ubt it 's a far cry from WAO
19:50:55 <nirik> yeah, and we send them to sysadmin. ;)
19:50:58 <skvidal> misc: no
19:51:03 <skvidal> misc: here's what will happen
19:51:07 <skvidal> they'll see it is a problem
19:51:09 <skvidal> and come tell us
19:51:19 <misc> ah, yeah, maybe too :)
19:51:25 <skvidal> it'll be like the most laggy paging system in the world
19:51:45 <threebean> ha :P
19:51:48 <threebean> distributed paging
19:51:53 <threebean> someone is *sure* to tell us :)
19:52:26 <nirik> :)
19:52:28 <skvidal> anyway
19:52:33 <relrod> skvidal: We let anyone idle in sysadmin-noc, doesn't that effectively make it available to anyone, anyway?
19:52:38 <skvidal> relrod: no
19:52:43 <skvidal> 1. we don;'t let ANYONE in
19:52:47 <relrod> #sysadmin-noc
19:52:48 <relrod> sorry
19:52:55 <skvidal> oh
19:52:59 <skvidal> and still no
19:53:04 <skvidal> there is social pressure in that channel
19:53:08 <skvidal> it exists in all places
19:53:13 <skvidal> there is no pressure to listen to fedmsg logs
19:53:15 <skvidal> anyway
19:53:19 <skvidal> do not let me concerns be a blocker
19:53:34 <skvidal> I just wanted to put them out there b/c when/if things go horribly wrong I enjoy saying I told you so ;)
19:53:39 <nirik> ha. ;)
19:53:54 <nirik> I think we can think about it in a new nagios world... it may or may not make sense then.
19:53:59 <skvidal> agreed
19:54:12 <threebean> cool
19:55:12 <nirik> anyhow, on to clouds. ;)
19:55:20 <skvidal> so the minor items I had were about ansible playbooks
19:55:34 <nirik> I setup tflink with an openstack login/account and he's playing with it to see how it will work for qa needs.
19:55:36 <skvidal> I got to test the host-reboot playbook in anger this week and it worked as intended
19:55:42 <skvidal> I see
19:55:45 * skvidal moves along
19:55:59 <nirik> sorry, we can go back too. ;)
19:56:08 <smooge> my next nagios tag will output to fedmsg the password hash of people who have problems logging in. That way they can grab their hash and see if that helps them remember it.
19:56:31 <smooge> sorry slow type
19:57:05 <pingou> ??
19:57:14 <skvidal> pingou: pretty sure smooge was kidding
19:57:22 <pingou> I surely hope so! :D
19:57:42 <nirik> so, we are adding more cloud stuff... we should keep working on finishing up stuff before we call it production.
19:57:43 <pingou> skvidal: I was like "wat" :)
19:57:56 <nirik> I think we can hash out much of whats left before too long.
19:58:40 <skvidal> everytime we add something persistent in the cloud I add it to ansible - provided that the provisioning process there remains the same - and we can migrate the data from the storage controller on the euca instance - we can move it about at will
19:59:20 <nirik> cool. I guess we need to hook up openstack to ansible as well...
20:00:03 <nirik> we can set them up with fasClient and two factor sudo anytime.
20:00:16 <nirik> anyhow, we will keep working on things there.
20:00:17 <skvidal> nirik: the only thing we need to connect openstack to ansible
20:00:18 <skvidal> afaict
20:00:23 <skvidal> is a functional, ssl'd ec2 api
20:00:26 <skvidal> that's it
20:00:38 <nirik> ok. I thought we had that, or was that not ssled?
20:00:45 <skvidal> not ssl'd
20:00:47 <skvidal> right
20:01:08 <nirik> ok
20:01:29 <nirik> ok, can poke at it more.
20:01:32 <nirik> #topic Fudcon recap
20:01:38 <nirik> it was great to meet up with everyone!
20:01:54 <nirik> anything people want to followup after fudcon ?
20:02:00 <abadger1999> Yeah -- and this year, it felt like we had a big healthy team :-)
20:02:04 <nirik> we didn't decide a FAD, but we have no shortage of choices.
20:02:31 <abadger1999> We started discussing app best practices and lmacken created this page => https://fedoraproject.org/wiki/Infrastructure/AppBestPractices
20:02:43 <abadger1999> We've been adding things we thought about to it since then.
20:02:55 <abadger1999> Feel free to peruse, add, and discuss.
20:03:07 <nirik> cool.
20:03:07 <threebean> :)
20:03:11 * pingou 'd love a app FAD
20:03:15 <lmacken> oo, we should probably mention kitchen in there :)
20:03:30 <abadger1999> Good idea.
20:04:34 <abadger1999> We never got around to graphining out all of our dependencies at fudcon but nirik, dgilmore, and I started thinking about it in terms of how it related to security.
20:04:36 <suehle> If anybody's here for the marketing team meeting, let's move over to #fedora-meeting-1 so these guys can finish.
20:04:54 <nirik> suehle: sorry, we could give you folks the room if you like.
20:04:58 <nirik> we are running slow today. ;(
20:05:13 <suehle> nirik, no problem at all!
20:05:24 <nirik> lets close out and move discussion over to #fedora-admin. ;)
20:05:27 <suehle> keep being productive and/or slow :)
20:05:29 <abadger1999> I'll send something to admin@ hopefully tomorrow summarizing that.
20:05:53 <skvidal> suehle: thppppt
20:06:44 <nirik> abadger1999: cool. I know we need to discuss more, we didn't come to too many conclusions.
20:06:53 <abadger1999> We moved some project hosting from fedorahosted to github.  Pull requests and the code review you can do with it has already been a big help.
20:06:53 <dgilmore> abadger1999: indeed its a complex entanglement
20:07:11 <suehle> skvidal, nirik was the one who said you were being slow, raspberry him!
20:07:32 * dgilmore refuses to use github
20:07:37 <abadger1999> https://github.com/fedora-infra
20:09:01 <nirik> would it be possible/easy to also hook hosted into them?
20:09:17 <abadger1999> pingou or threebean ^ ?
20:09:31 <pingou> we should be able to have a post-hook script
20:09:32 <abadger1999> nirik: just as a mirror of the github repo you mean?
20:09:38 <pingou> which mirrors to hosted
20:10:00 <pingou> someone also mention that we can ask github to be the frontend of projects hosted elsewhere
20:10:05 <pingou> it's not in the UI but can be asked
20:10:11 <pingou> that might be worth looking into
20:10:17 <nirik> yeah, that might be nice.
20:10:33 <nirik> then hosted gets all the changes and people can contribute there or github for those projects that want
20:10:49 <nirik> #topic Open Floor
20:10:51 <pingou> yup
20:10:59 <nirik> anything for open floor? we are over time... ;(
20:11:06 <abadger1999> <nod>  although we should think about if we want to do code review of all changes.
20:11:18 <abadger1999> If so, going through github would be better than going through feodrahosted.
20:11:53 <pingou> abadger1999: the UI would be gh and the git on hosted
20:12:53 <nirik> ok, anything else/
20:12:54 <nirik> ?
20:13:45 <nirik> ok, thanks for coming everyone!
20:13:47 <nirik> #endmeeting