18:00:01 <nirik> #startmeeting Infrastructure (2012-08-09)
18:00:01 <zodbot> Meeting started Thu Aug  9 18:00:01 2012 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:01 <nirik> #meetingname infrastructure
18:00:01 <zodbot> The meeting name has been set to 'infrastructure'
18:00:01 <nirik> #topic Greets!
18:00:01 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
18:00:01 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
18:00:07 * skvidal is here
18:00:09 * relrod here :)
18:00:34 * ctria almost back (i'll be off next week) and then i'm back
18:00:42 * nirik notes threebean is traveling and probibly won't be able to attend today
18:00:58 * puiterwijk is here
18:01:00 * pingou here
18:01:10 * abadger1999 here
18:01:44 * lmacken 
18:02:18 <nirik> ok, cool. Lets go ahead and get started...
18:02:27 <nirik> #topic New folks introductions and Apprentice tasks.
18:02:36 <nirik> If any new folks want to give a quick one line bio or any apprentices
18:02:36 <nirik> would like to ask general questions, they can do so in this part of
18:02:36 <nirik> the meeting. Don't be shy!
18:03:28 <ctria> i hope that you remember me. i'm in process of changing job and hopefully will have more time to spare this time. i officially will ask to join the apprentice group after next week
18:03:34 <ctria> that i'm returning from vacations
18:03:42 <nirik> welcome back ctria
18:04:20 <nirik> ok, if no one else new, will move on to the next topic...
18:04:23 <ctria> (but i could work on any tasks could be worked from a sandy beach next week)
18:04:24 <nirik> #topic Applications status / discussion
18:04:37 <nirik> ctria: relax and have a good vacation I'd say. ;)
18:04:53 <nirik> any application news from this last week or upcoming in the next?
18:05:00 <lmacken> I'm doing a new bodhi production release today, which includes fedmsg support :)
18:05:17 <nirik> there was a lot of fedmsg stuff landing in production as well recently.
18:05:18 * lmacken just has to make a few more tweaks so we don't flood the bus
18:05:34 <nirik> I messed up and thought the freeze was this tuesday, and it's not until next tuesday. ;)
18:05:36 <relrod> http://status.fedoraproject.org/ is live (ok, it's not really an app, it's a statically generated html file, but still)
18:05:36 <abadger1999> Next fas release is in stg right now.  Thanks to relrod and puiterwijk for putting it together :-)
18:05:44 <relrod> so
18:05:47 <relrod> please test that out
18:05:56 <relrod> and let puiterwijk and myself know if you have any suggestions
18:05:59 <nirik> relrod: it has it's own topic later in the meeting too.
18:06:04 <relrod> nirik: ah, oops
18:06:07 * relrod shuts up then :)
18:06:07 <lmacken> relrod, puiterwijk: nice work with that guys, it's awesome
18:06:11 <nirik> no worries.
18:06:36 <nirik> we have a #fedora-fedmsg channel with fedmsg bus live on irc now too.
18:07:03 <nirik> anything else that could/will land before freeze?
18:07:09 <lmacken> maybe busmon
18:07:17 * lmacken not sure of the state of our websockets stuff
18:07:27 <nirik> not very far along I'm afraid.
18:07:28 <lmacken> but threebean and I will be doing some hacking on it this weekend in person
18:07:42 <nirik> I still need to talk to RHIT about it... will see how that goes over.
18:08:07 <nirik> lmacken: cool.
18:08:25 <nirik> ok, moving along then...
18:08:32 <nirik> #topic Sysadmin status / discussion
18:08:48 <nirik> this section is for news from sysadmin folks... lets see...
18:08:57 * skvidal is just grumpy today
18:09:02 <skvidal> oh - you don't mean about the people
18:09:03 <nirik> We had some issues with the wiki upgrade last weekend...
18:09:03 <skvidal> sorry
18:09:05 <nirik> ha
18:09:21 <nirik> puiterwijk helped us out and tracked those down... so hopefully it's all happy now.
18:09:52 <nirik> also 2 wiki bug fixes... a rewrite where it was double encoding things and redirecting people wrong on login, and a fix to the rss feed to again show actual diffs.
18:10:43 <nirik> skvidal setup a releng01 with ansible...
18:11:22 <nirik> bunch of password resetting on mgmt interfaces, etc.
18:11:35 * relrod still is working on them, slowly but surely
18:11:37 <nirik> anything else folks would like to ask about or note?
18:11:44 <nirik> relrod: it's very appreciated.
18:11:57 <nirik> ok, moving on
18:12:04 <nirik> #topic Security FAD
18:12:10 <nirik> https://fedoraproject.org/wiki/FAD_Infrastructure_Security_2012
18:12:18 <nirik> I've done some more things to move this along...
18:12:27 <nirik> fixed up the wiki page, pinged people, etc.
18:12:40 <nirik> If you wish to attend, please add yourself to the attendees page.
18:12:48 <nirik> Also, we should try and finalize dates.
18:13:05 <nirik> I was thinking at this point that perhaps after f18 is released, but before holidays might work
18:13:31 <nirik> thoughts? rants? alterative suggestions?
18:13:48 <skvidal> just not during sept
18:13:49 <skvidal> kthx
18:14:02 <smooge> nirik, at that point is it close enough to FudCon to do it then?
18:14:02 * nirik looks at calendar
18:14:03 <pingou> skvidal: +1
18:14:04 <skvidal> oh and I'd like to personally request one other time be blacked out
18:14:10 <skvidal> the first week of october
18:14:27 <skvidal> smooge: we won't get anything done colocating with fudcon
18:14:29 <nirik> smooge: well, we could do that yeah... thats jan instead of nov...
18:14:33 <skvidal> we're too often dragged away
18:14:38 <abadger1999> <nod>
18:14:50 <abadger1999> We'd have to say "Week before fudcon" or some such
18:14:52 <pingou> skvidal: could be before or after fudcon no ?
18:14:59 <nirik> right, it would need to be before.
18:15:03 <nirik> come in early
18:15:04 <skvidal> pingou: that's fine
18:15:05 <skvidal> abadger1999: yah
18:15:07 <skvidal> I don
18:15:11 <smooge> actually I wasn't thinking of doing it then. I was worried it would get the "Thats awfully close to Fudcon" checkmark
18:15:11 <skvidal> t have an issue with that
18:15:11 <nirik> 2012-11-06 - f18 release
18:15:14 <skvidal> just not overlappying
18:15:20 <skvidal> and doing it AFTER fudcon is probably not wise
18:15:27 <skvidal> b/c everyone is strungout after
18:15:30 <skvidal> or rather
18:15:33 <skvidal> everyone I ever speak to is
18:15:37 <skvidal> maybe it's just me
18:15:57 <ctria> lol
18:15:58 <smooge> November sounds good
18:16:03 * abadger1999 agrees w/ skvidal
18:16:10 * nirik also
18:16:14 <smooge> and I agree w/ skvidal
18:16:15 <skvidal> abadger1999: you agree that i'm strung out after fudcon?
18:16:16 <nirik> so, its a narrow window there in nov...
18:16:17 <skvidal> nice
18:16:20 <abadger1999> hehe
18:16:36 <nirik> release is the 6th, and thanksgiving is 22nd.
18:16:40 <nirik> if we slip a week...
18:16:41 <smooge> skvidal, well if you wouldn't do those lines of cocaine off the backs of servers at fudcon it probably would not happen
18:16:45 <pingou> I can't convince you to make it for the FUDCon emea ? :D
18:17:02 <skvidal> pingou: I suspect spot/robyn would set us on fire
18:17:09 <smooge> very very much so
18:17:11 <nirik> pingou: I don't think we have budget for that. ;(
18:17:15 <pingou> skvidal: s/set us/set me/ ? :)
18:17:20 <skvidal> pingou: no, us
18:17:22 <skvidal> all of us
18:17:25 <pingou> ^^
18:17:32 * pingou burns
18:17:32 <skvidal> they have assassins everywhere
18:17:38 <smooge> kerosene is cheaper than tickets I think this is the usual answer
18:17:46 <spot> pingou: you get me and robyn. anyone else, you're welcome to cover out of your pocket. ;)
18:18:08 <pingou> spot: can we make exchange ?
18:18:14 <pingou> spot: ;-)
18:18:16 <skvidal> oooo - is pingou independently wealthy? I didn't know that
18:18:19 <spot> pingou: no exchanges, no refunds. ;)
18:18:21 <skvidal> excellent
18:18:30 <skvidal> pingou: can I have a grant?
18:18:37 <pingou> skvidal: if I was I would come to the security FAD ;-)
18:18:43 <skvidal> pingou: good point
18:18:45 <pingou> skvidal: of course, I'll pay you the beer when you arrive ;)
18:18:56 <skvidal> none for me
18:18:57 <nirik> right, so we could tenatively say: 2012-11-11 (arrive late afternoon/evening, get settled), 12, 13, 14 work on things, leave 2012-11-14?
18:19:05 <pingou> skvidal: then a very large coke
18:19:08 <nirik> but then we are having to do release from there if we slip. ;)
18:19:10 <halfline> ginger beer
18:19:38 <skvidal> halfline: only yankees drink such things ;)
18:20:02 <smooge> I thought aussies drank it
18:20:04 * nirik suggests ginger mead, but thats likely just me. ;)
18:20:15 <abadger1999> nirik: If I could find it, I would drink it ;-)
18:20:19 <skvidal> so... trailing off into the weeds
18:21:03 <abadger1999> 2012-11-11 works for me ... but I don't trust release not to slip.
18:21:09 <nirik> There's also the week after thanksgiving... 2012-11-25 to 28...
18:21:10 <smooge> I will put it that us hitting our deadline for F18 is not likely.. so I think the 16th is more likely the release date
18:21:35 <smooge> how about the 22-24 in Raleigh NC
18:21:47 <skvidal> what? why would any of us need to be around for the release. That's just silly. :)
18:21:59 <nirik> smooge: do they have turkey dinners there? :)
18:22:27 <skvidal> smooge: I will die a painful death if I am not around family over thanksgiving
18:22:30 <smooge> I don't know. I know that Westford has real turkeys in their parking lots then
18:23:03 <skvidal> nirik: moving on?
18:23:29 <nirik> problem with after thanksgiving is that many people are traveling then...
18:23:56 <skvidal> and flying on black friday?
18:24:01 <skvidal> that's a special kind of suffering
18:24:01 <smooge> oh sorry I was looking at the wrong month. I wasn't making it sit ontop of Thanksgiving
18:24:03 <abadger1999> Maybe we could plan on release sliping two+ weeks ;-)
18:24:13 <nirik> 26th-29th? come in 26th, leave 29th. 2 solid days of work
18:24:24 <pingou> 2012-11-11,12,13,14 might actually be hard for me, but we'll see
18:24:27 <skvidal> abadger1999: december in nc - how...... something... cold? uncomfortable but lacking any snow?
18:24:49 <smooge> bah. its not cold. its chilly but not cold
18:24:54 <abadger1999> As long as I'm not driving on glare ice I'm happy with cold weather.
18:25:11 * nirik doesn't care about cold as long as it doesn't cause flight delays. ;)
18:25:26 <abadger1999> true.
18:25:57 <nirik> or, for better budget spending: 26th-28th... come in 26th, leave night of 28th. Hack on things 27th and 28th days.
18:26:45 <smooge> hmm that works for me.
18:26:51 <nirik> votes? counter proposals?
18:27:01 <skvidal> 26th of november?
18:27:19 <skvidal> cool
18:27:55 <nirik> ok, if no objections I will take that to the list as a tenative?
18:28:03 <abadger1999> eh... 29th sounds better (just htinks about all the times late flights get rescheduled earlier in the day).
18:28:16 <abadger1999> but it still works for me.
18:28:22 <nirik> abadger1999: ok... we can see if we can swing that...
18:28:42 <abadger1999> I don't particularly care... just thinking of how much can get done.
18:28:44 <nirik> question: will RH be moved/moving to new tower then? I guess I can find out
18:28:50 <relrod> either way I'm going to miss a few days of school, so doesn't matter to me...just have to ask for the assignments beforehand and do them on flights. ;)
18:29:21 <nirik> so, if we are ok with that, I'll ask interested folks to go back and actually look up costs for those dates?
18:29:25 <nirik> and update the wiki page.
18:29:38 <nirik> so we can get a idea travel wise
18:30:23 <nirik> #agreed 2012-11-26 to 2012-11-29 for security fad in RDU
18:30:36 <nirik> #action everyone who would like to attend, please update wiki page.
18:30:49 <nirik> #info remote contributions also very welcome
18:31:03 <nirik> ok, anything else on this? or shall we move on?
18:31:22 <nirik> is there any people I didn't list that would be good to have to get the primary objective done?
18:31:39 * abadger1999 gets link to travel page for the meeting notes
18:31:48 <nirik> https://fedoraproject.org/wiki/FAD_Infrastructure_Security_2012/attendees
18:31:56 <nirik> we could just fold that into the fad page, no?
18:32:04 <abadger1999> https://fedoraproject.org/wiki/FAD_Infrastructure_Security_2012/attendees
18:32:06 <ctria> should remote register too?
18:32:24 <nirik> ctria: yeah, let us know if you are interested in helping, for sure... just note that you will be remote...
18:33:01 <nirik> ok then, moving along...
18:33:08 <nirik> #topic status.fedoraproject.org
18:33:20 * abadger1999 will fold attendees into the main page
18:33:48 <nirik> so, as many know, puiterwijk had an idea for a end user status application... so we could point people to it for outages instead of them having to check an irc /topic, etc
18:34:09 <nirik> we have such an application now thanks to puiterwijk and relrod. ;)
18:34:23 <nirik> #info status.fedoraproject.org runs in openshift currently.
18:34:37 <nirik> #info completely disconnected from our infrastructure
18:34:45 <ctria> great work
18:34:46 <nirik> #info updates are manually done by admins.
18:34:58 <puiterwijk> #info static webpage generated on push
18:35:05 <nirik> so, should we start using this? or do we have any questions or concerns about it?
18:35:21 <nirik> If we wanted to push this to some other site, how hard would that be?
18:35:27 <pingou> puiterwijk: stupid idea, rss ?
18:35:43 * nirik was also thinking rss might be nice if it was easy
18:35:58 <puiterwijk> pingou: maybe a good idea
18:36:09 <puiterwijk> #action puiterwijk will look into RSS with status.fp.o
18:36:10 <abadger1999> #info http://status.fedoraproject.org/
18:36:48 * abadger1999 needs to get better at reading URLs that aren't highlighted as links :-/
18:36:49 <nirik> also, there's one issue left: syncing ssh keys to grant access to folks who might need to update it...
18:37:25 * nirik has manually added a bunch of people for now, but we might setup a sync script on lockbox01 for it.
18:37:32 <ctria> could this get nagios alarms and automate results?
18:37:33 <relrod> lmacken also had an idea for auto refresh, and using something like https://github.com/lipka/piecon to show percentage of services up in the favicon, so people can see it at a glance
18:37:34 <nirik> or we could just leave it manual I suppose
18:37:34 * ctria runs
18:37:54 <nirik> ctria: we want it to NOT depend on any of our stuff being up/down/whatever.
18:37:56 <puiterwijk> I'm working on a python script to automate that from FAS
18:38:03 <puiterwijk> (the ssh key updating)
18:38:08 <nirik> puiterwijk: ok, cool.
18:38:20 <nirik> ctria: otherwise it could not update right if nagios was down, etc.
18:38:26 <abadger1999> Is it just checking by hitting a single url and seeing if that returns?
18:38:33 <nirik> also, nagios has a poor idea of what application is down.
18:38:44 <puiterwijk> abadger1999: no, it's manually updated by an admin
18:38:44 <nirik> abadger1999: what checking?
18:38:50 <nirik> it's a static page
18:38:51 <abadger1999> k
18:39:11 <abadger1999> ah -- I thought it was a periodically generated static page.  Misunderstood.
18:39:13 <nirik> so it could be misleading if there's an outage and no one updates it
18:39:19 <abadger1999> sorry.
18:39:36 <nirik> or if someone updates it and forgets to clear the outage (but we have a nagios check for that that should yell at us)
18:39:41 <ctria> another way is that it would get a heartbeat from services via fedmsg if it doesn't then service is off
18:40:15 <puiterwijk> ctria: and what if fedmsg is off? it won't know if that's a fedmsg failure or a connection failure
18:40:40 <ctria> common� you have to trust something� what if admins are off :)
18:40:58 <skvidal> ctria: fedmsg is a nice service - I do not think we have used it enough to get the point of trusting it
18:41:19 <skvidal> for notification of outages
18:41:25 <pingou> ctria: admins are off ???
18:41:37 <nirik> well, and also since this is not in phx2, we would need some way to get messages to it...
18:41:48 <nirik> pingou: some of us have always been a bit off. ;)
18:41:53 <skvidal> nirik: I'm sure mmcgrath would be happy to let us vpn in
18:41:55 <skvidal> ;)
18:42:04 <nirik> heh
18:42:32 <nirik> so, aside from enhancements, is everyone ok with us starting to use this and point people to it?
18:42:40 <pingou> +1
18:42:42 <ctria> +!
18:42:44 <nirik> or are their concerns/issues outstanding?
18:43:43 * nirik listens to crickets chirp
18:44:06 <puiterwijk> just one currently: that the ordering still needs to be fixed
18:44:21 <puiterwijk> (haven't had much time for that yet, but that's not crucial)
18:44:29 * skvidal will bite
18:44:32 <skvidal> what's wrong with the ordering?
18:44:36 <nirik> puiterwijk: oh, so how hard would it be to push this to another site if we wanted...
18:44:46 <puiterwijk> mizmo had advised to put major and minor outages on top (above the Ok services)
18:45:21 <puiterwijk> nirik: not too hard, only the management script would have to be adjusted to push somewhere else instead of git, and maybe the management script should regenerate the page
18:45:59 <nirik> puiterwijk: ok. I am just thinking of a case where openshift was down and all our stuff was up. We could change dns and point to one of our servers and push the page there so users wouldn't see problems.
18:46:21 <nirik> (as unlikely as that may be)
18:46:23 <ctria> nirik, if admins are up :)
18:46:40 <puiterwijk> nirik: that could be set up :)
18:47:09 <nirik> ok, so don't see any objections... I'd like to let it go for a few weeks before we officially announce it... so please do look for any issues before then.
18:47:33 <nirik> #agreed will start using status page, fix any issues, then announce in a few weeks....
18:47:54 <nirik> thanks for all the work on this puiterwijk and relrod. (and others too)
18:48:11 <puiterwijk> maybe we should pick a date, and if no showstoppers are still on that date, we will announce it?
18:48:22 <puiterwijk> to know when everything should be fixed?
18:48:32 <nirik> sure. how about the 22nd?
18:48:50 <puiterwijk> works for me
18:48:56 <relrod> nirik: 23rd? so we can ask at the meeting again?
18:48:59 <puiterwijk> or 23rd, at the meeting?
18:49:10 <nirik> sure. Thats fine too
18:49:13 * nirik makes a note
18:49:28 <nirik> #info status page will be announced on the 23rd if no stoppers are found
18:49:41 <nirik> #topic Upcoming Tasks/Items
18:49:44 <puiterwijk> oh, just so you know: from 19:00 UTC, nagios alerts will be sent when the services are triggered ;)
18:50:10 <nirik> hum?
18:50:19 <puiterwijk> still about status.fp.o
18:50:36 <nirik> oh, right...
18:50:43 <puiterwijk> if any services are in minor or major status (instead of "good"), nagios will complain about it so we won't forget to put it back when done
18:50:45 <nirik> so if you do any testing of the page, please put it in downtime
18:50:55 <nirik> so, upcoming items:
18:50:59 <nirik> #info 2012-08-14 to 2012-08-28 F18 Alpha Freeze
18:51:00 <nirik> #info 2012-08-15 hosted03-> hosted01/02 migration (tenative)
18:51:00 <nirik> #info 2012-08-28 F18 Alpha release.
18:51:00 <nirik> #info 2012-08-31 end of 2nd quarter
18:51:00 <nirik> #info 2012-09-01 nag fi-apprentices.
18:51:00 <nirik> #info 2012-09-08 drop inactive fi-apprentices.
18:51:02 <nirik> #info 2012-09-10 to 2012-09-24 Skvidal is out
18:51:04 <nirik> #info 2012-09-18 to 2012-10-02 F18 Beta Freeze
18:51:10 <nirik> #info 2012-09-27 to 2012-09-28 Smooge at Puppetconf
18:51:10 <nirik> #info 2012-10-02 F18 Beta release
18:51:12 <nirik> I'm waffling about the hosted move and when to do it...
18:51:21 <nirik> anything else folks want to note?
18:51:27 <nirik> or schedule, or plan?
18:51:41 <skvidal> do we have an ETA on cloud network?
18:51:57 <nirik> not that I have heard. ;( smooge: you heard anything?
18:52:09 <nirik> if we get internal nets we can start installing/setting things up
18:52:11 <smooge> no
18:52:18 <smooge> I am going to ping management
18:52:54 <nirik> ok. Might be good to drop a ticket on it, in case other folks can work on it for us?
18:53:21 <nirik> dunno.
18:54:06 <nirik> ok, anything more for upcoming? if not, moving on...
18:54:16 <nirik> #topic Open Floor
18:54:21 <nirik> anything for open floor?
18:54:22 <skvidal> I have something
18:54:29 <nirik> take it away...
18:54:35 <skvidal> there was a meeting last week in #fedora-buildsys
18:54:42 <skvidal> to discuss non-koji buildsys goals
18:54:50 <skvidal> we're not discussing getting rid of koji
18:54:54 <skvidal> nor replacing it
18:55:02 * relrod has to head out to bring the dog to the vet :( - will read the tail end of the logs when I return.
18:55:04 <skvidal> this is for builds of pkgs that we cannot do in koji for one reason or another
18:55:11 <nirik> ah yeah.
18:55:14 <skvidal> I wrote up the requirements here
18:55:15 <skvidal> https://fedoraproject.org/wiki/User:Skvidal/BuildSystem
18:55:23 <nirik> cool.
18:55:24 <skvidal> and we're working on ui/backend requirements here
18:55:35 <skvidal> https://fedoraproject.org/wiki/User:Skvidal/BuildSystem/Specifications
18:55:40 <nirik> this is something we hope to offer backed by the new private cloud, right?
18:55:45 <skvidal> feel free to comment/edit, whatever
18:55:50 <skvidal> nirik: it's definitely related :)
18:55:59 <nirik> yeah.
18:56:02 <skvidal> if there are any pieces folks want to look at - please let me know
18:56:09 <skvidal> or come by #fedora-buildsys
18:56:14 <skvidal> or post to buildsys@lists.fedoraproject.org
18:56:48 <skvidal> anyway that's all
18:56:56 * skvidal loves the sound of crickets
18:56:56 <nirik> cool.
18:57:16 <nirik> I think there's tons of nice stuff there... I look forward to breaking^Wusing it.
18:58:00 * jsmith can't wait to use it too...
18:58:35 <nirik> ok, anything else? or shall we close out the meeting now?
18:59:09 <nirik> ok, thanks for coming everyone. ;)
18:59:12 <nirik> #endmeeting