18:00:01 <nirik> #startmeeting Infrastructure (2012-04-26)
18:00:01 <zodbot> Meeting started Thu Apr 26 18:00:01 2012 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:01 <nirik> #meetingname infrastructure
18:00:01 <zodbot> The meeting name has been set to 'infrastructure'
18:00:01 <nirik> #topic Robot Roll Call
18:00:01 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
18:00:01 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
18:00:09 * abadger1999 here
18:00:18 <tibbs> Wow, that time already?
18:00:47 * mdomsch has divided attention, as usual
18:00:55 <codemaniac> \o all
18:01:08 * ianweller here
18:01:17 <nirik> morning everyone.
18:01:24 * skvidal is here
18:01:26 <skvidal> or something
18:02:33 <nirik> we are working on hosted migration, which didn't go all that smoothly... ;( So, probibly this will be a short meeting. ;)
18:03:34 * threebean is here
18:03:53 <nirik> #topic New folks introductions and Apprentice tasks.
18:03:54 <nirik> If any new folks want to give a quick one line bio or any apprentices
18:03:54 <nirik> would like to ask general questions, they can do so now. Anyone?
18:05:24 <nirik> ok... moving on then
18:05:36 <nirik> #topic two factor auth status
18:05:38 <nirik> any news here?
18:05:44 <skvidal> no
18:05:48 <skvidal> think about this week :)
18:06:08 <nirik> yeah, no kidding. ;)
18:06:14 <nirik> #topic Staging re-work status
18:06:26 <nirik> I am waiting on this until we have bapp01 retired...
18:06:31 <nirik> so, hopefully soonish.
18:06:43 <nirik> #topic Applications status / discussion
18:06:47 <nirik> any application news?
18:07:05 <nirik> abadger1999 / lmacken / threebean / pingou / CodeBlock
18:07:18 <lmacken> threebean and I are almost done with the proper tagger & packages deployment
18:07:18 <abadger1999> been working on smolt all week :-(
18:07:19 <threebean> lmacken pushed tagger into production
18:07:36 <abadger1999> I'm getting ready to toss it out the airlock.
18:07:39 <nirik> #info we moved gather-easyfix into production. Thanks pingou!
18:07:41 <pingou> working on hyperkitty and kittystore now :)
18:07:50 <pingou> thank you nirik for taking care of this :)
18:08:01 <nirik> #info tagger and packager deployment is almost done.
18:08:08 <lmacken> got quite a few emails today about people wanting to help fix bugs in bodhi, thanks to the easyfix stuff :)
18:08:15 <pingou> nice! :)
18:08:21 <nirik> info smolt is a pile.
18:08:27 <nirik> oops... forgot the #. ;)
18:08:55 <nirik> abadger1999: so where is it now? runs on app07, but poorly/
18:09:16 <abadger1999> nirik: I'm setting it up in pupet to run on app07
18:09:29 <abadger1999> but it's generating a lot of 0 length files for stats.
18:09:33 <abadger1999> otoh, I looked on bapp01
18:09:40 <abadger1999> and although there's less 0 length files...
18:09:43 <nirik> abadger1999: some of them always were 0 I thought?
18:09:51 <abadger1999> over half of hte files there are 0 length as well.
18:10:09 <nirik> yeah.
18:10:17 <abadger1999> so I'm not sure if this is new or if there's some timing issue or what.
18:10:43 <nirik> does the main stats.html one appear?
18:10:44 <abadger1999> at this point, I'd like to just push it out since we already want to get rid of smolt.
18:11:00 <nirik> yeah.
18:11:13 <nirik> push out whats working, call it good, work on announcing sunsut
18:11:15 <nirik> sunset even
18:11:18 <abadger1999> -rw-r--r-- 1 root root    0 Apr 26 18:01 stats.html
18:11:23 <nirik> :(
18:11:37 <abadger1999> vs
18:11:39 <abadger1999> -rw-r--r-- 1 root root 191843 Apr 25 00:18 stats.html
18:11:48 <nirik> yeah, thats the page many people hit...
18:11:51 <abadger1999> the latter is what's on bapp01; former is what's on app07
18:12:35 <nirik> so, perhaps we take the last bapp01 one, slap a "This data is from 2012-04-25 and will not be updated, please see the census project" and leave it static?
18:12:46 <abadger1999> We could do that.
18:12:55 <abadger1999> disable the cron job
18:13:10 <abadger1999> okay.  I'll do that.
18:14:04 <nirik> some people might not like that, but if we are sunsetting it anyhow...
18:14:41 <nirik> any objections? other ideas?
18:14:46 <abadger1999> What's the latest update on using census?
18:15:17 <nirik> I've not heard much recently.
18:16:05 <nirik> #action nirik to follow up on census progress.
18:16:13 <nirik> any other general application news?
18:16:30 * dgilmore is about to start on a composedb app
18:16:56 <nirik> dgilmore: cool. Are you doing that yourself? or working with others?
18:17:29 <nirik> anything we can do to assist at this point?
18:17:42 <dgilmore> nirik: just myself
18:18:02 <dgilmore> trying to work out whether to use Django TurboGears or something else
18:18:17 <dgilmore> im hoping that it will be ready for f18 dev
18:18:32 <dgilmore> development
18:18:37 <dgilmore> i.e when we branch
18:18:54 <nirik> cool. Might write up a plan/outline... might be some folks will want to help out with it.
18:19:11 <nirik> #info dgilmore working on composedb app.
18:20:03 <nirik> #topic Upcoming Tasks/Items
18:20:11 <nirik> #info 2012-04-29 to 2012-05-03 - Kevin out in the wilds of NM
18:20:12 <nirik> #info 2012-05-01 to 2012-05-15 - F17 Final Freeze.
18:20:12 <nirik> #info 2012-05-01 - nag fi-apprentices.
18:20:12 <nirik> #info 2011-05-03 - gitweb-cache removal day.
18:20:12 <nirik> #info 2012-05-09 - Check if puppet works on f17 yet.
18:20:12 <nirik> #info 2012-05-10 - drop inactive fi-apprentices
18:20:14 <nirik> #info 2012-05-15 - F17 release
18:20:21 <nirik> note that I will be out early next week...
18:20:36 <smooge> I can cover freeze tickets and such
18:20:37 <nirik> so if anyone needs anything from me before then, catch me today or friday.
18:21:21 <nirik> anything else upcoming folks want to schedule or note?
18:21:43 <nirik> oh, did I get the freeze wrong.
18:22:31 <nirik> yeah, it's a week out from that on the 08th
18:23:37 <nirik> #topic Open Floor
18:23:43 <nirik> anyone have anything for open floor?
18:24:11 * nirik will close out in a minute if nothing comes up
18:24:17 <mdomsch> MM brokenness?
18:24:24 <mdomsch> sorry I was out of pocket on Tuesday
18:24:59 <mdomsch> not sure why it chose that day to blow up and not before, if there really were a bunch of invalid values in host.bandwidth_int
18:24:59 <nirik> mdomsch: oh yeah.
18:25:07 <nirik> us either. ;(
18:25:13 <nirik> that did seem to fix it though...
18:25:14 <mdomsch> I have a patch in MM 1.4 to force that column NOT NULL
18:25:30 <nirik> we also hotfixed the script to error on those moving forward.
18:25:38 <nirik> so if it happens again we should be able to see it right away.
18:25:39 <mdomsch> I suspect there are similar kinds of failures latent
18:26:07 <skvidal> fun!
18:26:17 <nirik> on the plus side, I have a much better understanding now of how all the parts fit and work in mm. ;)
18:26:26 <mdomsch> well, that's something :-)
18:26:48 <mdomsch> thanks all for scrambling to fix it
18:26:51 <skvidal> and to that end - there sure seem to be a lot of moving parts in MM
18:27:19 <mdomsch> skvidal: yes...
18:27:24 <skvidal> I added a hotfix to MM so that if we start seeing tracebacks resulting in a BAD Request again
18:27:30 <skvidal> we'll at least get a piece of the traceback
18:28:21 <mdomsch> skvidal: it's the result of being our highest request volume application, and my first attempt at a sizeable program in python
18:28:30 <skvidal> I wasn't chastising you
18:28:45 <skvidal> I know exactly how things evolved - it's not something to be blamed for
18:29:00 <nirik> to be fair, all our apps logging is... not ideal. ;)
18:29:02 <skvidal> as nirik said - it took a fair amount of poking for us to figure out where it was broken
18:29:09 <mdomsch> sqlobject caching, python threading, conspired to make me split out the mirrorlist_server into its own cache, separate from the database
18:29:24 <mdomsch> and yes, almost no logging
18:29:44 <skvidal> I guess what I would like to see happen is this
18:29:55 <skvidal> could we force some bad data into the db
18:30:02 <skvidal> and see where things go sideways?
18:30:14 <skvidal> b/c as it is we were scrambling to figure out on which system things actually started breaking
18:30:23 <nirik> we could do that in stg?
18:30:29 <mdomsch> sure...
18:30:55 <nirik> in our copious spare time. ;)
18:30:57 <mdomsch> wouldn't need a whole lot of entries
18:31:25 <mdomsch> I've been doing my MM 1.4 work lately in a private instance that I hope to make front linux.dell.com for its few mirrors
18:31:39 <mdomsch> it's been handy for finding bugs, but it's not a full fuzzing approach
18:32:49 <mdomsch> and yes, exposing the tracebacks somehow, instead of eating them, would be useful for debugging
18:33:00 * nirik nods.
18:33:06 <nirik> ok, anything else, or shall we call it a meeting?
18:33:59 <mdomsch> call it
18:35:11 <smooge> hit the button nirik-frank
18:36:09 <nirik> #endmeeting