19:00:00 <nirik> #startmeeting Infrastructure (2013-04-25)
19:00:00 <zodbot> Meeting started Thu Apr 25 19:00:00 2013 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:01 <nirik> #meetingname infrastructure
19:00:01 <nirik> #topic welcome y'all
19:00:01 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
19:00:01 <zodbot> The meeting name has been set to 'infrastructure'
19:00:01 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
19:00:08 * relrod here
19:00:18 * lmacken 
19:00:19 * ausmarton is here
19:00:29 * pingou here
19:00:30 <jerzyr> is here - rl9x4 ;)
19:00:33 * threebean is here
19:00:47 * abadger1999 here
19:00:49 * tflink is here
19:01:25 <nirik> cool. lets go ahead and start in...
19:01:31 <nirik> #topic New folks introductions and Apprentice tasks.
19:01:37 * cyberworm54 is here
19:01:39 <nirik> any new folks? or apprentices with questions or comments?
19:02:08 <cyberworm54> ......
19:02:19 <nirik> do note everyone should feel free to chime in with questions or comments as we go. ;)
19:02:39 <nirik> #topic Applications status / discussion
19:02:46 <nirik> so, new application news?
19:02:57 <nirik> #info fedocal and blockerbugs apps are heading to production
19:03:13 <nirik> #info pkgdb release to prod today already. ;)
19:03:22 <threebean> #info new fedoratagger in production
19:03:33 <pingou> something to test and provide feedback on: http://209.132.184.188/
19:03:44 <pingou> (lazy Sunday work)
19:04:04 <nirik> cool. nice looking there pingou!
19:04:31 <pingou> nirik: I will not hide where the inspiration from the theme comes from ;)
19:04:39 <pingou> s/from/for/
19:05:02 * skvidal is here
19:05:03 <nirik> threebean: if you have time, a email with the tagger changes to devel list or something would be good. Like abadger1999 did for the pkgdb one?
19:05:05 <skvidal> sorry I am late
19:05:11 * nirik wonders if we shouldn't send those to devel-announce
19:05:18 <nirik> welcome skvidal
19:05:33 <threebean> nirik: will do.. I'm just waiting on a little feedback from hughsie first.
19:05:43 <nirik> cool.
19:05:50 <threebean> datagrepper should be on its way to staging soon, we're just waiting on a package review.
19:05:54 <pingou> threebean: not sure we'll have it before tomorrow, it's 21:00 for him
19:05:55 <nirik> #info ask02 created and in service now.
19:06:31 <threebean> https://bugzilla.redhat.com/show_bug.cgi?id=955781
19:06:39 <nirik> I need to take ask01 out soon and reinstall it. Might wait for tomorrow on that tho
19:07:37 <nirik> pingou / tflink: so the openid support in fedocal/blockerbugs is waiting on a flask openid? or just time to recode the login stuff?
19:08:03 <pingou> nirik: mostly flask openid, a little bit of coding/ui changes for fedocal
19:08:15 <tflink> nirik: mostly coding changes and testing for blockerbugs
19:08:45 <tflink> I was waiting for things to stabilize after all the changes for infra before swapping out FAS
19:08:46 <nirik> ok, just trying to see where things were...
19:09:00 <nirik> python-flask-openid-1.0.1-1.el6 is in epel-testing
19:09:29 <kushalkhandelwal> Hi , I am Kushal Khandelwal an engineering undergraduate from India and I would like to be part of Infrastructure team.
19:09:40 <nirik> welcome kushalkhandelwal
19:09:58 <kushalkhandelwal> Thank you for the welcome nirik
19:10:21 <kushalkhandelwal> How should I get started , any guidance would be helpful
19:10:34 <nirik> kushalkhandelwal: were you more interested in sysadmin tasks? or application development?
19:11:00 <kushalkhandelwal> Sysadmin tasks
19:11:11 <kushalkhandelwal> and more interested in being a packager
19:11:36 <nirik> ok, see me after the meeting in #fedora-admin and we can add you to our sysadmin apprentice group...
19:11:43 <nirik> for packager, you want #fedora-devel
19:12:07 <nirik> anyone have other application news?
19:12:19 <kushalkhandelwal> Sure , would do that nirik
19:12:34 <pingou> threebean: also had a nice and productive week-end
19:13:18 <nirik> #topic Sysadmin status / discussion
19:13:25 <nirik> ok, on the sysadmin side... lets see.
19:13:58 <nirik> #info added hosted03/hosted04/collab03/collab04/fedocal01/fedocal02/blockerbugs01/blockerbugs02/ask02 in the last few days.
19:14:28 <threebean> :)
19:14:34 <jerzyr> :)
19:14:39 <nirik> #info outage for hosted move tomorrow.
19:14:42 <abadger1999> (re: python-flask-openid -- it's available for infra boxes.  But you'll want to make sure you apply the hotfix from puppet too)
19:14:47 <nirik> I'm going to move hosted02 -> hosted03 tomorrow.
19:15:08 <nirik> early next week, I am going to have an outage to move collab02->collab03
19:16:00 <nirik> we need to get monitoring added for all the new hosts. I might file some easyfix tickets for those.
19:16:37 <nirik> any other things we want to note or discuss on the sysadmin side?
19:17:12 * skvidal tries to remember
19:17:16 <abadger1999> bugzilla update?
19:17:23 <nirik> oh yeah.
19:17:25 <abadger1999> that's both sysadmin and app
19:17:30 <nirik> #info bugzilla will be updated soon to 4.4
19:17:50 <nirik> we have started to check things, but not everything is checked/done yet
19:18:06 <nirik> lmacken: any idea if bodhi is ok with new python-bugzilla?
19:18:10 * nirik checks the rest of the list.
19:18:51 <nirik> I think we are left with:
19:18:53 <nirik> FAS bugzilla sync
19:18:53 <nirik> PKGDB bugzilla sync
19:18:55 <tflink> IIRC, I had to change my code for modifying and updating bugs with the new python-bugzilla
19:18:59 <nirik> Bodhi
19:19:00 <abadger1999> pkgdb-sync definitely broken
19:19:03 <nirik> cnucnu
19:19:22 <skvidal> nirik: any movement on the backups - or is that waiting on netapp?
19:19:27 <abadger1999> Seems taht bugzilla server is just too slow
19:19:44 <nirik> abadger1999: might try the python-bugzilla list too... since you didn't get answers on irc.
19:19:52 <abadger1999> <nod>
19:20:12 <nirik> skvidal: yeah. So, I need to write up a mail on our current backups and what we have, then we need to wait for new netapp space to do something better.
19:20:53 <nirik> skvidal: I am thinking we can work something with rdiff-backup and then backup that to tape as desired. and also drop a bunch of things that aren't useful.
19:20:53 <abadger1999> My plan for today was to explore just gettingit working with the json url then report to python-bugzilla list the hack and see if they have any ideas for improving it.
19:21:04 <skvidal> nirik: yah agreed
19:21:08 <abadger1999> If it really is serverside slowness, I don't htink python-bugzilla will have any solutions
19:21:19 * Smoother1rOgZ is here
19:21:22 <skvidal> nirik: we've chatted about it before - but I think we can make our backups a lot simpler to deal with the common restore cases
19:21:25 <nirik> abadger1999: :( yeah, is the json one available in prod?
19:21:35 <nirik> skvidal: completely agreed.
19:21:41 <abadger1999> nirik: hmm... actually, I kinda doubt it.  I'll check.
19:21:51 <nirik> skvidal: also, we can plan in opt in backups for cloud instances, etc.
19:22:04 <skvidal> nirik: indeed
19:22:25 <nirik> #info will send email about backups and revamping them to the list soon.
19:23:23 <nirik> ok, anything else sysadmin wise?
19:23:59 <skvidal> nirik: where do you want to go next on ansible migration?
19:24:04 <skvidal> since we're out of freeze
19:24:07 <nirik> oh, on the bugzilla thing still... I couldn't find a hard date on when the update would happen, but it was mentioned before to be 26th/27th...
19:24:10 <skvidal> any particular simple wins you want?
19:24:45 <nirik> yeah, so I was trying to migrate enough tasks over to be able to run against an arm SOC and set it up as a releng or qa machine.
19:24:52 <pingou> The upgrade will take place on the evening of April 27th (EDT).
19:24:55 <pingou> for bz
19:24:56 <nirik> so, if we could finish that up to that point it would be excellent.
19:25:00 <pingou> from the bz-announce-list
19:25:12 <nirik> pingou: I saw mention of that a while back, but nothing recently confirming it...
19:25:24 <pingou> nirik: the email is from Monday
19:25:29 <nirik> ok, cool.
19:25:42 <nirik> so, we don't have much time. ;)
19:25:54 <tflink> yeah, that date was set a while ago but I thought it was pushed out until that email went out on monday
19:26:08 <pingou> https://www.redhat.com/archives/bugzilla-announce-list/2013-April/msg00002.html
19:26:17 <nirik> #info bugzilla upgrade scheduled for evening of the 27th
19:26:25 <pingou> also of interest for the apps dev, lists a changes for the RPC calls
19:27:08 <nirik> #topic Private Cloud status update / discussion
19:27:20 <nirik> so, skvidal has been fighting a fight with the cloud this week. ;(
19:27:31 <skvidal> and am still doing it
19:27:47 <abadger1999> pingou: thanks!
19:27:47 <skvidal> it's like I have a special job in life  - to fight private clouds
19:27:49 <nirik> yeah, such a rabbit hole. ;(
19:27:53 <skvidal> on the plus side
19:28:02 <skvidal> it is an acknowledged issue upstream
19:28:10 <skvidal> so we're not just 'special' in experiencing it
19:28:13 <nirik> #info ready to install on fed-cloud01/03 for a test cloudlet
19:28:18 <nirik> thats good at least
19:28:30 <skvidal> nirik: rdo on cloudlet?
19:28:42 <skvidal> nirik: seems like a good place to jump to, to me
19:28:43 <nirik> so, is rdo just the repos?
19:28:45 <nirik> or ?
19:28:51 <skvidal> nirik: ?
19:28:55 <skvidal> nirik: it's the pkgs
19:29:00 <skvidal> and the installer stuff
19:29:04 <nirik> I mean, does it use puppet and have it's own installer?
19:29:12 <skvidal> sadly, yes
19:29:15 <nirik> ok. ;(
19:29:27 <skvidal> aiui it is the packstack stuff
19:29:31 <nirik> but yeah, a grizzley one to test both rdo and grizzley out would be good.
19:29:38 <skvidal> nod
19:29:48 <nirik> and that would get us on a setup thats very 'standard' I would hope
19:30:11 <nirik> #info will look at using RDO to install test cloudlet
19:30:29 <nirik> skvidal: so, right now are those new nodes in the cloud? or out?
19:30:39 <skvidal> in
19:30:42 <skvidal> 05 and 07 are in
19:30:49 <skvidal> and RIGHT now they are working fine
19:30:55 <nirik> ok, cool.
19:30:56 <skvidal> the tricks are, of course, the dhcp lease timeouts
19:31:01 <skvidal> which I set down quite low this morning
19:31:05 <skvidal> means more dhcp noise
19:31:07 <nirik> so any idea how much capacity we have available? still lots?
19:31:10 <skvidal> but we  are LESS likely of hitting the issue
19:31:23 <skvidal> well - we added room for what? 48 cpus?
19:31:30 <skvidal> so. yah - lots
19:31:35 <nirik> also, we still need to add more cinders I guess.
19:31:36 <skvidal> and network-wise we have plenty of room
19:31:40 <tflink> skvidal: yeah, I noticed that on one of my nodes this morning :)
19:31:42 <skvidal> yes  - cinder is the next bit
19:31:50 <skvidal> tflink: oh yah - no option, sadly
19:32:07 <tflink> it explains why there were so many dhcp messages in syslog
19:32:13 <skvidal> yah
19:32:21 <skvidal> it has to do with dnsmasq-dhcp holding onto leases
19:32:32 <skvidal> despite having their config changed and being kill-HUP'd
19:32:40 <nirik> #info will look at adding cinder volumes from each of the compute nodes to increase available volume space.
19:32:46 <skvidal> they are holding onto lease timeouts from terminated instances
19:33:08 <skvidal> nirik: on my list of things today is to try out the f19 alpha image as an instance in the cloudlet
19:33:18 <nirik> yeah, that would be nice too for sure.
19:33:27 <nirik> #info skvidal working on f19a image for cloud
19:33:40 <skvidal> nirik: I talked to mattdm about it
19:33:48 <nirik> just use ks and make our own?
19:33:49 <skvidal> nothing dramatic left on it - so I'm going to dupe our f18 img
19:33:52 <skvidal> nirik: nod
19:33:57 <skvidal> well dupe our f18 ks
19:34:02 <nirik> ok
19:34:19 <skvidal> I'm still chasing potential solutions to rhe external ip routing issue
19:34:21 <skvidal> with openstack
19:34:35 <nirik> cool. Hopefully we can find a workaround.
19:34:36 <skvidal> anyway - off in the weeds
19:34:43 <nirik> any other cloudy news?
19:34:56 <skvidal> nothing leaps to mind
19:35:10 <nirik> #topic Upcoming Tasks/Items
19:35:34 <nirik> <stand by for info dump>
19:35:34 <nirik> #info 2013-04-26 - 19UTC hosted migration/switch hosted to openid
19:35:34 <nirik> #info 2013-04-27 - bugzilla 4.4 upgrade? (tenative)
19:35:34 <nirik> #info 2013-04-30 - 21UTC lists.fedoraproject.org migration.
19:35:34 <nirik> #info 2013-05-01 nag fi-apprentices
19:35:34 <nirik> #info 2013-05-08 drop inactive apprentices
19:35:34 <nirik> #info 2013-05-14 to 2013-05-28 BETA infrastructure freeze
19:35:34 <nirik> #info 2013-05-28 F19 beta release
19:35:40 <nirik> any other items folks would like to schedule or note?
19:35:56 <pingou> àny date for fedocal and tflink's app?
19:36:14 <nirik> I was thinking later today or tomorrow.
19:36:21 <nirik> the instances are all built
19:36:33 <nirik> we just need to move the modules into production and setup proxy stuff
19:36:38 <pingou> tomorrow will be fine for me, but early then (I have star-wars marathon planned in the evening)
19:36:45 <nirik> nice. ;)
19:36:54 <abadger1999> Note, I saw a howto this week on amking tg1 apps use openid to auth.  So it would be possible to migrate unported apps to fas-openid if we decide to establish a drop-dead date sometime in the future.
19:37:16 <nirik> abadger1999: cool. What apps are TG1 again?
19:37:29 <pingou> nirik: I had the same question ;)
19:37:45 <abadger1999> nirik: right now -- pkgdb, elections, bodhi, mirrormanager
19:37:47 <abadger1999> fas
19:37:49 <pingou> currently, FAS, pkgdb, elections, mirrormanager, bodhi
19:37:57 <abadger1999> (but fas won't use fas_openid for obvious reasons :-)
19:38:04 <nirik> why not? :)
19:38:07 * nirik kids
19:38:11 <abadger1999> :-)
19:38:24 <pingou> the most problematic in this list is MM
19:38:35 <nirik> so yeah, that could be valuable if that works easier than porting them.
19:38:38 <abadger1999> Everything except for mirrormanager has a planto migrate to a different framework in the coming year
19:38:44 <nirik> yeah.
19:38:56 <abadger1999> the ones going to flask have a provider written curtousey of puiterwijk
19:39:14 <nirik> ok
19:39:17 <abadger1999> the ones going to pyramid and tg2 may need to have a provider written for them.
19:39:27 <nirik> so, also coming up, but I have no dates for yet:
19:39:33 <pingou> packages is the last TG2 app, correct?
19:39:43 <nirik> #info new bladecenter should arrive sometime in the next few weeks hopefully.
19:39:46 <abadger1999> (If they need group info, they'll need more than a generic openid identity provider)
19:39:56 <nirik> #info new netapp space should arrive sometime in the next few weeks or so
19:40:02 <abadger1999> pingou: I think so.  threebean, lmacken-- do you guys know that?
19:40:13 <abadger1999> pingou: oh -- and maybe the undeployed badger app?  tahrir?
19:40:21 <abadger1999> *sigh* s/badger/badges/
19:40:22 <pingou> abadger1999: good point
19:41:09 * abadger1999 needs to write a badger app  Then everyone else can have my same difficulty
19:41:27 <cyberworm54> so...stupid question.....what happens if I restart libvirtd and vms are running? >.>
19:41:39 <cyberworm54> actually wrong channel
19:41:49 <cyberworm54> I meant to post that on #rhel
19:41:52 <nirik> :)
19:41:55 <cyberworm54> sorry
19:41:57 <tflink> abadger1999: an app to badger people? interesting :)
19:42:11 <nirik> #topic Open Floor
19:42:17 <nirik> anyone have items for open floor?
19:42:20 <abadger1999> tflink: Maybe it could tie into fedocal ;-)
19:42:35 <nirik> so, one thing I've not done much on lately...
19:43:06 <nirik> https://fedoraproject.org/wiki/Infrastructure_FedoraBugzilla <- do we want to bring that up on list(s) and start getting more input?
19:44:12 * nirik might try and poke at it more, we will need to gather requirements I think to move forward much
19:44:21 <tflink> didn't someone bring up the idea of bugs associated with multiple components?
19:44:28 <nirik> tflink: yeah.
19:44:50 <nirik> there's lots of ways to rework things if starting from the ground up.
19:44:59 <nirik> or things to try and accomodate anyhow.
19:45:24 <tflink> assuming that it's worth all
19:45:31 <tflink> the effort, anyways
19:45:43 <nirik> yeah.
19:45:49 <nirik> and it would be a lot of effort.
19:45:57 * pingou has the same feeling: is it worth it?
19:46:33 <nirik> well, it's really:
19:46:45 <tflink> nirik: did you ever run the scalability test you were talking about?
19:47:01 <nirik> fedmsg and openid / not having those
19:47:11 <nirik> tflink: the 15,000 component test?
19:47:14 <tflink> yeah
19:47:28 <nirik> yeah, I did.
19:47:41 <nirik> it makes any pages with that on it take like 12-14 seconds to load
19:48:03 <tflink> yikes
19:48:22 <nirik> so, another thought was to do a tracker per component, but then there needs to be a way to move things between and find the tracker you want
19:48:23 <skvidal> tflink: so we looked at other ticket systems
19:48:35 <skvidal> tflink: notably debian doesn't have a list/pull down of pkgs :)
19:48:59 <skvidal> if we did one tracker per component we'd need to write some brand new code to move things between components
19:49:11 <skvidal> the long and short is - dumping 15000 items in a list works not well
19:49:45 <nirik> I can look at making a better demo for roundup thats closer to what we want and see if I can get openid in it so anyone can test it.
19:50:15 <skvidal> nirik: so - unless we can crack the nut of the list of pkgs
19:50:18 <tflink> it would be nice to have a single account system as well
19:50:28 <skvidal> I am not sure we're better off in roundup
19:50:49 <nirik> tflink: yeah, the 2 things that even make us consider this: fedmsg and openid support. ;)
19:51:18 <nirik> skvidal: yeah, although it sounds like bugzilla is going to not use the list anymore either soon, or at least they were talking about not.
19:51:32 <skvidal> nirik: they expecting users to input the pkg name?
19:51:33 <tflink> nirik: good point, I wasn't thinking about openid and FAS as the same thing
19:51:36 <skvidal> I didn't see that discussion
19:52:04 <skvidal> tflink: well and if we were clever it might be somehting like $component.bugz.fedoraproject.org
19:52:31 <tflink> so it's the pages that load all 15,000 components that take a long time to load?
19:52:42 <nirik> it looks liek they are looking at a widget that searches name/description and shows some small number of matches.
19:52:52 <nirik> tflink: in bz? or roundup?
19:52:56 <tflink> either
19:53:06 <nirik> yeah, both are slow with 15k components.
19:53:12 <nirik> it's just a lot of data
19:53:31 <skvidal> nirik: gotcha
19:53:39 <tflink> I wonder how hard it would be to do a google-style dynamic dropdown based on text entry
19:53:46 <nirik> roundup is a lot faster for other pages... but that could be that it has less data, etc.
19:53:49 <tflink> but that sounds kinda like what nirik is talking about
19:54:00 <nirik> tflink: yeah, thats exactly what they were talking about doing for bz.
19:54:02 <skvidal> tflink: that's exactly what he';s saying
19:54:10 <nirik> I'm not sure if it's done/landing, but they were talking about it.
19:54:18 * skvidal has another solution for this
19:54:21 <skvidal> fewer packages
19:54:23 <skvidal> :)
19:54:33 <tflink> always an option :)
19:54:42 <nirik> yeah, lets drop all the ones that start with 'p' and that would save us a lot.
19:54:50 <tflink> would we have saner package:component mapping this way?
19:55:10 <nirik> tflink: another NTH thing would be subcomponents.
19:55:12 <tflink> I suppose it's not that bad, now that I think about it harder
19:55:27 <nirik> kernel -> filesystems -> btrfs
19:55:35 <tflink> just ran into some oddness when I was trying to map updates->bzcomponents
19:55:56 <nirik> anyhow, I think we are still not sure if this is worth it, but further discussion would be good I think.
19:56:07 <nirik> also, possibly more in person/higher BW discussion at flock or something.
19:56:09 <skvidal> nirik: I hear 'g' is a troublesome letter too
19:56:25 <pingou> skvidal: followed by k?
19:56:30 <skvidal> yes
19:56:34 <nirik> just leave 'x'
19:56:45 <rdieter> t*
19:57:02 <nirik> anyhow, any other open floor items? or shall we call it a meeting?
19:57:08 <abadger1999> l*
19:57:12 <smooge> it is a meeting
19:57:32 <pingou> congratulations smooge !
19:57:54 <nirik> thanks for coming everyone. :) Do continue over in #fedora-admin, #fedora-apps and #fedora-noc
19:57:58 <nirik> #endmeeting