19:00:01 <nirik> #startmeeting Infrastructure (2011-06-23)
19:00:01 <zodbot> Meeting started Thu Jun 23 19:00:01 2011 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:01 <nirik> #meetingname infrastructure
19:00:01 <zodbot> The meeting name has been set to 'infrastructure'
19:00:02 <nirik> #topic Robot Roll Call
19:00:02 <nirik> #chair goozbach smooge skvidal codeblock ricky nirik abadger1999
19:00:02 <zodbot> Current chairs: abadger1999 codeblock goozbach nirik ricky skvidal smooge
19:00:06 * skvidal is here, sorta
19:00:07 <nirik> Hello everyone.
19:00:12 * Flare183 is here
19:00:14 <goozbach> heddo!
19:00:24 * janfrode here
19:00:29 * CodeBlock here
19:00:38 * jpuellma is here (and new)
19:01:14 * abadger1999 here
19:01:14 <nirik> welcome everyone.
19:01:22 * nirik will wait another minute or so before starting.
19:01:44 * CodeBlock tries to figure out why his Rails app is going crazy, in that minute :)
19:02:03 * pingou here
19:02:08 * goozbach wonders if he can get down to the 5guys and back to his desk in that minute...
19:02:23 <nirik> #topic New folks introductions and apprentice tasks/feedback
19:02:31 <nirik> goozbach: did you make it? ;)
19:02:41 <nirik> anyhow, any new folks like to introduce themselves?
19:02:43 * Flare183 still feels "new"
19:02:59 <nirik> or any questions on apprentice tasks or easyfix tickets?
19:03:22 <Flare183> I'm wondering if I could work on ticket umm
19:03:31 <pingou> what about the captch for fas finally ?
19:03:40 <Flare183> 2762
19:03:50 <abadger1999> .ticket 2762
19:03:51 <zodbot> abadger1999: #2762 (Add logic to fasClient to only write authorized_keys when necessary) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2762
19:03:52 <nirik> .ticket 2762
19:03:55 <zodbot> nirik: #2762 (Add logic to fasClient to only write authorized_keys when necessary) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2762
19:04:06 <Flare183> For I could actually join the fi-Apprentice
19:04:10 <Flare183> +team
19:04:19 <abadger1999> pingou: Sorry about that -- it keeps sitting in my home dir, 3/4 merged.
19:04:20 <smooge> is here.. sorry
19:04:29 <nirik> Flare183: that would be a great one for the programming/applications side of things...
19:04:35 <pingou> abadger1999: no problem, just got curious out of a sudden :)
19:05:34 <nirik> Flare183: I can get you added to apprentice after the meeting if you want to look at sysadmin side of things as well.
19:05:36 <abadger1999> Flare183: I'd be happy to get you into the appropriate groups if you're interested in doing more programming type stuff.
19:05:46 <Flare183> nirik: Well I am a programmer at heart, but I also love to work with server's, and due to the fact that I worked with Ubuntu's Infra team. I would love to help with Fedora's Infra team.
19:05:51 <nirik> follow your interests. ;)
19:05:55 <Flare183> nirik: I would love that.
19:05:56 <abadger1999> <nod>
19:06:10 <abadger1999> Flare183: We do a lot of programming in fedora infrastructure.
19:06:11 <nirik> if you have the time/energy there's no reason you can't work on both.
19:06:18 <abadger1999> Flare183: So your help is definitely appreciated :-)
19:06:25 <nirik> but of course it's important not to burn out too fast. ;)
19:06:38 <Flare183> Well, whatever I can do, I'll do my best to do so :)
19:07:13 <Flare183> nirik: And Yes I would love to be added to the apprentice team.
19:07:25 <Flare183> hehe, I'll try not too ;)
19:07:29 <nirik> cool. Will do that right after the meeting.
19:07:39 <Flare183> Awesome
19:07:45 <nirik> any other apprentice thoughts/discussion? or new folks wanting to chime in.
19:07:56 <abadger1999> nirik: While we're on apprentices... LokoMurdok is looking for more work.
19:08:07 <abadger1999> nirik: But he's leaning more towards sysadmin side of things so
19:08:18 <abadger1999> It's time for me to pass him on to someone else to mentor/promote.
19:08:25 <nirik> cool. I think he said he might be interested in monitoring stuff... gomex has been working on those, but there are still some open ones.
19:08:32 <abadger1999> He's done two tickets... not sure what's next.
19:08:34 <nirik> I'd be happy to help out.
19:08:38 <abadger1999> Cool.
19:09:24 <nirik> ok, moving on then...
19:09:40 <nirik> #topic Upcoming Tasks/Items (nirik)
19:10:22 <nirik> I have a list... let me see whats coming up soon.
19:11:02 <nirik> here's a nice info dump:
19:11:07 <nirik> #info 2011-06-23 FPCA drop dead.
19:11:07 <nirik> #info 2011-06-24 f13 end of life.
19:11:07 <nirik> #info 2011-06-24 rdu mirror mapping due
19:11:07 <nirik> #info 2011-06-28 rdu mirror changeover
19:11:07 <nirik> #info 2011-06-29 send infra meeting agenda
19:11:08 <nirik> #info 2011-06-30 infra meeting
19:11:09 <nirik> #info 2011-07-01 mail fi-apprentice folks.
19:11:13 <nirik> #info 2011-07-01 blogs end of life.
19:11:15 <nirik> #info 2011-07-09 Remove inactive fi-apprentice people.
19:11:17 <nirik> #info 2011-07-11 - 14: smooge and nirik at phoenix
19:11:41 <nirik> I am also planning on migrating our puppet server early next week to a new instance.
19:12:06 <pingou> remaining puppet01 or the name will change as well ?
19:12:11 <nirik> any other upcoming plans or projects?
19:12:34 <nirik> pingou: it's going to become 'lockbox01'... but it will also have a puppet CNAME for a long time I suspect.
19:12:59 <pingou> thanks
19:13:22 <nirik> we didn't want to call it puppet02 or the like, because if we someday move to bcfg2 or something the name seems rather tied to a particular software.
19:14:02 <nirik> ok, if nothing else upcoming will move on.
19:14:02 <smooge> renaming of DNS project
19:14:08 <pingou> cname solves the problem, can be cahnged easily and remain self-explicit :)
19:14:08 <nirik> oh yeah.
19:14:09 <skvidal> we're going to rename dns?
19:14:14 <skvidal> zing! :)
19:14:24 <smooge> skvidal, I like to chain my cnames
19:14:31 <nirik> lets call it "YP" ? :)
19:14:40 <skvidal> nirik: I hear hesiod is all the new rage
19:14:48 <nirik> smooge: you mean the adding ns04/ns05 ? or the rdns thing?
19:15:04 <smooge> adding ns04/ns05 making the IPv6 of ns02 correct
19:15:22 <smooge> it is currently sitting on our various domain registrars
19:15:23 <nirik> yeah, is there any eta on when they might add that info?
19:15:26 <nirik> yeah. ;(
19:16:02 <nirik> well, I guess we wait and hope for the best.
19:16:41 <nirik> skvidal: how's those rdu machines going?
19:16:58 <skvidal> nirik: got 1 up and reinstalled, using grub and our normal ks instructions
19:17:02 <nirik> cool.
19:17:07 <skvidal> I'm going to test my instructions for redoing them
19:17:10 <skvidal> on 03
19:17:19 <skvidal> and if that works come back through on 01 with a mallet
19:17:22 <nirik> sounds good.
19:17:29 <skvidal> so I should be ready for tomorrow to request the right ports
19:17:34 <skvidal> to make it all magically work - or something
19:17:40 <smooge> whee
19:17:45 <nirik> ok, any other upcoming plans/projects people want to mention?
19:18:06 * nirik moves along... to...
19:18:10 <nirik> #topic FPCA deadline
19:18:11 <skvidal> anyone want to work on epylog modules and/or getting it updated in the el6
19:18:13 <skvidal> oh well
19:18:17 <nirik> skvidal: Sorry.
19:18:21 <smooge> skvidal, I will do so
19:18:28 <smooge> I was going to play with it this weekend
19:18:32 <smooge> or tomorrow
19:18:57 <abadger1999> FPCA: I was going to reap people from groups and from ownership of packages today, after the meeting.
19:19:12 <abadger1999> But I heard that nicholas Mailhot was going to sign the fpca tonight.
19:19:17 <skvidal> smooge: <shrug> okay - I was thinking it was  the perfect task for someone not @redhat.com if someone wanted it. but if you want it - that's cool
19:19:22 <abadger1999> So I've pushed back to tomorrow.
19:19:25 <smooge> oh ok
19:19:39 <smooge> I will help any volunteer who wants to sink their teeth in it then
19:20:11 <abadger1999> That will keep a large number of font packages (and the packages needed for packaging fonts) from switching owners.
19:20:31 <smooge> oh I was going to say if there was a list of font packages.. I would take them
19:20:33 <abadger1999> (switching owners since a design team person and a fonts sig person have volunteered to take over if he doesn't sign).
19:20:40 <smooge> I have been waiting to see if person would show up
19:20:43 <abadger1999> heh, and now an infra guy :-)
19:21:01 <nirik> cool.
19:21:12 <nirik> sounds reasonable to wait until later tonight...
19:21:17 <nirik> and then pull that trigger.
19:21:27 <abadger1999> I'll run the package stats one last time today to see where we're at but the last run looked reasonable: http://toshio.fedorapeople.org/fpca/
19:22:13 <abadger1999> There are a few packages on this list: http://toshio.fedorapeople.org/fpca/packages_losing_owner.txt
19:22:38 * nirik is happy. Someone snagged xcowsay.
19:22:53 <CodeBlock> :D
19:22:56 <abadger1999> that will cause some dep breakage but those can be figured out when rel-eng does the "orphaned packages getting blocked" run.
19:23:19 <abadger1999> nirik: thank limburgher :-)
19:23:48 <nirik> can do.
19:23:53 <nirik> ok, anything further on FPCA?
19:24:18 <abadger1999> I'll talk to notting and maybe publish an early version of the orphaned packages causing dep breakage as kind of a wrapup for devel list on the FPCA orphaning.
19:24:23 <abadger1999> That's all from me.
19:24:24 <smooge> no one took xcowsay?
19:24:30 <nirik> abadger1999: ok.
19:24:35 <smooge> I will take that one then
19:24:38 <nirik> smooge: no, someone did. ;) So I didn't have to.
19:24:48 <nirik> you can't ship a distro without xcowsay!
19:25:01 <smooge> its like not having fortune
19:25:10 <nirik> :)
19:25:14 <nirik> ok, moving on then...
19:25:27 <nirik> #topic RHEL6 plans
19:25:33 <nirik> thought I would add this as a topic.
19:25:43 <nirik> I setup a app02.stg box a few weeks ago...
19:25:50 <nirik> to test our apps on RHEL6.
19:26:06 <nirik> sadly, I made it 64bit... so I am going to repave it as 32bit and see if it works better that way.
19:26:35 <nirik> we are of course going to move things to rhel6 as we re-install them/migrate to new hardware, but are there any other places we could look at upgrading things?
19:27:03 <smooge> what do you mena?
19:27:23 <smooge> as in upgrading say a remote site?
19:27:39 <nirik> yeah. So, reinstall host, then bring guests back up on it...
19:27:56 <nirik> how much pain are we looking at there for migrating from 5/xen to 6/kvm? or should that all just work ok?
19:28:16 <skvidal> for the virt hosts or the guests?
19:28:26 <skvidal> hosts should be relatively painless
19:28:31 <nirik> well, either.
19:28:31 <skvidal> guests will be dependent on their applications
19:28:40 <nirik> right.
19:28:42 <skvidal> do we have all of our apps working on rhel6 pytonn, etc?
19:28:45 <skvidal> err python
19:28:53 <nirik> not yet... see above about app02.stg.
19:28:57 <skvidal> right
19:29:07 <nirik> community is not yet working on it, and the wiki is slow.
19:29:17 <nirik> but others are.
19:29:33 <skvidal> is MM
19:29:34 <skvidal> ?
19:29:46 <nirik> it's running, I don't know how to tell if it's working right. ;)
19:30:01 * skvidal is trying to think of the big ticket items users wil lcare about
19:30:05 <skvidal> also we have torrent
19:30:11 <skvidal> which is just going to be a giant clusterfrak
19:30:19 <nirik> yeah, which we need to figure out what software to go to. ;(
19:30:38 <smooge> I think any updates from xen->kvm is going to be a repave per site
19:30:54 <skvidal> smooge: yes
19:31:07 <nirik> if we have to redo the site, then we need to be able to migrate things around to avoid outages, IMHO.
19:31:08 <skvidal> smooge: but the repave should be smooth provided the guest apps work on el6
19:31:49 <nirik> anyhow, just bringing up the issue. We should look at where we can start scheduling and migrating more stuff over.
19:32:07 <skvidal> do we want to look up about our donated colo machines
19:32:10 <nirik> we have some xen replacement machines later this year and those will help move more, but we will have to do remote sites, etc sometime.
19:32:10 <smooge> yeah.. so we would repave at someplace simple and move up from there. [the ipv6 issues we are seeing at internetx1 with EL6]
19:32:11 <skvidal> and the thoughts on them?
19:32:18 <skvidal> like serverbeach## are a good bit long in the tooth
19:32:44 <nirik> skvidal: you mean hardware wise?
19:32:46 <skvidal> yes
19:32:57 <skvidal> that hw came on when mmcgrath started iirc
19:33:01 <skvidal> which is 4.5 yrs ago
19:33:11 * nirik hasn't looked. Yeah, we might want to look and see if we can get new hardware donated, or what we want to use older stuff for.
19:34:01 <nirik> not sure the politics there... but having more diversity is good if we can keep it.
19:34:24 <nirik> I'll have an app02.stg reinstalled in the next few days... and we can go from there.
19:34:41 <nirik> I think value01/02 might be another easy set to migrate to 6.
19:35:03 <skvidal> at this point they are smolt and zarafa, right?
19:35:08 <skvidal> b/c blogs is done
19:35:11 <nirik> and zodbot
19:35:21 <smooge> well mostly I think the politics on donated hardware is coming up to the colo what we get out of newer stuff and what they get for donating it to us.
19:35:41 <smooge> sorry slow typing
19:35:52 <nirik> hosted would be another one to get migrated....
19:36:15 <nirik> anyhow.
19:36:20 <nirik> just food for thought.
19:36:25 <skvidal> hosted is a good point
19:36:27 <skvidal> and one to bring up
19:36:35 <skvidal> do we have the complete set of features required in hosted?
19:36:41 <skvidal> ie: if we setup a new hosted on el6
19:36:45 <skvidal> how do we know everything is working?
19:36:49 <skvidal> feature-wise
19:37:04 <smooge> not sure. it would be something we probably would want ot "stage"
19:37:22 <nirik> I don't know. Testing of trac/mailman/the various scms would be the big thing.
19:37:45 <skvidal> nirik: 'the scms' is the hard part
19:37:49 <nirik> trac is likely to be fun... I don't know if it migrates db's or needs some kind of tweaking to transfer the old data.
19:38:04 <skvidal> I guess a plus of mailman is that it hasn't changed in like 10yrs
19:38:08 <nirik> yeah, one way we could possibly do it is:
19:38:19 <pingou> you should be able to move the sqlite db for trac rather easily iirc
19:38:43 <nirik> hosted03. Get all setup and working as far as we can tell. Get some projects to opt into transfering over to it. Have them use it, set a date and migrate the rest at that point...
19:39:03 <nirik> that way we get some diversity of people using it.
19:39:17 <nirik> pingou: it's a big version jump. ;(
19:39:48 <pingou> nirik: ah it's also updating trac itself
19:40:00 <nirik> we've also talked about splitting it out in the past, so this might be a time to do that as well.
19:40:02 <smooge> nirik, sounds good
19:40:13 <smooge> I wonder if we could go for a farm setup
19:40:22 <nirik> trac-0.10.5-3.el5 -> trac-0.12.1-4.el6
19:40:28 <pingou> :/
19:40:44 <skvidal> smooge: you mean like give up being IT-folks and start raising crops?
19:40:47 <nirik> anyhow, I think CodeBlock was going to work up some plan. ;)
19:41:07 <smooge> skvidal, well only after I make my first million from an IPO.
19:41:16 <CodeBlock> hi *reads scrollback*
19:41:35 <skvidal> smooge: I thought you did that already :)
19:41:36 <CodeBlock> sorry, trying to hack out an entire rails app before tomorrow evening... :)
19:41:36 <nirik> CodeBlock: hosted migration/upgrade/clustering again. ;)
19:41:54 <nirik> CodeBlock: no worries. it will come up again.
19:42:09 <smooge> no I meant having a cluster of hosteds with a proxy in front that would send [a-c] to hosted01, [d-g] to hosted02 etc
19:42:09 <nirik> #topic Meeting tagged tickets:
19:42:10 <nirik> https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=~Meeting&order=priority
19:42:33 <nirik> smooge: yeah
19:42:44 <nirik> any meeting tagged tickets folks want to hit on?
19:42:50 <smooge> one sec
19:43:12 <smooge> I am waiting for fedorahosted to respond :P
19:43:22 <nirik> yeah, I think we all hit it at once. ;)
19:43:41 <smooge> 2574 will be my main task tomorrow other than epylog
19:44:16 <nirik> ok. Yeah, I cleaned up the last publictest we had that was not re-installed recently...
19:44:20 <smooge> I need to figure out how to dump sysadmin groups from fas and then try my hand at a func module to see how long they have been seen
19:44:32 <nirik> I think we will get some people in sysadmin* dropping out after FPCA if they are not around.
19:45:21 <smooge> looked like 2 or 3
19:45:40 <nirik> yeah.
19:46:00 <nirik> smooge: oh, did you hear anything on virthost-comm01?
19:46:30 <nirik> or we can discuss that out of meeting.
19:47:04 <nirik> #topic Open Floor
19:47:23 * goozbach raises his hand
19:47:35 <nirik> go ahead... :)
19:47:38 <smooge> no I will ping sean and see if he got waylayed with another outage or not. If I don't see anything by 5pm we will reboot and see if that fixes it
19:47:41 <smooge> oops sorry
19:48:01 <nirik> smooge: sounds good.
19:48:02 <goozbach> I've looked at my time commitments I've made recently and decided I need to step away from fedora infra for a while
19:48:25 <goozbach> I'll still hang out, but not be able to do what little parts I was able to help with
19:48:37 <skvidal> goozbach: fair enough
19:48:46 <skvidal> goozbach: better to say it than to just fade away
19:48:53 <skvidal> goozbach: thanks
19:48:59 <goozbach> if/when I get my feet back under me I'll be back
19:49:09 <nirik> goozbach: we understand completely. :) Thanks for letting us know.
19:49:47 <goozbach> and maybe in 2-3 months I'll be able to show you something cool which will make my absence all the more worth it. :)
19:49:53 <goozbach> no prob
19:50:00 <skvidal> hah
19:50:01 <goozbach> anyway my job here is done
19:50:10 * StylusEater arrived late
19:50:11 <nirik> Cool. :) Good luck with whatever it is...
19:50:18 <skvidal> goozbach: yah "whatever" it is
19:50:24 <nirik> anyone else have anything for open floor?
19:50:32 <goozbach> I've trained nirik to do a proper meeting announcement and followup email
19:50:39 <goozbach> so I've worked myself out of a job :)
19:50:45 <nirik> hopefully I will not forget. ;)
19:50:53 <skvidal> so I worked on epylog last week out of anger
19:50:57 <goozbach> set a calendar notice :D
19:50:59 <skvidal> and we have log reports going out now twice a day
19:51:21 <StylusEater> nice
19:51:22 <skvidal> if you're in sysadmin-noc or above
19:51:26 <skvidal> and you want to see the reports
19:51:51 <skvidal> and start beating on: 1. new modules 2. systems to get them to shut the hell up 3. developers to fix their loud, annoying daemons
19:51:56 <skvidal> lemme know
19:52:12 <skvidal> alternatively
19:52:13 <smooge> goozbach, one item. If you have any sysadmin-X groups... you can leave the groups and we can put you in again when you get more time.
19:52:22 <skvidal> I could setup a sysadmin-log group
19:52:25 <skvidal> and just email members of that
19:52:45 <smooge> I will joing. I love logs..
19:52:49 <goozbach> smooge: I'm just in sysadmin and sysadmin-apprentace IIRC
19:53:27 <skvidal> also - I was wondering about the log frequency
19:53:29 <smooge> goozbach, ok, those are fine.
19:53:35 <nirik> skvidal: might be easier to keep track of as group... then people could remove themselves easily... then again a puppet commit it easy. Don't much care.
19:53:53 <smooge> skvidal, 12 hours is interesting. I need to figure out a way to have a page that keeps track of each log post
19:53:56 <nirik> 2x a day is working ok for now... perhaps we move it back to 1xday when it's more pruned?
19:53:57 <skvidal> I was considering */6 so we get them every 4 hours
19:54:05 <skvidal> smooge: keeps track of?
19:54:09 <skvidal> ah - okay
19:54:11 <smooge> well a summary page
19:54:12 <skvidal> so y'all want less..
19:54:13 <skvidal> gotcha
19:54:32 <skvidal> smooge: just need an index - but I didn't want to do that until I put the mod_auth_pgsql in place for authing
19:54:32 <smooge> so I can go look at yesterday, the week before and tuesdya
19:54:38 <smooge> ah got it
19:54:48 <skvidal> so I just disabled indexes in that dir
19:55:00 <goozbach> I'm out of FI-aprentace
19:55:03 <skvidal> goozbach: ok
19:55:46 <StylusEater> I've not been able to work on the tickets I assigned myself just yet but I still plan on it.
19:55:51 <goozbach> I'll stick around in sysadmin just for the emails and info
19:56:05 <nirik> skvidal: also, there was talk about setting up SEC? is that something we might want to do after epylog is mostly ok looking and then decrease the epylog freq and get sec to report on things more reailtime that are important?
19:56:17 <skvidal> sure
19:56:21 <nirik> or thats a down the road thing we still want to think about?
19:56:22 <skvidal> sec is already in fedora/epel
19:56:37 <skvidal> so it's easy to pop it on log02 and muck with it
19:56:52 <nirik> yeah, I've not used it. Will play with it this weekend perhaps.
19:57:01 <skvidal> it's not terribly complicated
19:57:09 <skvidal> essentially you feed logs to it via rsyslogd
19:57:16 <nirik> sounds like swatch, except I hope it's not as horrible.
19:57:17 <skvidal> it scans for regexes, quickly, and does stuff
19:57:27 <skvidal> it was reliable when I last used it
19:57:35 <skvidal> we used it + nsca to sound off on certain events
19:58:25 <nirik> cool.
19:58:39 <skvidal> it goes by the full name: simple event correlator
19:58:51 <skvidal> if you're googling looking for 'sec' can be difficult
19:58:53 <skvidal> but that's the pkg name
19:58:59 <nirik> ok, any other open floor items? or shall we move back to #fedora-admin / #fedora-noc?
19:59:23 <smooge> not from me
19:59:30 <smooge> thanks for eveyrone who could make it
19:59:39 <nirik> ok, thanks for coming everyone. Go forth and Infra!
19:59:43 <nirik> #endmeeting