19:59:45 <mmcgrath> #startmeeting Infrastructure
19:59:45 <zodbot> Meeting started Thu Sep 17 19:59:45 2009 UTC.  The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:59:45 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:59:51 <mmcgrath> #topic Who's here?
19:59:52 * ricky 
19:59:56 * nirik is in the back.
19:59:59 * SmootherFrOgZ is
20:00:02 * skvidal nods
20:00:07 * collier_s here
20:00:09 * a-k is here
20:01:25 <mmcgrath> Ok, doesn't look like we have any meeting tickets so we'll just get right into it
20:01:51 <mmcgrath> #topic FAS accounts and capcha
20:02:05 <mmcgrath> One thing I wanted to make known is we recently implemented a capcha in FAS.
20:02:08 <mmcgrath> by we I mean abadger1999
20:02:18 <mmcgrath> the problem was mostly caused from spammers we think.
20:02:39 <mmcgrath> STill more research to do but as many as 2 in every 3 account we have in FAS was created by a spammer.
20:02:46 <mmcgrath> or at least someone who didn't verify their email.
20:02:54 <mmcgrath> But!  no more :)
20:03:01 <ricky> :-)
20:03:06 <smooge> here
20:03:11 <mmcgrath> we re-consider freeing up those account names as well.  Something worth discussing.
20:03:20 <mmcgrath> they'd not have been in any groups or anything so it might be safe.
20:03:24 <mmcgrath> might not be though, more research required.
20:03:37 * ricky is all for freeing up accounts that have never been verified (ie never logged on)
20:03:48 <mdomsch> yo
20:03:53 <mmcgrath> mdomsch: hey
20:03:59 * ianweller rolls in for da lulz
20:04:08 <mmcgrath> anyone have any questions or comments on that?
20:04:28 <mdomsch> wow, that's crazy
20:04:52 <mmcgrath> mdomsch: indeed :)
20:04:55 <notting> so, does this screw up all our account/registration growth stats?
20:05:11 <mmcgrath> notting: it does as far as what we have done in the past but we can get new stats I think.
20:05:12 <abadger1999> mmcgrath: It'll be interesting if our initial registration drops
20:05:22 <mmcgrath> abadger1999: I'm almost positive it will.
20:05:23 <ricky> Did we ever really assume that # accounts = # of active people?
20:05:33 <mmcgrath> ricky: depends on 'we'
20:05:37 <abadger1999> It'll only drop significantly if the accounts are being created by a machine.
20:05:46 <mmcgrath> I really don't consider a contributor a contributor unless they have a fedorapeople account.
20:05:54 <ricky> I guess the question is if those specific numbers are usually quoted to the press or anything
20:06:13 <ricky> Same here - I get my counts from ls /home/fedora | wc -l on fedorapeople.org :-)
20:06:18 <mmcgrath> ricky: some are.  *but* generally we encourage people to use 'contributor' when siting those numbers.
20:06:23 <abadger1999> People that sign up but never verify email would still get through the captcha.
20:06:25 <mmcgrath> and we define a contributor as cla_done + one group
20:06:28 <mmcgrath> which is the fedorapeople count.
20:06:37 <mmcgrath> and that count will stay the same
20:07:31 <mmcgrath> So that's really all there is on that.
20:07:43 <mmcgrath> One thing I wanted to talk about was with affix
20:08:04 <mmcgrath> but I don't see him around right now so we'll wait.
20:08:09 <mmcgrath> he's looking for search engines for us.
20:08:24 <mmcgrath> Ok, so next topic
20:08:28 <mmcgrath> #topic PHX1 to PHX2 move
20:08:55 <mmcgrath> So smooge and I have been working to get the various network maps, inventory and other related directions to RH's IT department.
20:09:04 <mmcgrath> Much of that stuff is available in gobby.
20:09:21 <smooge> makes gobby gravy
20:09:25 <mmcgrath> smooge also put some network diagrams up http://smooge.fedorapeople.org/ideas/
20:09:43 <mmcgrath> It looks like we're going to be moving into 5 racks.
20:09:49 <mmcgrath> and spread properly so power isn't an issue.
20:10:00 <mmcgrath> we might expand into the 6th rack but I'm not counting on it, at least not for November.
20:10:05 <smooge> and look at smaller pdu's as we expand
20:10:13 * Oxf13 
20:10:15 <Oxf13> here
20:10:18 <mmcgrath> smooge: yeah, are you a fan of those 1U horizontally mounted PDUs?
20:10:45 <smooge> mmcgrath, more that I am a fan of PDU's that aren't treated like checkbooks :)
20:11:02 <smooge> most PDU's have too many plugs for what you can use with modern equipment
20:11:35 <mmcgrath> maybe we should start getting servers with 4PS's in them :)
20:11:43 <mmcgrath> then double the PDU's
20:11:49 <mmcgrath> oh wait, that's the same as just doubling the pdus
20:12:07 <mmcgrath> anywho :)
20:12:20 <mmcgrath> So I still have some outstanding questions, I'm sure smooge does too
20:12:27 <mmcgrath> but it looks like this is going to be about a 48 hour outage.
20:12:39 <mdomsch> mmcgrath, when?
20:12:40 <smooge> no we should look at DC racks
20:12:53 <mmcgrath> mdomsch: after F12 ships, before the end of November.
20:12:54 <smooge> mdomsch, depends on how much F12 slide there is
20:13:01 <mdomsch> smooge, DC still doesn't buy much
20:13:12 <mmcgrath> Oxf13: whats the latest "what are the odds of F12 slipping"?
20:13:21 <Oxf13> no change
20:13:25 <mmcgrath> k
20:13:30 <Oxf13> still looking good
20:13:36 <mmcgrath> So does anyone have any questions or concerns about this move?
20:13:40 <mdomsch> mmcgrath, can we migrate bapp1 to another datacenter before then?
20:13:44 <mmcgrath> it's going to be a lot of planning, then a week or two of hell.
20:13:51 <mmcgrath> mdomsch: yeah we can add that to our list.
20:14:12 <mmcgrath> I've moved stuff I had budgeted for the last quarter of this FY into a more major purchse just thi sweek.
20:14:21 * ricky will probably be in his own hell week of at that point :-(
20:14:23 <mmcgrath> we'll have some servers in PHX2 hopefully in a couple of weeks that we can use for that.
20:14:36 <mmcgrath> ricky: sucker, at least we don't get graded :)
20:14:55 <mmcgrath> mdomsch: so right nwo the list of things we want to 'pre-move' is db1, db2, bastion3 (new) app1 and bapp1.
20:14:57 <mdomsch> mmcgrath, sure we do
20:15:19 <mmcgrath> mdomsch: I always thought this was more pass fail :)
20:15:27 <smooge> ricky its just midterms... you should skip them and come on out
20:15:29 <mdomsch> failure is not an option
20:16:00 <smooge> failure is always an option.. in fact most of my academic career is based on that.
20:16:11 <smooge> but then again one should not take Zonker Harris as a role model
20:16:18 <notting> if we slip, does that mean we relocate fudcon to mesa and do the move then?
20:16:24 <ricky> Haha
20:17:29 <mmcgrath> notting: actually not the worst idea I've ever heard :)
20:17:30 <Oxf13> hey, it'll be warmer than Toronto
20:18:00 <skvidal> smooge: zipper, then?
20:18:01 <mmcgrath> One other thing we wanted to make people aware of is in PHX2 we're going to start seperating our network segments.
20:18:12 <mmcgrath> We're basically going to move to a 3 network system
20:18:14 <smooge> nah zipper is a real loser
20:18:21 <mmcgrath> 1) Buildsystem network
20:18:28 <mmcgrath> 2) Combined nfs / storage type network
20:18:31 <mmcgrath> 3) public network.
20:18:47 <mmcgrath> For those familiar with the environment, it's not too hard to figure out what will go where.
20:19:04 <ricky> Where does something not public like bapp1 go?
20:19:11 <mmcgrath> ricky: on the public network
20:19:15 <ricky> Oh, OK :-)
20:19:24 <abadger1999> smooge: You're just too old to appreciate him.
20:19:32 <mmcgrath> the public network isn't so much "public IP space" as it is just for all general network stuff like the 834 network is now in PHX.
20:19:48 <ricky> Ah, OK
20:19:50 <mmcgrath> Ok, anyone have any questions on this?  If not we'll move on.
20:19:55 <smooge> actually general network sounds better
20:20:10 <Oxf13> smooge: and sargent router?
20:20:15 <mmcgrath> Ok
20:20:17 <mdomsch> ricky, http://fpaste.org/oPMp/
20:20:20 <mmcgrath> #topic Favicon.ico
20:20:25 <mmcgrath> a-k: whats the poop on this?
20:20:59 <a-k> There are 4 favicons in puppet, but only 2 get sourced, and none of them are actually used by existing html, as far as I can tell so far
20:21:15 <mmcgrath> a-k: what ticket number was that?  I forget?
20:21:26 * a-k checks
20:21:53 <a-k> .ticket 1669
20:21:54 <zodbot> a-k: #1669 (the old favicon must die.) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1669
20:22:06 <mmcgrath> danke.
20:22:10 <mmcgrath> a-k: and you saw the note about smolts.org?
20:22:37 <a-k> Yes.  Did you want to have no favicon at all on smolts?
20:22:49 <mmcgrath> We don't have one so just leave it null.
20:22:56 <a-k> OK
20:23:00 <mmcgrath> a-k: anything else on that?
20:23:30 <mdomsch> a-k: favicon doesn't have to get referenced in our html
20:23:38 <a-k> I had a couple suggestions in the ticket and Ricky liked one of them.  If anybody has other ideas, let me know.
20:23:38 <mdomsch> it's automatically looked for by browsers
20:23:59 <a-k> mdomsch: yes.  In document root as favicon.ico.
20:24:09 <mmcgrath> a-k: alrighty, thanks.
20:24:11 <a-k> That was one of my suggestions.
20:24:20 <mmcgrath> #topic mod_evasive
20:24:40 <mmcgrath> So, some of you have probably seen make the mod_evasive module a bit mo' betta
20:24:58 <mmcgrath> This is largely because cvs1 has had some load issues recently
20:25:07 <mmcgrath> RH's internal search engine has been very agressive.
20:25:18 <mmcgrath> and google's crawler doesn't honor Crawl-Delay
20:25:34 <dgilmore> stupid bots
20:25:35 <mmcgrath> you can force google's crawler to crawl more slowly but it only lasts for 90 days.
20:25:39 <mmcgrath> So
20:25:49 <mmcgrath> I've decided to setup mod_evasive to be a bit more agressive about banning people.
20:26:16 <ricky> Does it have whitelists like denyhosts?  ;-)
20:26:20 <mmcgrath> It's actually pretty hard to predict how or when someone will get banned because the various apache children don't talk to eachother.
20:26:22 <mmcgrath> ricky: it does.
20:26:25 <ricky> Ah, cool
20:26:48 <mmcgrath> I still want the content on cvs.fedoraproject.org to be searchable or I'd have banned it altogether.
20:26:56 <mmcgrath> it might be worth looking at alternatives to viewvc though.
20:27:09 <mmcgrath> or at least figuring out why it's so freaking slow.
20:27:12 <ricky> s/viewvc/cvs/ :-D
20:27:13 <notting> migrate all projects away from cvs?
20:27:19 <ianweller> cough get rid of cvs cough
20:27:22 <smooge> not our problem
20:27:33 <ricky> One side effect of viewvc is that it creates a bunch of junk rcs* files in /tmp
20:27:39 <ricky> I wasn't able to figure out why though
20:27:56 <smooge> ouch
20:28:05 <mmcgrath> It's probably time for someone with time and courage to try starting the SCM Sig back up again.
20:29:02 <mmcgrath> Anywho, any questions about that?
20:29:42 <abadger1999> svn!
20:29:54 <abadger1999> Or should that be !svn ?
20:29:56 <abadger1999> :-)
20:30:04 <mmcgrath> moving on :)
20:30:07 <mmcgrath> #topic pgpool
20:30:08 <smooge> bitkeeper
20:30:21 <mmcgrath> welp, we tested pgpool in staging and it's been working on db2 for a while with no issues (and no connections)
20:30:33 <mmcgrath> mdomsch: after the meeting do you want to enable pgpool in production with mirrormanager?
20:30:40 <smooge> no connections?
20:30:52 <ricky> It could be good to start off with making the crawlers use pgpool
20:30:57 <mmcgrath> smooge: it's been deployed but the firewall's been active.
20:31:00 <ricky> Then the the mirrormanager and transifex apps
20:31:02 <mdomsch> mmcgrath, if you wish
20:31:12 <mdomsch> mmcgrath, I'll have to go to another meeting, so can't be there if it breaks
20:31:19 <mmcgrath> mdomsch: you won't have to do anything for that but it'd be good to have you around incase the sky falls.
20:31:23 <mmcgrath> the revert is simple enough and all.
20:31:34 <mdomsch> and I've got a minor MM update to push, to fix the can't-create-a-netblock bug
20:31:45 <mmcgrath> ah, well goodie.
20:31:51 <mdomsch> trying to to test that on stg
20:32:07 <mdomsch> but that's separate; so do your thing
20:32:21 <mmcgrath> mdomsch: will do.
20:32:24 <mdomsch> and remember it won't take effect for all the crawlers for a couple hours
20:32:33 <mmcgrath> <nod>
20:32:49 <mmcgrath> FWIW, I saw a measurable (not major) performance increase in my tests.
20:32:53 <mmcgrath> I can't explain that
20:32:56 <mmcgrath> but it was there.
20:33:17 <mmcgrath> perhaps the logging in phase of psql is slower then it could be :)
20:33:25 <mmcgrath> Ok, anyway.  With that
20:33:28 <mmcgrath> #topic Open Floor
20:33:34 <mmcgrath> Anyone have anything they'd like to discuss?
20:33:38 <ricky> Can we get a quick overview of how the ipv6 stuff went?
20:33:50 <mmcgrath> Sure.
20:33:52 <ricky> Are all the major problems ironed out?
20:33:54 <mmcgrath> I'll do the quick version
20:34:11 <mmcgrath> we started, some people couldn't reliably use TCP traffic (and probably other types of traffic)
20:34:22 <mmcgrath> the iptables rule on the list won't work for us because RHEL5 doesn't support it yet.
20:34:32 <mmcgrath> but by specifying a lower MTU, I've not heard a single complaint since.
20:34:38 <mmcgrath> and we're *STILL* waiting on the glue record AFAIK.
20:34:43 <mmcgrath> I'm not sure why that is though
20:34:47 <ricky> Cool - that's something that happened on the person's side and not our side?
20:34:59 <mdomsch> mmcgrath, right, no glue yet
20:35:11 <mmcgrath> ricky: well it could be done in either place actually.
20:35:17 <mmcgrath> but by doing it on our server, others don't have to do it.
20:35:38 <ricky> Ah, good
20:35:43 <mmcgrath> oh and mdomsch did some neat stuff as he mentioned on the list wrt MM and geo ip
20:36:01 <mmcgrath> mdomsch: I saw you prodding warthog9 again about geoip dns.  We starting to think that over again?
20:36:19 <mdomsch> mmcgrath, not really; i was just looking for how to implement a better backend for bind
20:36:30 <mdomsch> the zonefile for doing BGP lookups takes 1GB RAM
20:36:49 <mmcgrath> ah
20:36:49 <mdomsch> so I didn't do that; I custom-coded it in MM, takes 7MB
20:36:58 <smooge> wow thats big..
20:37:07 <mmcgrath> mdomsch: when you figured out how to do it in 7M did you do a dance?
20:37:07 <smooge> and then its small
20:37:08 * mmcgrath would have.
20:37:12 <mdomsch> oh yeah
20:37:18 <mdomsch> celeste was scared
20:37:22 <ricky> Heheh
20:37:26 <mmcgrath> hahah, it's like you've invented fire
20:37:34 <mmcgrath> well good work on taht too
20:37:40 <smooge> removed all the 0's?
20:37:41 <mdomsch> that's not in production yet
20:37:47 <mdomsch> but coming along
20:37:49 <mmcgrath> related to dns and geoip, I'd still like to see that as a TODO sometime.
20:37:51 <mmcgrath> but not urgent.
20:38:01 <mdomsch> ok, on that note
20:38:07 <mdomsch> we have an offer for hosting from China Unicom
20:38:14 <mmcgrath> we're probably getting to the point where we need to re-think our content distribution network.
20:38:15 <smooge> cool.
20:38:20 <mdomsch> they're looking for size estimates for servers
20:38:32 <smooge> can I go to do the buildout?
20:38:33 <mdomsch> what do we want to put there, and what server resources do we need?
20:38:59 <ricky> mmcgrath: wikipedia apparently uses powerdns for geodns
20:39:07 <mmcgrath> mdomsch: do you think they were thinking about providing servers as well?
20:39:08 <mdomsch> smooge, catch Ivory on IRC
20:39:16 <ricky> Maybe something to take a look at (and it has a bind-style zone file backend)
20:39:23 <mdomsch> mmcgrath, yes, I believe so.  2 asks:
20:39:26 <mmcgrath> ricky: yeah I think those are the big winners right now in that market.  pdns or bind + a patch
20:39:28 <mdomsch> 1) they set up and run a mirror
20:39:36 <mdomsch> 2) they give us dedicated hosting and Xen guests
20:40:01 <mmcgrath> So we wouldn't have to worry about the hardware at all with the xen guests?
20:40:04 <mmcgrath> I'd totally go for that
20:40:11 <mdomsch> that's the ask.  We'll see.
20:40:11 <mmcgrath> it's worked out well for us in BU
20:40:14 <smooge> I think we would like xen dom0 if possible . domU's is nice
20:40:17 <mmcgrath> yeah
20:40:31 <mdomsch> so if anyone sees Ivory on IRC, be nice
20:40:37 <mmcgrath> heheh
20:40:39 <smooge> np.
20:40:39 <mdomsch> and if anyone speaks Chinese, that would be a big plus!
20:40:45 <mdomsch> as I don't
20:40:49 <smooge> crap. neither do i
20:40:51 <mmcgrath> yeah, do we have any native chinese speakers in the house?
20:40:55 <Oxf13> I can barely order it off a menu
20:41:01 * ricky wishes he could read/write, but nope :-(
20:41:02 <mdomsch> his english is pretty good, but he's concerned about it.
20:41:11 <mmcgrath> mdomsch: I'd know how to ask Ivory to order spicy tofu.
20:41:16 <mmcgrath> and then tell him it was good :)
20:41:18 <smooge> his english is probably better than mine
20:41:32 <ricky> Something like "la dou fu" :-)
20:41:34 <mmcgrath> mdomsch: I'll get some specs to you about what would be good to have over there.
20:41:52 <mdomsch> on another unrelated note
20:41:57 <mdomsch> I'll be at LinuxCon all next week
20:42:18 <smooge> mdomsch, cool
20:42:27 <Oxf13> ditto
20:42:41 <smooge> where is that this year?
20:42:48 <Oxf13> Portland
20:43:23 <smooge> cool. and wet
20:43:48 <smooge> blackberries are really good right now I have been told
20:44:08 <smooge> on my note, xen13 is now RHEL-5.4 and should not reboot as often
20:44:22 <mmcgrath> smooge: cheers!
20:44:23 <smooge> bastion1 should also really really think its bastion1
20:44:34 <ricky> Nice :-)
20:45:03 <ricky> Did it go pretty smoothly, or were there any bumps along the way?
20:45:28 <ricky> rhel 5.4 domU went pretty smoothly
20:45:29 <smooge> xen13 needed /etc/grub hand edited.
20:45:39 <ricky> Ah, yow
20:45:49 <smooge> so I had to reboot twice.. as I forgot the first time
20:46:00 <smooge> and fas1 took 3 xen creates to start up
20:46:11 <mmcgrath> smooge: what needed to be altered in grub?
20:46:11 <smooge> and I didn't see why in the logs
20:46:27 <smooge> it was booting by default to an old kernel.
20:46:31 <ricky> Strange
20:46:37 <ricky> Ah, then I take my "yow" back :-)
20:46:38 <smooge> so when I updated it moved from booting from 1 to 2
20:46:50 <smooge> so I moved it to 0
20:46:58 <mmcgrath> ahh
20:47:19 <a-k> smooge: /etc/sysconfig/kernel ?
20:47:47 <ricky> It's set to yes on xen13
20:48:02 <mmcgrath> ricky: what about DEFAULTKERNEL
20:48:04 <ricky> Not sure why it wouldn't have gotten updated automatically.  Maybe we had manually edited things before?
20:48:05 <mmcgrath> is it kernel or kernel-xen
20:48:08 <ricky> That's set to kernel
20:48:15 <mmcgrath> that should probably get set to kernel-xen
20:48:22 <ricky> Ahh
20:48:30 <mmcgrath> Ok, anyone have anything else to discuss?
20:48:40 <mmcgrath> If not we'll close the meeting in 30
20:48:46 <smooge> ricky, I think it was manually edited in the past when trying to figure out which kernel worked
20:48:52 <smooge> nothing else from me.
20:49:08 <ricky> One more thing
20:49:20 <ricky> Should we email mirror-list-d about the i2 mirror soon?
20:49:32 <mdomsch> what i2 mirror?
20:49:32 <mmcgrath> ricky: good question
20:49:32 <ricky> The networking issues there seem to be resolved for the most part
20:49:41 <mdomsch> at rdu?
20:49:43 <mmcgrath> mdomsch: the RDU i2 mirror seems to be up and running well
20:49:44 <ricky> Yup
20:49:50 <mdomsch> ah, good to know
20:49:50 <abadger1999> mmcgrath: how's kvm testing going?
20:49:51 <mmcgrath> ricky: what was the last speed test?
20:49:51 <ricky> (And same question for sync1/sync2)
20:50:03 <mmcgrath> abadger1999: oh, good question.  I'll get to it in just a sec.
20:50:11 <mmcgrath> ricky: sync1/2 isn't ready for public consumption yet.
20:50:36 <mdomsch> so we don't need static routes for RDU i2 anymore?
20:50:46 <mmcgrath> mdomsch: that's my impression
20:50:49 <mmcgrath> ricky: what IP was it at?
20:50:59 <ricky> download-i2.fedora.redhat.com?
20:51:18 <ricky> I'm getting excellent speeds from osuosl1 right now
20:51:49 <ricky> (it showed >140 MB/s in the beginning and eventually returned to >7 MB/s which is still good)
20:51:50 <mmcgrath> mdomsch: ^^
20:51:58 <mmcgrath> Oh, and I wanted to talk about cloud stuff too
20:52:01 <mmcgrath> Ok, real quick.
20:52:13 <mmcgrath> abadger1999: the kvm stuff is going ok.  We were having horrible IO performance issues on app7
20:52:23 <ricky> Although starting a download from ibiblio1 at the same time caused the osuosl1 one to drop down to <500 KB/s :-(
20:52:24 <mmcgrath> after changing some drivers around it's better but still not as fast as some of the app servers.
20:52:27 <mmcgrath> await has been a big problem.
20:52:32 <ricky> So it's not very balanced it seems
20:52:45 <mmcgrath> abadger1999: but research continues :)
20:52:56 <mmcgrath> the good news is we're in no rush to switch to it so we can make it behave exactly was we want to.
20:53:02 <mmcgrath> SmootherFrOgZ: you around?
20:53:09 <mmcgrath> The cloud stuff has been going well, SmootherFrOgZ has been hard at work.
20:53:14 <SmootherFrOgZ> mmcgrath: yep
20:53:14 <mmcgrath> we keep running into lots of little issues.
20:53:26 <mmcgrath> a million paper cuts causing us to be unable to use cumulus.fedoraproject.org
20:53:41 <mmcgrath> SmootherFrOgZ: any luck with the nics?
20:54:17 <SmootherFrOgZ> we're currently debuging them to get them work properply
20:54:36 <SmootherFrOgZ> actyually nic show up with wrong interface name
20:54:45 <mmcgrath> SmootherFrOgZ: ok, well good work on it so far, I know this issue in particular has been quite irksome
20:54:59 <mmcgrath> Ok, anyone have anything else they'd like to discuss?
20:55:03 <mmcgrath> if not we'll close the meeting in 30.
20:56:40 <mmcgrath> Allright
20:56:42 <mmcgrath> #endmeeting