20:00:40 <mmcgrath> #startmeeting
20:00:40 <zodbot> Meeting started Thu Dec 10 20:00:40 2009 UTC.  The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:40 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:47 <mmcgrath> #topic Who's here?
20:00:47 * mmcgrath is
20:00:52 * dgilmore 
20:00:56 <mmcgrath> smooge: ping
20:01:08 <smooge> here
20:01:10 * a-k is
20:01:22 <smooge> putting clothes in dryer
20:01:50 <mmcgrath> So the outage is this weekend, I figured we should all go over the timelines and such
20:01:53 <mmcgrath> jwb: This might interest you
20:01:57 <mmcgrath> Oxf13: ^^
20:02:00 * skvidal is here
20:02:06 * jwb is semi here
20:02:16 <mmcgrath> So here's the basics
20:02:31 <mmcgrath> #action - Tonight we'll migrate db1, db2 and the vpn to PHX2
20:02:43 * nirik is sitting in the back in the cheap seats.
20:02:47 <a-k> mmcgrath: #topic?
20:02:50 <mmcgrath> #action - Sometime before Sat. Morning mail will start to go through bastion3 in PHX2
20:02:54 <mmcgrath> #topic Timeline
20:02:55 <smooge> #topic movement to PHX
20:03:00 <mmcgrath> #action - Tonight we'll migrate db1, db2 and the vpn to PHX2
20:03:01 <mmcgrath> #action - Sometime before Sat. Morning mail will start to go through bastion3 in PHX2
20:03:07 * mmcgrath doesn't get this bot stuff sometimes
20:03:16 <mmcgrath> The mail part is still in an unknown state to me
20:03:25 <mmcgrath> I've got a ticket open, I've pinged people about it's priority, I'm waiting to hear back.
20:03:49 <mmcgrath> #action - Smooge is working on some resolvers for us, those should be in place ASAP but isn't a blocker for the move.
20:03:54 <mmcgrath> it is a blocker to start turning things on though.
20:04:02 <dgilmore> mmcgrath: do we have the new bastion host up?
20:04:10 <mmcgrath> dgilmore: yes, bastion3.fedoraproject.org is up and running
20:04:26 <mmcgrath> Ok, this takes us to Friday morning.
20:04:42 <smooge> #action - Smooge will email out named scheme for PHX2 resolve domain
20:04:47 <mmcgrath> #action jwb/ someone in releng will tell me as soon as they are done writing to /mnt
20:05:03 <mmcgrath> #action at that time I'll mount the public mirrors as read only.
20:05:07 <jwb> mmcgrath, will likely be two sets of things there
20:05:12 <jwb> updates, and rawhide
20:05:18 <mmcgrath> jwb: totally fine, just let me know when it's done.
20:05:24 <jwb> k
20:05:28 <mmcgrath> Ok, this is where the coordination stuff happens.
20:05:35 <mmcgrath> I'll be flying to PHX at noon on Friday.
20:05:39 <mmcgrath> smooge: when do you leave?
20:06:16 <smooge> I leave at ~0900 tomorrow morning
20:06:32 <mmcgrath> that's localtime?
20:06:37 <dgilmore> smooge: MST
20:06:51 <mmcgrath> k
20:07:05 <mmcgrath> We both have access to PHX1 and PHX2 from Friday to Wed I believe
20:07:25 * SmootherFrOgZ is around
20:07:37 <smooge> sorry 0800 tomorrow mornign ariving at ~0900 MST
20:07:42 <mmcgrath> k
20:07:56 <mmcgrath> So I'll be heading to PHX1 in the afternoon to give everything a final look over
20:07:59 <mmcgrath> this is still Friday.
20:08:12 <smooge> mmcgrath, when do you get there
20:08:14 <mmcgrath> I'll probably shut down stuff like db1 since it's not being used at that point.
20:08:20 <smooge> nm.. see now
20:08:35 <mmcgrath> I get in at 3:30
20:08:48 <smooge> 03:30 or 15:30
20:08:51 <mmcgrath> 15
20:08:55 <skvidal> what's the plan for mail at this point?
20:09:05 * mdomsch is here finally
20:09:05 <mmcgrath> skvidal: I said that already :)
20:09:14 <skvidal> oh
20:09:17 <skvidal> it had just scrolled off
20:09:22 <skvidal> so I missed it on the divider :(
20:09:24 <skvidal> sorry
20:09:26 <mmcgrath> skvidal: I've got a ticket open and have pinged people about it's priority
20:09:29 <mmcgrath> no worries.
20:09:48 <mmcgrath> I'm hoping for that to get done tonight, if it's not done by then I'm going to need to hand that off to someone to keep bugging people until it gets done because I won't be around on Friday
20:09:56 <mmcgrath> skvidal: but the good news is we aren't the blockers on that AFAIK.
20:10:02 <mmcgrath> bastion3 is setup and listening and ready to forward mail.
20:10:14 <mmcgrath> Ok, so here's what is left
20:10:27 <mmcgrath> #action Friday night they'll be configuring the new netapp head unit in PHX2
20:10:41 <mmcgrath> Then sleep
20:10:59 <mmcgrath> #action Smooge and Mike will meet at PHX1 at 05:15 am
20:11:12 <mmcgrath> #action Disable nagios
20:11:34 <mmcgrath> #action make a final backup of sigul and a few other bits (this will probably be done the night before, just forgot to meniton it)
20:11:38 <mmcgrath> #action power everything down
20:11:50 <mmcgrath> #action movers will arrive at 05:45 to move
20:11:56 <jwb> action pray
20:11:58 <jwb> :)
20:12:00 <mmcgrath> The movers will actually be helping us unhook cables and such.
20:12:01 <mmcgrath> oh
20:12:16 <dgilmore> mmcgrath: buildsys we should disable friday night
20:12:17 <mmcgrath> #action I'll be taking a sigul drive, and smooge will have the other.
20:12:30 <jwb> ah, good
20:12:30 <mmcgrath> dgilmore: we can do it that morning if we want or the night before
20:12:35 <smooge> mmcgrath, make sure its got both as being bootable.
20:12:53 <mmcgrath> The movers themselves will actually be loading the entire racks onto the truck.
20:12:55 <smooge> sorry thats meant more as I need to ...
20:13:03 <smooge> mmcgrath, they will be?
20:13:15 <mmcgrath> yep, I have no idea how that'll work but I bet it's going to be wicked awesome.
20:13:18 <smooge> we aren't getting new racks in PHX or they will move them from one to another there?
20:13:21 <dgilmore> mmcgrath: lets do it at night  make sure all in progress builds complete
20:13:28 <mmcgrath> dgilmore: sure
20:13:46 <mmcgrath> #action meet the movers at PHX2 where they'll unload the racks
20:14:02 <mmcgrath> #action we'll then be moving servers from the old racks (which I guess are now sitting near the new racks) and re-racking everything.
20:14:10 * mmcgrath will bring bandaids
20:14:22 <mmcgrath> now this is where part of our story has changed a bit
20:14:32 <mmcgrath> we were under the impression contractors would be on site to re-wire everything
20:14:35 <mmcgrath> that's not true
20:14:42 <pjones> bandaids?  consider stopping by autozone and getting some work gloves.
20:15:07 <mmcgrath> Jonathan (the guy normally onsite at PHX2) is going to be pre-wiring some of the racks.
20:15:17 <mmcgrath> depending on how far he gets we may have a lot of cabling to do ourselves or none.
20:15:19 <smooge> pjones, I have had a rack go through teflon armored gloves.. they need blood to work
20:15:28 <mmcgrath> :)
20:15:41 <mdomsch> as electronics need smoke
20:15:42 <pjones> surely you mean kevlar?
20:15:57 <mmcgrath> so lets see where that leaves us...
20:16:04 <mmcgrath> #action start powering services back on
20:16:10 <mmcgrath> let me get the priority list...
20:16:11 <pjones> (and in any case; knives go through kevlar more easily than they go through canvas.  the high velocity of something like a bullet is where kevlar gets its strength from...)
20:16:46 <mmcgrath> #action get the app servers online first
20:16:49 <mmcgrath> #action then the buildsystem
20:17:00 <mmcgrath> I only put them in that order because the app servers should be very straight forward.
20:17:05 <dgilmore> do we have dhcp?
20:17:12 <mmcgrath> dgilmore: yeah, and we run it
20:17:19 <dgilmore> should we reconfigure all the builders first for the new networks
20:17:30 * mmcgrath is skipping the network config stuff for a later part of the meeting
20:17:34 <dgilmore> ok
20:17:37 <mmcgrath> because some of it is quite a bit different
20:17:40 <mmcgrath> but we'll get to that.
20:17:55 <mmcgrath> Now it's while we're re-racking this stuff that we will need help from people.
20:18:07 <mmcgrath> I've already worked on a checklist via CSI
20:18:14 <mmcgrath> basically I'm looking for people to re-certify these hosts
20:18:17 <mmcgrath> what does that mean?
20:18:21 <smooge> pjones, I meant kevlar.. but teflon would expect why I couldn't hold anything
20:18:29 <mmcgrath> verify remote management, power and cyclades works, as well as network.
20:18:37 <mmcgrath> this is stuff anyone in sysadmin-main can do.
20:19:01 <mmcgrath> Ok, so any questions about that?
20:19:09 <smooge> mmcgrath, pointer to checklist
20:19:22 <mmcgrath> smooge: I'm going to be adding that info to the ticket in a moment -
20:19:25 <mmcgrath> https://fedorahosted.org/fedora-infrastructure/ticket/1845
20:19:30 <smooge> ok cool
20:19:57 <mmcgrath> Ok, so as far as timeline goes and the physical work of things are there any questions?
20:20:10 <mmcgrath> Sometime after the move we'll be moving the db hosts back to their normal machines.
20:20:35 <mmcgrath> Ok, so the next topic is about what the PHX2 world will look like.
20:20:46 <mmcgrath> #topic PHX2 - what does it look like?
20:21:04 <mmcgrath> So in PHX2 we're going to have 3 networks and might add a 4th one later.
20:21:22 <mmcgrath> The 3 networks are as follows
20:21:32 <mmcgrath> 1) public network - 10.5.126
20:21:37 <mmcgrath> 2) build network - 10.5.125
20:21:43 <mmcgrath> 2) storage network - 10.5.127
20:21:56 <mmcgrath> the majority of our services will end up on the buidl network
20:21:59 <mmcgrath> that should be 3) storage :)
20:22:12 <mmcgrath> and the majority of our services will end up on the public network, not build.
20:22:15 * mmcgrath needs more sleep.
20:22:19 <mmcgrath> the build network will be mostly for releng
20:22:29 <mmcgrath> and the storage network will be for nfs traffic and possibly backups.
20:22:46 <mmcgrath> I'm going to be doing tests to find out where backups best work and have the least impact on other things.
20:23:12 <mmcgrath> This also means that several of our hosts will now have multiple IP addresses.
20:23:33 <mmcgrath> for example relengX will be on the build and storage network.
20:23:38 <mmcgrath> we won't be routing them.
20:23:46 <mmcgrath> Anyone have any questions on this?
20:23:49 <sandorlev> hello
20:24:28 <mmcgrath> Ok.  so that's really it.
20:24:46 <mmcgrath> I'm working on a spreadsheet right now that I'd appreciate some second looks at.
20:24:59 <mdomsch> I presume we'll only have DNS A records for the inbound service IPs on those dual-homed boxes
20:25:12 <mmcgrath> mdomsch: I believe so.
20:25:18 <mdomsch> e.g. build1's IP on network 3 won't have an A record
20:25:20 <mmcgrath> we'll be running our own DNS servers in PHX2, smooge is working on that now.
20:25:49 <mmcgrath> mdomsch: correct but it will probably have a reverse.
20:25:53 <mmcgrath> smooge: what do you think?
20:25:56 <smooge> i am going to have two boxes eventually but at the moment just one.. ns001.phx2.fedoraproject.org
20:26:12 <mdomsch> mmcgrath, makes sense
20:26:25 <mdomsch> reverse is good to keep track of allocated addresses at least
20:27:01 <smooge> the hosts will just be on the phx2 subdomain which will be a 'hidden' domain (not on public dns since it will contain 192 and 10.x ips)
20:27:47 <mmcgrath> smooge: also on the meeting yesterday I think it was requested that we RH to still be able to do lookups and reverse lookups so we might be stuck doing fedora.phx2.redhat.com
20:27:52 <mmcgrath> unless there's an easy way to just mask it for them?
20:27:55 * mmcgrath hasn't thought about that much.
20:27:56 <smooge> I am going to keep it simple in version one because we don't ahve a lot of time.. but dual homed would be later
20:28:33 <smooge> mmcgrath, I think in either case they will be able to do zone transfers from our servers
20:28:37 <mmcgrath> smooge: wise idea
20:28:44 <mmcgrath> Ok, so any questions about this?
20:28:50 <smooge> so we could call it fedoraproject.int and it should be good
20:29:12 <mmcgrath> at present there is no firewall preventing access to or from any of the 3 networks I've listed.
20:29:17 <mmcgrath> we'll be doing that later, just ran out of time.
20:29:20 <nb> i would suggest not using .int because .int is a real TLD
20:29:30 <smooge> nb it was a joke sorry
20:29:36 <nb> smooge, oh ok :)
20:29:45 <mmcgrath> Ok, so anyone else have any questions about this?
20:29:51 <mmcgrath> I really have no estimate for how long this will take.
20:30:08 <mmcgrath> I've never moved this many machines before and I have no idea how much is already done and how many people we'll have helping
20:30:19 <smooge> mmcgrath, you are leaving on Tuesday correct?
20:30:41 <smooge> smooge will be leaving on Wednesday so we wont lose coverage afterwords
20:30:52 <smooge> and we have problems like "Oh crap that disk drive didn't make it."
20:30:58 <mmcgrath> smooge: I'm not sure actually I need to look at my trip info.
20:31:28 <smooge> in case of real emergency I will just get back in the car and drive back til we are stable.
20:31:46 <mmcgrath> so that's really all I have to discuss
20:31:51 <mmcgrath> anyone else have any questions or comments?
20:31:59 <mmcgrath> if not I'll get back to my IP list and let smooge get back to DNS stuff.
20:32:14 <nb> anything non-main people can do to help?
20:32:20 <mmcgrath> #topic open floor
20:32:33 <mmcgrath> nb: I'm hoping the non-main people will help with logistics and troubleshooting
20:32:38 <nb> ok
20:32:47 <mmcgrath> when the services start coming back online and interacting with eachother I suspect there will be lots of little bugs
20:32:56 <mmcgrath> like say, app2's network is behaving poorly
20:33:18 <mmcgrath> nb: but also keeping watch in #fedora-admin and #fedora
20:33:42 * stickster wants to say a big "thank you" from all the people in Fedora that won't know to whom they should say it... for working so hard to keep us in business during the move.
20:34:01 <mmcgrath> nb: but yeah, when stuff comes back online smooge and I will probably say stuff like "the wiki is back"
20:34:05 <mmcgrath> we'll need verification of that :)
20:34:26 <nb> ok
20:34:28 <sijis_afk> i know i've missed most of the meeting.. but its this weekend, right?
20:34:33 <mmcgrath> sijis_afk: correct.
20:34:46 <mmcgrath> stickster: :)
20:34:49 <mmcgrath> any other questions?
20:34:50 <sijis_afk> i should be around and help if possible
20:35:07 <mmcgrath> when I'm done with this IP list I could use some extra eyes
20:35:07 * nb will be around saturday and sunday afternoon/evening
20:35:23 <mmcgrath> Anyone that'll be around in the next hour or so just look, try to find duplicates or where I've done something stupid.
20:35:29 <nb> mmcgrath, ok
20:35:47 <mmcgrath> If no one has anything else, I'll close the meeting in 30
20:36:39 <mmcgrath> alllrighty
20:36:46 <mmcgrath> #endmeeting