19:00:00 <nirik> #startmeeting Infrastructure (2011-04-28)
19:00:00 <zodbot> Meeting started Thu Apr 28 19:00:00 2011 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:00 <nirik> #meetingname infrastructure
19:00:00 <zodbot> The meeting name has been set to 'infrastructure'
19:00:01 <nirik> #topic Robot Roll Call
19:00:01 <nirik> #chair goozbach smooge skvidal codeblock ricky nirik
19:00:01 <zodbot> Current chairs: codeblock goozbach nirik ricky skvidal smooge
19:00:14 <goozbach> morning/afternoon/evining
19:00:23 * ricky 
19:00:47 * CodeBlock here
19:00:59 <CodeBlock> nirik: you have an alias for that, don't you. :P
19:01:19 <nirik> naw. I keep meeting stuff in a text file... copy and paste it in when doing a meeting. ;)
19:01:26 <nirik> I should make a macro or something.
19:01:28 <abadger1999> hola
19:01:29 <smooge> here
19:01:43 <ricky> Welcome back, abadger1999 :-)
19:01:51 <abadger1999> ricky: Thanks :-)
19:01:56 <nirik> yeah, welcome back abadger1999. ;)
19:02:14 <goozbach> I'mma gonna be keeping notes in-line with meeting so our minutes are more easily digested
19:02:14 <nirik> ok, lets go ahead and dive in...
19:02:20 <nirik> goozbach: cool. ;)
19:02:20 <goozbach> it'll be a bit noisy
19:02:36 <nirik> should I do things in the order of the agenda ?
19:03:23 <nirik> #topic Upcoming Tasks
19:03:45 <goozbach> nirik: it's your meeting, I just keep notes :D
19:03:52 <nirik> Just wanted to go over stuff upcoming that I have:
19:04:02 <nirik> later today I intend to cut over to log02 from log01
19:04:26 <goozbach> #info migrate log01 to log02 later today (nirik)
19:04:29 <nirik> argh.
19:04:35 <nirik> have to step away for a sec
19:05:03 <goozbach> time out, we've got time I believe
19:05:40 * CodeBlock would step in, but I don't know what else nirik has planned this week
19:05:58 <CodeBlock> db01 move at some point, I think
19:06:48 <nirik> sorry, dog required help
19:06:58 <nirik> ok, so yeah, log01->log02 later today.
19:07:05 <nirik> next week, db1->db05
19:07:25 <goozbach> #info db move next week db1 to db05
19:07:44 <nirik> 2011-05-02 I had down as a tenative for FPCA landing.
19:07:45 <ricky> What are the changes from log01 to log02?  RHEL6 upgrade and storage, I assume?
19:08:13 <goozbach> # info FPCA 2011-05-02
19:08:19 <nirik> ricky: yeah, rhel5->rhel6 and so we can not have anything on xen10
19:08:23 <nirik> which is ending life.
19:08:42 <nirik> db01 is on xen12 which is also ending life soon.
19:08:50 <ricky> Ah, OK.
19:09:06 <nirik> abadger1999: does 2011-05-02 still sound ok for FPCA? do we need to do any prep for that? announcements, etc?
19:09:30 <abadger1999> nirik: I'm trying to squeeze a few more features in but I think it's still okay.
19:09:48 <abadger1999> nirik: We should take a short outage.
19:10:00 <nirik> ok.
19:10:01 <abadger1999> nirik: Since there's a quick db schema update that needs to happen.
19:10:53 <nirik> ok, how long? we should file a ticket and get the announcement sent out.
19:11:23 <abadger1999> nirik: Does 5:00-6:00 EST sound good?  (probably will only take a few minutes but an hour gives us leeway for unanticipated breakage/revert if necessary).
19:11:35 <nirik> ok.
19:11:43 <abadger1999> Err -- 5-6PM
19:12:02 <nirik> so, does this come with the 'everyone must sign the FPCA' to stay active? or ?
19:12:17 * nirik digs up the ticket
19:12:20 <abadger1999> Nope,  It will have: (1) new people must sign the FPCA.
19:12:31 <abadger1999> (2) current signers of the CLA may sign the FPCA.
19:12:46 <nirik> .ticket 2481
19:12:47 <zodbot> nirik: #2481 (Fedora switching from the CLA to FPCA) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2481
19:13:01 <abadger1999> After it's deployed I'll talk to the Board/spot/jsmith about notifying people to sign the FPCA.
19:14:17 <abadger1999> We'll decide on a time to remove non-signers of the FPCA from cla_done after we get a feel for whether people are dragging their feet or not.
19:14:23 <nirik> ok
19:15:18 <nirik> sounds good. If you could file the outage ticket and get the announcement ready? or would you like me to?
19:16:09 <abadger1999> I can do it tonight.
19:16:18 <nirik> excellent. thanks.
19:16:19 * abadger1999 adds a reminder
19:16:33 <nirik> also, upcoming on 2011-05-05 we have talk going away. ;)
19:17:00 <nirik> also cvs is down now, and I will be removing it next week
19:17:14 <nirik> so, that brings us up to out final freeze at 2011-05-10
19:17:28 <nirik> anyone have any items they would like to try and get in before the freeze not mentioned yet?
19:17:53 <nirik> Oh, one more, also on 2011-05-10 is the pkgs01 branches fix.
19:18:10 * StylusEater_work is late
19:18:14 <goozbach> #info CVS dying next week for last
19:18:32 <goozbach> #info 2011-05-10 is the pkgs01 branches fix.
19:18:52 <goozbach> #info final freeze at 2011-05-10 -- Anything else need to get done before?
19:19:20 <nirik> we have a bunch of stuff we can schedule after f15 goes out, but we can talk about those in coming weeks.
19:19:21 * CodeBlock can't think of anything
19:19:26 <nirik> ok, moving on...
19:19:31 <nirik> #topic Application logging info gathering
19:19:47 <nirik> So, we currently have logs from things back to long long ago.
19:19:58 <nirik> smooge asked legal if we should have any specific log retention policy.
19:20:07 <nirik> they wanted to know what kind of logs and type of info was in them.
19:20:16 <nirik> So, I made: https://fedoraproject.org/wiki/Infrastructure_logs_information
19:20:32 <nirik> if everyone could fill in there info we know about that would be great.
19:21:06 <smooge> they are working through what needs are done and will get back to us probably after the summit
19:21:31 <goozbach> #action everyone review https://fedoraproject.org/wiki/Infrastructure_logs_information
19:22:15 <nirik> smooge: ok. I was thinking we were gathering more info for them...
19:22:28 <nirik> I don't really know what info some of our apps log.
19:22:52 <goozbach> we don't need to review then?
19:23:05 <smooge> I will double check through my emails
19:23:15 <smooge> no it is a good idea for us to review
19:23:31 <goozbach> ok action item stands
19:23:33 <smooge> because crap could be in there no one remembers being there
19:23:43 <ricky> I think it's good for us to review, but "asking legal" always scares me a little bit :-)
19:24:01 <nirik> we should come up with a retention policy tho at some point...
19:24:15 <nirik> 'keep all logs forever' at some point ends in tears. ;)
19:24:32 <smooge> it has to be done in the case of logs though. People go to jail for realz when you don't
19:24:55 * StylusEater_work wonders why we didn't first setup a retention policy then ask the lawyers
19:24:59 <nirik> ok, so fill in info there if you know it ;)
19:25:04 * skvidal is here, finally.
19:25:10 <CodeBlock> hi there skvidal
19:25:11 <nirik> StylusEater_work: well, same effect I suspect in the end.
19:25:15 <nirik> hey skvidal
19:25:22 <nirik> so, shall we move on?
19:25:39 <smooge> we had a policy it was "keep it forever". Now we want to change it
19:25:50 <smooge> there moving on
19:25:56 <nirik> #topic Meeting tagged tickets
19:26:03 <nirik> https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=~Meeting&order=priority
19:26:35 <nirik> anyone have anything they would like to shout out from meeting tickets?
19:27:08 <StylusEater_work> I made the suggestion we look at OpenStack in addition to Eucalyptus for Ticket #2572
19:27:14 <StylusEater_work> .ticket 2572
19:27:16 <zodbot> StylusEater_work: #2572 (Consider using Eucalyptus for test VM hosting) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2572
19:27:45 <nirik> is openstack packaged?
19:28:06 <StylusEater_work> I don't think either are...
19:28:20 <StylusEater_work> but there are rumblings on the openstack list
19:28:20 <nirik> yeah, which is kinda a bummer. ;(
19:28:33 <nirik> looks like openstack-objectstorage is
19:28:43 <nirik> openstack-swift
19:29:03 <StylusEater_work> ahh there are rhel6 packages
19:29:14 <StylusEater_work> https://github.com/griddynamics/openstack-rhel
19:29:18 <smooge> at this point without stuff being packaged and accepted into Fedora.. it is all a bit of cart <= horse
19:29:26 <nirik> yeah.
19:29:31 <goozbach> nice!
19:29:42 <nirik> I'd love for there to be one we could use. ;)
19:30:05 <smooge> the other thing was I believe this ticket can be closed for another reason...
19:30:20 <StylusEater_work> ?
19:30:36 <smooge> we either needed to build our own cloud or pay someone else to do so.
19:30:49 <smooge> we aren't able to pay and we don't want to build our own cloud :)
19:31:27 <StylusEater_work> valid points ... :-)
19:31:32 <nirik> There could be advantages to our own... but we need something mature enough and packaged, which is not existing yet so far as I know. ;)
19:31:58 <nirik> anyhow, shall we move on? or anything more on meeting tickets?
19:32:14 <smooge> secondary1 ticket I will work on for next
19:32:15 <smooge> week
19:32:25 <nirik> cool.
19:32:36 <nirik> whats left there? a secondary02 instance and thats about it?
19:32:57 <ricky> So in general, are we now incrementing numbers whenever we rebuild something?
19:33:16 <nirik> ricky: I have been, but I'm open to better ideas/practices.
19:33:50 <nirik> it allows a pretty easy backout if something blows up on the new one.
19:34:06 <smooge> For services where we need to transition them (keep secondary01 up while we get secondary02 going or db systems)
19:34:27 <ricky> OK.  Will think a bit on it.  It might make us maintain CNAMEs or some other complex stuff
19:34:38 <ricky> I generally allow backing out by just not destroying the old host
19:34:40 <skvidal> cnames aren't complex
19:34:43 <skvidal> but yah - I see your point
19:34:53 <smooge> I would say we rename them later but I got gunshy after the wrath of Seth on my last attempt :)
19:35:00 <skvidal> so much pain
19:35:01 <skvidal> :)
19:35:06 <ricky> Let's use hexadecimal next time
19:35:28 <nirik> well, after db01 goes away in favor of db05, we could reuse db01 again. ;)
19:35:35 <smooge> ricky, I asked for that last release and it didn;'t get a second +1.
19:35:51 * CodeBlock always thought rename -- that's why I built the new noc01 as noc03-tmp, with intention of it eventually being renamed back to noc01 after the real noc01 was killed
19:35:52 <smooge> I figured dbOxOOf was clear
19:36:00 * skvidal hates everyone
19:36:15 <goozbach> #agreed wait to build our own cloud till things are packaged and included
19:36:20 <smooge> its ok skvidal eunice and the dog still put up with your
19:36:21 * nirik doesn't care too much, but renaming seems like extra work and we are unlikely to run out of integers. ;)
19:36:53 <ricky> Yeah, it's mainly that we have to update some scripts, and update some sysadmin's finger muscle memory (although yeah, CNAMEs would be another solution to that)
19:37:02 <smooge> yeah.. we just need to update scripts and documentation
19:37:13 <smooge> so that if it says db01 we know its now db05
19:37:30 <nirik> well, when moving to the new one we have to update those anyhow right?
19:37:43 <nirik> but I agree admin memory might be ill served.
19:37:57 <ricky> If it's just a regular rebuild like we want to start doing, then hopefully docs won't change much
19:38:30 <ricky> But then there'll be one period every once in a while where you have to +1 to all your hostames
19:38:43 <ricky> Or update CNAMEs and wait for DNS, or whatnot
19:38:53 <nirik> yeah.
19:38:53 <ricky> Anyway, just something to think about, feel free to move on :-)
19:38:55 <sijis> crazy thought which may not be practical. what if physical server names are 'db05' but we have a cname that's dbv01 and have all the scripts use dbv01
19:39:16 <nirik> sijis: could get a bit confusing...
19:39:17 <sijis> so the only change neded would be the cname from db06 to db01
19:39:18 * skvidal twitches
19:39:24 <skvidal> that's just a cname
19:39:35 <nirik> If we are going to look at cnames, functional ones might be better, IMHO...
19:39:37 <sijis> how about aliases inside
19:39:42 <nirik> db-wiki, db-smolt
19:39:58 <nirik> then you could also seperate them easier to different backends.
19:40:27 <skvidal> nod
19:40:27 <sijis> sure. my only though was physical -> virtual name and have things poing to the virtualname instead ot the physical name
19:40:27 <skvidal> +1
19:40:31 <sijis> (if that makes sense)
19:40:31 <ricky> Yes!  Even mysql01 and postgres01, etc would make me happier :-)
19:40:40 <ricky> **happier than the way it is now
19:41:00 <nirik> doesn't db02 have both on it?
19:41:08 <ricky> See?  It's confusing :-(
19:41:15 <ricky> db01 = mysql, db02 - postgres for apps
19:41:15 <skvidal> :)
19:41:21 <ricky> db04 or whatever it is now = postgres for koji
19:41:25 <nirik> anyhow, I think this merits thought... if someone comes up with a good idea/plan, feel free to propose it.
19:41:53 <StylusEater_work> nirik: do we have a page on the wiki with a problem definition?
19:42:09 <StylusEater_work> nirik: other than the typical conundrum on system naming...
19:42:14 <nirik> not that I know of... might be more suited for a mailing list discussion?
19:42:34 <nirik> basically the issue is this... we have a db01 machine in puppet, we want to replace it with another new shiny instance.
19:42:52 <nirik> if we also call it db01, then there is a time where we have to cut over and the old machine gets no more updates, etc.
19:43:29 <nirik> and there may be puppet changes to make the new db01 happy that would have to be reverted if we went back to the old one.
19:43:40 <fresham> nirik: not sure if this helps, but we always did generation in the ID in the labs I worked in
19:44:00 <StylusEater_work> fresham: db01-0, db01-1 ... etc.?
19:44:01 <nirik> fresham: generation in the ID?
19:44:08 <nirik> ah.
19:44:21 <StylusEater_work> fresham: or something different?
19:44:22 <fresham> nirik: so db0101 would be the first one, db0102 could be like a temp or transfer, then db0201 would be the reincarnation of db0101
19:44:41 <nirik> huh. Another one to think about...
19:44:58 <fresham> that's just an example, I'm sure there' s a more elegant way of doing it
19:45:01 <fresham> anyway...
19:45:03 <nirik> if someone wants to bring this to the mailing list we can try and come up with a plan there.
19:45:29 <nirik> #topic New folks introductions/status
19:45:35 * skvidal starts to decipher nirik-speak
19:45:38 <StylusEater_work> nirik: non-functional names would fix it...
19:45:44 <StylusEater_work> err...oops
19:46:06 <goozbach> #action bring up naming and replacing machines in mailing list (anyone)
19:46:10 <nirik> so, any new folks want to introduce themselves?
19:46:15 <fresham> I'm new!
19:46:23 <goozbach> #info welcome fresham
19:46:25 <goozbach> :)
19:46:28 <fresham> My name's Cody
19:46:32 <fresham> Thanks!
19:46:37 <nirik> or any new folks want to give us feedback on recent weeks? :)
19:46:41 <nirik> welcome fresham
19:46:46 <fresham> thx
19:47:23 <nirik> is there any particular areas you are interested in?
19:47:54 <fresham> I looked at fedorahosted for the programs we use. Koji and Bohdi seem particularly interesting.
19:48:10 <fresham> As I mentioned on the mailing list, I really like writing scripts to automate things.
19:48:14 <nirik> cool. ;)
19:48:31 <smooge> welcome.
19:48:38 <nirik> the one I know of off hand is a script to automate hosted requests.
19:48:44 <fresham> smooge: thanks!
19:48:48 <CodeBlock> ^
19:48:49 <nirik> CodeBlock has already written one, but it was lost in a HD crash. ;)
19:48:54 <fresham> lol
19:49:09 <fresham> That sounds straightforward
19:49:25 <nirik> CodeBlock: would you be willing to help fresham with that? or would it be better to just re-do it yourself? or ?
19:49:33 <CodeBlock> well - I originally wrote one, its on hosted01 in my ~, but it was kind of hacky. Then I started re-writing it, and that was lost. :/
19:49:43 <nirik> ah, ok.
19:49:51 <fresham> I'm currently without sponsor and credentials
19:50:08 <CodeBlock> nirik: more than willing to help, that was technically my first assigned project when I joined FI..and it's still not done :P So if someone wants to pick it up...
19:50:56 <CodeBlock> .fas fresham
19:50:57 <zodbot> CodeBlock: 'fresham' Not Found!
19:50:57 <nirik> fresham: not a problem. Do hang out in #fedora-admin and/or #fedora-noc and we will get you what you need to work on what you want to work on...
19:51:09 <fresham> try crfrazer
19:51:27 <CodeBlock> fresham: ping me later in the channels nirik mentioned, and I can sponsor you into the fi-apprentice group for starters
19:51:38 <CodeBlock> and go from there
19:51:44 <fresham> Can do! Thanks!
19:51:45 <nirik> which brings us to our next topic:
19:51:50 <nirik> #topic Status of the sysadmin-apprentice group
19:52:15 <nirik> so, we are using this now. How's it working? anything we can do to improve it?
19:52:29 <nirik> (well, it's fi-apprentice)
19:52:52 <nirik> I think it's been good... have had one nagios guy looking at old tickets and able to check our config, etc.
19:53:07 <CodeBlock> yeah
19:53:50 <skvidal> so what's the discussion?
19:53:52 <casep> It need more promotion, I still don't know where it's :)
19:54:12 <nirik> goozbach: you added this to the agenda? or someone sent to you?
19:54:24 <nirik> some background:
19:55:02 <nirik> fi-apprentice is a group that has login access to a lot of infrastrcuture machines. They can login and look at things. They have no write privs. no sudo or the like. They can only look...
19:55:18 <nirik> using this people can get in and see how things are setup and propose solutions to open tickets and the like.
19:55:24 <goozbach> nirik: I've been hearing bout it since I apprenticed, I'm still not in it :)
19:55:37 <goozbach> so I've taken the mantel of secratary for the meetings :)
19:55:40 <nirik> .members fi-apprentice
19:55:41 <zodbot> nirik: Members of fi-apprentice: addutko +codeblock jbass29503 @kevin rafaelgomes @skvidal @smooge
19:55:52 <nirik> goozbach: we can sponsor you in...
19:56:00 <goozbach> that'd be nice
19:56:05 <goozbach> please
19:56:22 <smooge> hmm I thought we had more people in it.
19:56:29 <smooge> ok I will start sticking people in it.
19:56:40 <nirik> smooge: sounds good.
19:57:07 <smooge> Now that I have someone to sluff off the throne to I can go back to being the power behind it and getting new minions to control
19:57:09 <nirik> we will likely cull people from this group after a while if we don't hear anything from them, so do use the access for something. Working on tickets, making suggestions, etc.
19:57:32 <smooge> correct.
19:57:33 <CodeBlock> smooge: btw, can you make me a sponsor of 'sysadmin'? I can sponsor in some sysadmin-* groups and fi-apprentice, but not sysadmin (which means being able to sponsor in fi-apprentice does little good)
19:57:47 <nirik> CodeBlock: I thought sysadmin was open to anyone joining?
19:57:56 <smooge> nirik, no we changed that last year
19:58:07 <nirik> huh. ok
19:58:20 * nirik supposes he should be a sponsor there too.
19:58:21 <smooge> basically we had a LOT of driveby signups because people were thinking "I sysadmin my home box I need to be in this group."
19:58:22 <ricky> Too many applications from people we'd never heard of :-(
19:58:33 <nirik> ah, the common problem. ;( ok
19:58:37 <goozbach> .fasinfo goozbach
19:58:38 <zodbot> goozbach: User: goozbach, Name: Derek Carter, email: goozbach@friocorte.com, Creation: 2008-04-08, IRC Nick: goozbach, Timezone: US/Eastern, Locale: en, Extension: 5103664, GPG key ID: 1024D/D031A544, Status: active
19:58:42 <zodbot> goozbach: Approved Groups: sysadmin cla_fedora cla_done
19:58:45 <skvidal> like that guy
19:58:54 <ricky> Hehe
19:58:56 <skvidal> :)
19:59:01 <smooge> they would sign up for sysadmin and sysadmin-test
19:59:35 <goozbach> but hey! I'm not letting my lack of sponsorship keep me from contributing
19:59:41 <smooge> nope.
19:59:49 <goozbach> :D
19:59:55 <CodeBlock> Alright two minutes left, so I'll leave with this (because I want to leave $dayjob in a minute too) - zodbot has been successfully moved to value01, so if anyone has any problems with it or meeting log website, etc, let me know -- but it seems to be working fine.
19:59:58 <nirik> ok, we are coming up on an hour... anything else on this?
20:00:06 <skvidal> yaya CodeBlock
20:00:09 <smooge> I need to get some food. my blood sugar crashed
20:00:10 <nirik> CodeBlock: excellent.
20:00:16 <nirik> #topic Quick Open Floor
20:00:21 <nirik> anything for open floor real quick?
20:00:30 <skvidal> is anyone looking for a project?
20:00:56 <CodeBlock> skvidal: such as? :P
20:01:16 <skvidal> 2 items come to mind
20:01:20 <skvidal> 1. the publictest-minder
20:01:36 <skvidal> 2. puppet nodes + func = nagios hosts+ checks + deps
20:01:48 <skvidal> they are both fairly standalone projects
20:01:54 <nirik> skvidal: might be worth filing tickets on these with background/requirements so we can point people to them?
20:01:59 <StylusEater_work> publictest-minder?
20:02:03 <skvidal> nirik: fair enough
20:02:05 <goozbach> #info codeblock moved zodbot to value01, any issues?
20:02:29 <nirik> StylusEater_work: a way to keep track of our publictest boxes, who's using them, how long for, etc.
20:02:43 * skvidal points to nirik's description
20:02:48 * StylusEater_work looks forward to skvidals description
20:03:07 <goozbach> #info StylusEater_work added this to mailing list: http://lists.fedoraproject.org/pipermail/infrastructure/2011-April/010215.html
20:03:12 <skvidal> that's the majority of it - just a way of tracking why we did this
20:03:17 <skvidal> and how long it is needed for
20:03:45 <CodeBlock> skvidal: web interface kind of thing or what?
20:03:47 <StylusEater_work> skvidal: talk to you in fedora-admin about it
20:03:52 <nirik> ok, thanks for coming everyone! Continue over in #fedora-admin / #fedora-noc.
20:03:53 <skvidal> ok
20:03:56 <skvidal> CodeBlock: nah
20:04:02 <goozbach> yay #info meeting over
20:04:10 <goozbach> #info meeting over
20:04:11 <nirik> #endmeeting