19:00:17 <nirik> #startmeeting Infrastructure (2011-04-14) 19:00:17 <zodbot> Meeting started Thu Apr 14 19:00:17 2011 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:17 <nirik> #meetingname infrastructure 19:00:17 <zodbot> The meeting name has been set to 'infrastructure' 19:00:17 <nirik> #topic Robot Roll Call 19:00:17 <nirik> #chair goozbach smooge skvidal codeblock ricky nirik 19:00:17 <zodbot> Current chairs: codeblock goozbach nirik ricky skvidal smooge 19:00:19 <zodbot> rbergeron: Error: Can't start another meeting, one is in progress. 19:00:21 <rbergeron> ohh 19:00:29 * CodeBlock here 19:00:29 <nirik> humf 19:00:30 <nirik> oops 19:00:32 * rbergeron always does this to you guys 19:00:43 <rbergeron> if anyone pops in, send them to f-m-1 19:00:43 <goozbach> wait, what? 19:00:45 <rbergeron> that's one, not L 19:00:46 <smooge> what 19:00:46 <rbergeron> :) 19:00:58 <goozbach> hah! 19:01:00 <goozbach> ok ok 19:01:02 * goozbach here 19:01:04 <smooge> remember L 19:01:08 <rbergeron> :) 19:01:09 * jpattonwx here, lurking 19:01:49 * jsmith lurks 19:02:40 * sijis is around 19:03:00 * crashmag says hi 19:03:20 <nirik> sorry about that... 19:03:27 <nirik> somehow I didn't realize they were the same time. ;( 19:03:36 <smooge> well we have it first 19:03:47 <smooge> who needs a release anyway 19:04:11 <nirik> ok, lets gets started I guess. 19:04:20 <nirik> #topic New Folks Intros 19:04:28 <nirik> any new folks lurking here this week? 19:04:36 <crashmag> yap :) 19:04:37 <nirik> want to say hi and such? 19:04:45 * jsmith hasn't said hi in a long time :-p 19:05:04 <nirik> crashmag: welcome. 19:05:05 <smooge> I will be able to help people get into infrastructure after this release I believe 19:05:33 <CodeBlock> I can sponsor some people after the semester ends, too 19:05:43 * skvidal is here 19:05:46 <skvidal> sorry I was late 19:05:52 <CodeBlock> skvidal: no. unexcusable. 19:05:53 <jpattonwx> I think I will wait until after semester here too before worrying about sponsorship 19:05:55 <nirik> crashmag: any thoughts on areas you are interested in? do lurk in #fedora-admin and/or #fedora-noc and ask questions and chime in when you can. 19:06:07 <skvidal> CodeBlock: I know - I shall go flagellate myself 19:06:08 <CodeBlock> skvidal: just like my bad grammer. inexcusable* 19:06:45 * nirik points new folks to https://fedoraproject.org/wiki/Infrastructure/GettingStarted lots of good info there. 19:07:07 <crashmag> nirik, well I work for an ITSM company. I realise all of this is based upon trust, and that takes time. So I was hoping that theres a way at working on "lower level" tickets to begin with. 19:08:02 <nirik> crashmag: sure. ;) Do look at our trac instance and https://fedoraproject.org/wiki/Infrastructure_Cleanup_Tasks_2011 19:08:16 <nirik> there are some tasks that don't need much access at all... 19:08:25 <crashmag> started using linux around 02. my geek factor can be measured by my 3 servers at home. was hoping to help out and learn by doing so :) 19:08:56 <nirik> excellent. 19:09:27 <nirik> often lurking in the above channels can get you a feel for things going on, and people are usually happy to explain, etc. 19:09:55 <smooge> oooh itsm I just closed some tickets on that 19:10:08 <nirik> ok, shall we touch on Beta tickets? 19:10:18 <nirik> #topic Fedora 15 Beta tickets. 19:10:58 <nirik> so beta will be released tuesday. 19:11:05 <nirik> we have several tickets to make sure that goes well. 19:11:15 <nirik> .ticket 2704 19:11:16 <zodbot> nirik: #2704 ([Fedora 15 Beta] New Website) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2704 19:11:45 * nirik sees sijis is answering for websites in the other meeting right now. ;) 19:11:46 * CodeBlock looks at sijis 19:12:39 <nirik> we can come back to this one. 19:12:45 <sijis> ok. 19:12:45 <nirik> .ticket 2707 19:12:46 <zodbot> nirik: #2707 ([Fedora 15 Beta] Permissions on mirrors) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2707 19:12:59 <nirik> we need content first, then we can check this. 19:13:16 <nirik> .ticket 2708 19:13:17 <zodbot> nirik: #2708 ([Fedora 15 Beta] Mirror manager repository redirects) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2708 19:13:27 <nirik> will need to check with mdomsch on that one. 19:13:56 <nirik> sijis: any changes in size or anything for the Beta website? any other notable changes? 19:14:41 <sijis> nope. the size will increase a little but not significant 19:14:48 <sijis> about 5-10mb (if that) 19:14:55 <sijis> its because of the countdown banner 19:15:02 <nirik> ok, cool. 19:15:31 <nirik> .ticket 2706 19:15:32 <zodbot> nirik: #2706 ([Fedora 15 Beta] Release Day) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2706 19:15:43 <nirik> .ticket 2710 19:15:45 <zodbot> nirik: #2710 ([Fedora 15 Beta] Infra change freeze) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2710 19:15:45 <smooge> thats yours as boss 19:16:03 <nirik> Thats for release day issues, and a reminder that we are in freeze. ;) 19:16:06 <nirik> smooge: yep. ;) 19:16:12 <CodeBlock> :P 19:16:23 <nirik> I think thats all the beta tickets. Anyone have anything else Beta related? 19:16:26 <smooge> I closed the disk space one. we are way under 1TB at the moment 19:17:21 <nirik> excellent. 19:17:47 <nirik> #topic Upcoming items and outages 19:17:52 <smooge> I will be starting a sync of F13 over to archives soon and a sync of secondary1 over to the netapp also 19:18:12 <nirik> I have a list in the post on the items I know of that are coming up after this freeze. 19:18:27 <nirik> smooge: ok. 13 has a bit more life, but not too much. 19:18:48 <nirik> whats left on secondary1? staging stuff, some video content from old fudcons, etc? 19:18:49 <CodeBlock> smooge: not sure what that entails, but if you need some help and feel like teaching me, by all means, would love to learn :) 19:19:35 <smooge> yeah I sync stuff over just to make it easier for when it is EOL 19:19:44 <nirik> The end of next week we will be having an outage to move db03, nfs01, and pkgs01 (so it will be an outage for building/packagers). 19:20:10 <smooge> and fas01 19:20:43 <nirik> oh, I didn't have that one on my list. That was a rhel6 guest on a xen5 host we wanted to move? 19:21:02 <smooge> correct 19:21:14 <smooge> it will cut a bunch of issues down 19:21:27 <smooge> as it silently reboots a lot and sometimes not so silent 19:21:37 <smooge> it just needs to be built onto virthost03 19:21:38 <nirik> ok. will add to the list. We will figure out exact timing on them and I will send out an outage notice. 19:22:05 <smooge> fas01 contains various certs and such so when it is going down it will degrade other items 19:22:39 <nirik> ok. We will need to determine that. Ideally we can sync it to a new one in advance so the outage window is small. 19:23:03 <nirik> db03 will take the longest, as it will be a dump/reload on new machine. 19:23:24 <CodeBlock> indeed 19:23:32 <smooge> pkgs01 may be the simplest 19:23:47 <nirik> yeah, it should be just shutdown in one place and bring up in another. ;) 19:24:18 <ranjibd> db02 can be done like that too right? 19:24:25 <ranjibd> sorry db03 19:24:43 <nirik> well, no, as we are switching it from rhel5 older db version to rhel6 19:24:57 <nirik> it's now a rhel5 box with a custom postgresql on it. 19:25:07 <nirik> it will go to rhel6 with the normal version. 19:25:43 <nirik> eventually we will move the other db's over to rhel6 too... but not next week 19:26:22 <nirik> after next week we have puppet upgrade and fpca landing, then after final we are going to upgrade nagios/noc01. ;) 19:26:35 <CodeBlock> yep 19:26:38 <nirik> anyone have any other upcoming items they would like to discuss planning on. 19:27:02 * CodeBlock still wants to start playing with moving hosted to EL6, but that is looking like a summer project 19:27:32 <nirik> yeah, that would be great to get done... 19:27:33 <nirik> someday. 19:27:51 <nirik> ok, meeting tickets... 19:27:56 <nirik> #topic Meeting tickets 19:28:00 <nirik> https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=~Meeting&order=priority 19:28:42 <nirik> a grab bag of things. ;) 19:29:05 <nirik> #info on cvs01, I am waiting to hear back from the last 2 projects on there. They may move to fedorahosted, or fedorapeople. 19:29:43 <nirik> Anyone see a meeting ticket they would like to comment on? 19:31:11 <CodeBlock> what ever happened with moving blogs off of infra? 19:31:31 <CodeBlock> (i.e. ticket 2591) 19:32:22 <nirik> it's not really moving forward. ;) I think ricky was going to look into it more when he had time... 19:32:38 <nirik> it needs someone talking to wordpress.com or the like... and then talking to the blog users. 19:32:41 <smooge> yeah. it needs a key person to push it forward. 19:32:45 <smooge> sorry slow type 19:32:46 <sijis> yeah, i'm not sure what's left on that. 19:33:06 <smooge> I would say if its not in the ticket.. then everything is left on that :) 19:33:17 * jsmith has some friends at wordpress.com, but they're busy cleaning up from the recent security incident 19:33:19 <sijis> is it contact wp and see if they will host that stuff for us and contacting users and seeing if they need help moving (or are we just moving stuff for them) 19:33:23 <jsmith> I'd be happy to make some introductions 19:33:33 <sijis> oh yeah. i read that this morning 19:33:34 <nirik> jsmith: That would be cool. 19:34:05 <nirik> yeah, contact them and see what they can do for us... then once we know that, contact all our blog users and tell them about it... 19:34:26 <nirik> I think it will depend on what they want to do if we can help move them or if users move their own. 19:34:37 <nirik> and we will need to set a sunset date... 19:34:56 <sijis> i believe they can export all their content with a single click. 19:35:07 <nirik> yeah, should be pretty easy. 19:35:37 <nirik> we also want to explain to them why we are doing this and such... so no one is blindsided or confused. 19:35:56 <sijis> good point. its noted 19:36:27 <nirik> so, if anyone would like to drive that forward, please feel free. 19:36:46 <nirik> #topic Other tickets 19:36:56 <nirik> Any other tickets folks would like to bring up or note? 19:37:32 <ranjibd> nagios upgrade? skvidal was telling last time it didn't work due to some issues 19:37:47 <skvidal> ranjibd: nirik and CodeBlock have the knowledge there 19:37:57 <nirik> yeah, ran into issues with our irc bot, and backed out the upgrade. 19:38:08 <nirik> CodeBlock is planning on doing it again after f15 final. 19:38:17 <nirik> Hopefully it will go smoothly this time. ;) 19:38:39 <CodeBlock> indeed 19:38:50 <ranjibd> puppet upgrade ? can i help any way? 19:38:53 <sijis> is that why we have zodbot and fedbot? 19:39:45 <smooge> possibly.. it was looking at what we could break out 19:39:48 <nirik> sijis: nope. ;) fedbot is one I run locally, mostly for #fedora. It's got some non packaged plugins and such, also it's locked down to not respond to lots of things, which we wouldn't want zodbot to do. 19:40:48 <nirik> ranjibd: it should be pretty easy. Basically just upgrade the master, upgrade a 5 and 6 client and test... it should go smoothly. 19:41:13 <nirik> I don't think splitting zodbot is going to help any. 19:41:22 <ranjibd> nirik: all the clients have same version ? 19:41:43 <skvidal> splitting zodbot won't help the migration - but it will make the bot in general make a lot more since where it should live 19:42:03 <skvidal> having our meetingbot on our nagios server makes no sense to me 19:42:22 <nirik> skvidal: yeah... but where should the meeting part go? 19:42:35 <skvidal> nirik: away 19:42:47 <skvidal> nirik: away from our nagios server 19:42:50 <skvidal> it makes me nervous 19:42:59 <nirik> ranjibd: yeah. 19:43:08 <ricky> For what it's worth, there's no reason that zodbot has to be on the same machine as noc01 to get nagios notifications 19:43:24 <ricky> So moving zodbot to value or something is easy 19:43:27 <nirik> skvidal: ok, we can figure something out. 19:43:37 <nirik> yeah, value might make some sense. 19:43:59 <nirik> we do need to serve the meetbot html stuff. 19:44:24 * nirik makes a note. 19:44:40 <nirik> splitting it won't help the lag on restart unless we split it out by channels... 19:45:02 <nirik> we may be able to prune the channels it's joining... 19:45:43 <ricky> Sorry, I missed the beginning due to internet issues - what does splitting zodbot mean? 19:46:00 <sijis> it would be two bot.. zod and bot :) 19:46:19 <sijis> ok. i need a break. (lame joke) 19:46:26 <nirik> there was talk about splitting it into fasbot, nocbot, meetbot 19:46:33 <smooge> smoogebot 19:46:35 <nirik> sijis: :) 19:46:51 <skvidal> I suggested splitting it logically 19:46:55 <skvidal> into its functional components 19:47:03 <skvidal> and it's authenticated-components 19:47:09 <skvidal> nocbot has more info thant meetboot 19:47:21 <skvidal> fasbot has different access than meetbot or nocbot 19:47:39 <skvidal> conflating all 3 seemed like a recipe for problems 19:47:42 <sijis> isn't that just more stuff to oversee than 1 bot? 19:47:57 <skvidal> also - from a services and meta-services standpoint 19:48:18 <skvidal> it does not make sense to have a user-facing service (zodbot) living on a meta-service server (noc01) 19:48:34 <skvidal> just like it wouldn't make sense to run a webserver for the wiki on our dhcp server 19:48:41 * nirik nods. 19:48:51 <skvidal> that way if noc01 goes down 19:48:58 <skvidal> we don't have to worry about it impacting user-facing services 19:49:00 <nirik> I think moving it is a good idea. Splitting it I am more meh on. 19:49:38 <CodeBlock> I like the idea of splitting it out, but that's me. 19:49:57 <ricky> Same - supybot is pretty simple/stable, and it only really touches data that's public anyway 19:50:06 <ricky> So while I wouldn't mind splitting it out, I don't see it as a huge priority or a huge gain 19:50:20 <skvidal> is everyone on board with moving it OFF of noc01? 19:50:24 <CodeBlock> yes 19:50:25 <nirik> yes. 19:50:27 <smooge> yes 19:50:36 <skvidal> great 19:50:37 <smooge> let me rephrase, YES 19:50:38 <sijis> i do agree that it doens't make sense that its on noc1. it should be in value1/2 as ricky mentioned 19:50:45 <skvidal> value## seems like a winner 19:50:52 <ricky> Yup, only change would be firewall changes to accept supybot-notify + pointing nagios at the new IP 19:51:02 <ricky> Ideally it should not be on a VPN host so that we get notifications if VPN goes down 19:51:04 <nirik> so, how about we move it sooner rather than later... and if someone wants to drive splitting it down the road, they can 19:51:30 <nirik> value already has a webserver for http://meetbot.fedoraproject.org/ 19:51:32 <nirik> ? 19:51:59 <ricky> We currently treat value like an app server, so we could proxy that through the proxies like we normally do if we want 19:52:13 <ricky> Or it'd be fine to poke a hole for it too. 19:52:34 <nirik> ok. 19:53:01 <nirik> does someone want to take the task of moving it? I guess I can... 19:53:05 <CodeBlock> heh, less to worry about for the noc01/noc03-tmp switchover then 19:53:11 <CodeBlock> nirik: I can if you don't want 19:53:32 <nirik> CodeBlock: sure, feel free. :) It will require doing it when no meetings are around... 19:53:41 <CodeBlock> when are we looking to do it? And what kind of an outage notif needs sent out? 19:53:51 <smooge> bah.. I think we time it for rbergeron's most important meeting of the week 19:54:02 <ricky> Probably an outage notification to devel-announce + infrastructure 19:54:10 <nirik> well, we could add it to the pile next week, but thats getting lots of stuff. 19:54:16 <ricky> And as a college student, I'm sure you'll be around at 3 AM some days, right? :-D 19:54:20 <nirik> it shouldn't be frozen, so I suppose we could do it most anytime. 19:54:29 <nirik> oh, that reminds me... 19:54:38 <ricky> (half-kidding - on one hand, changes at 3 AM aren't that great - on the other hand, zodbot's dead simple) 19:55:40 <nirik> What would folks think of the following addition to outage notifications: Once you write up one, have one other person review and approve before sending. It's so easy to mess up timezone stuff, or the like. 19:56:02 <smooge> that would be good 19:56:03 <ricky> Good idea :-) 19:56:09 <CodeBlock> ricky: Yeah 3AM is perfectly fine for me ;D 19:56:10 <skvidal> I have a counterproposal 19:56:11 <smooge> it would make most of my emails though less funny 19:56:13 <ricky> Doesn't have to be formal or anything, just an "OK" in IRC 19:56:20 <skvidal> how about we make all timezones Eastern Standard 19:56:20 <ricky> Right? 19:56:29 <skvidal> so we don't have to mess around with silly things like timezones at all 19:56:30 <skvidal> :) 19:56:32 <nirik> yeah, no red tape, just another pair of eyes. 19:56:33 * skvidal kids 19:56:39 <nirik> skvidal: fine with me. ;) 19:56:44 <CodeBlock> ;D 19:57:02 <ranjibd> +1 for UCT 19:57:06 <nirik> anyone object to the above? will add to sop... 19:57:21 <jsmith> +1 for second set of eyes 19:57:52 <nirik> cool. 19:57:57 <nirik> #topic Open Floor 19:58:01 <nirik> Anything for open floor? 19:58:39 <CodeBlock> 3 weeks or so left in my semester then I'll have a lot more time to be around :) 19:58:39 <ricky> Wow, perfect timing... this has got to be a first! 19:58:50 * CodeBlock assumes same for ricky 19:59:15 <ricky> Yup, will have a lot more time this year 19:59:36 <nirik> cool. ;) 19:59:59 <nirik> ok, if nothing else, will close out the meeting in a minute here. 20:00:49 <nirik> Thanks for coming everyone! 20:00:52 <nirik> #endmeeting