18:00:11 <nirik> #startmeeting Infrastructure (2014-05-22) 18:00:11 <zodbot> Meeting started Thu May 22 18:00:11 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:11 <nirik> #meetingname infrastructure 18:00:11 <nirik> #topic aloha 18:00:11 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:00:11 <zodbot> The meeting name has been set to 'infrastructure' 18:00:11 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:00:45 <dgilmore> hola 18:01:01 <janeznemanic> hi 18:01:08 <lorddemon> Buenas tardes 18:01:10 <hammad> Hello ! 18:01:16 <charul> Hi :) 18:01:19 * bwood09 is here 18:01:23 <mhaynes> Good afternoon. 18:01:29 <mapyth> Hello everybody! :) 18:01:29 <smooge> hllo 18:01:31 <nirik> hello everyone. 18:01:32 <ootbro> hi all 18:01:37 <nirik> #topic New folks introductions and Apprentice tasks. 18:01:49 <nirik> any new folks like to introduce themselves in a line or two? 18:02:00 <nirik> or apprentices with questions or comments? 18:02:11 <ootbro> (waiting for new folks first) 18:02:17 <mapyth> Hello everyone, I am Mayank from India, would be hacking on bugspad this summer! 18:02:24 <charul> Hi everyone. I am Charul and have started working on Shumgrepper project. 18:02:27 * relrod here 18:02:40 * danofsatx-work is here 18:02:45 <hammad> Hello, This is Hammad, Working on fedora-college, that inherently comes under infra. 18:03:00 <lorddemon> Hizo everyone i am Gonzalo from Bolivia 18:03:20 <nirik> great! welcome everyone. 18:03:51 <nirik> Those of you doing summer coding, would you have links to your projects for us to read up on? 18:04:04 <brnzi> Hello guys, My name is Bruno! I am from Brazil, but living in Los Angeles,Ca! 18:04:44 <mapyth> https://github.com/kushaldas/bugspad the project I would be working on! 18:04:51 <mapyth> making a UI for it. 18:04:55 <hammad> https://github.com/hammadhaleem/fedora-college Fedora College. 18:05:24 <charul> https://github.com/fedora-infra/shumgrepper for Shumgrepper 18:05:47 <nirik> great. :) 18:06:10 <mpduty> hello everyone 18:06:34 <nirik> good luck in your coding. ;) 18:06:51 <ootbro> ready for the apprentices ? 18:06:56 <mapyth> I and my mentor kushal, were discussing about some revisions in the timeline of the project, to include the suggestions received from the infra team, for 18:06:57 <charul> thanks nirik :) 18:07:14 <nirik> ootbro: sure, fire away... 18:07:20 <ootbro> following up from the map/landscape/overview "new project" item last week (and the e-mail I sent to the mailing list)..... I haven't gotten any additional source material, so I'll start with what I listed in the e-mail. 18:07:26 <mapyth> and will be discussing tomorrow! 18:07:34 <nirik> mapyth: ok. Might be good to post to the list on it and that way we can get replies from anyone intrested... 18:07:58 <nirik> ootbro: sounds good. 18:08:59 <ootbro> nirik has already said that the first two steps sounded good -- an overall list of servers with a location, function, production status (prod, staging, testing, etc.) 18:09:10 <ootbro> which starts to put the pieces on the table. 18:09:15 * nirik nods. 18:09:32 <ootbro> and add a basic description for each general function that tells what that family of servers does. 18:09:32 <mapyth> nirik: okay sure! shall I put it on summer-coding mailing list? 18:09:57 <nirik> mapyth: I'd say the infrastructure list if you want infrastructure input into things... 18:10:50 <ootbro> In getting my ssh access fixed (thanks, again, nirik :) )... I found an update that needs to be done to the sshaccess.txt file 18:10:55 <nirik> For any other new folks, do see: http://fedoraproject.org/wiki/Infrastructure/GettingStarted if you haven't already, and we can point you in the right direction in #fedora-admin and/or #fedora-apps agter the meeting. 18:11:37 <nirik> ootbro: those docs are in the 'infra-docs' git repo... which actually apprentices do have write access to. ;) Just go to lockbox01 and 'git clone /git/infra-docs' and modify it and commit and push. 18:12:04 <mapyth> nirik: okay. will finalize with my mentor tomorrow and post. 18:12:06 <ootbro> ok. thanks. that was my next question -- how to get an update posted. :D 18:12:26 <nirik> ootbro: easy peasy. ;) (Hopefully) 18:12:50 <ootbro> I'll dig into the documentation for the "how" part of commit and push 18:13:09 <nirik> yep. should be lots of git docs out there... possibly too many. :) 18:13:16 <ootbro> :) 18:13:22 <ootbro> (done) 18:13:31 <nirik> ok, any other new folks or apprentices with questions ? 18:13:36 <nirik> welcome again to all new folks. 18:14:02 <nirik> #topic Applications status / discussion 18:14:18 * nirik sees most if not all our applications folks aren't around. ;( 18:14:29 <nirik> There continues to be some fallout from the pkgdb2 rollout. 18:14:46 <nirik> #info process-epel-requests script being worked on to work with pkgdb2 18:14:51 <brnzi> hi @<nirik> I am new, I am actually writing my introduction right now! :-) 18:15:04 <nirik> #info bugzilla component sync is also not working right, still need to investigate. 18:15:09 <nirik> brnzi: cool. ;) 18:15:14 <brnzi> I am looking for a sponsor.. :-) 18:15:42 <nirik> look for something interesting to you to work on first. ;) 18:15:52 <brnzi> :-) 18:16:21 <nirik> any other applications news today? 18:16:51 <nirik> #topic Sysadmin status / discussion 18:16:59 <nirik> on the sysadmin side... 18:17:11 <nirik> we did a mass reboot tuesday, everything seems to have gone just fine. 18:17:43 <nirik> Our build system is 100% working for the first time in a while... all our arm SOC's, buildvm's, buildhw are all up and running along nicely. 18:17:55 <danofsatx-work> yay! 18:17:58 <nirik> the ansible migration is rolling along 18:18:18 <nirik> I'm hoping to move the last things off our old app servers soon and retire them all. 18:18:35 <nirik> that will be nice. 18:18:49 <smooge> yay 18:19:02 <lorddemon> Sound good 18:19:58 <nirik> our backup server almost ran out of inodes last night... will be trying to clean up what we can there. 18:20:14 * threebean arrives late 18:20:15 <smooge> what was that a recursive backup? 18:20:24 <smooge> someone not playing nice? 18:20:37 <nirik> smooge: there was one gnome backup taking up a lot, it's now been fixed... 18:20:53 <nirik> but there's still a lot of inodes taken up. If my find ever finishes I can see what dirs have a lot. 18:21:36 <nirik> #info mass reboot last tuesday, went fine. 18:21:50 <smooge> for (i=0; i<infinity; i++); do ln -s foo $foo.$i done 18:21:51 <nirik> #info buildsys is 100% up and operational. All arm, buildvm, buildhw boxes working 18:22:09 <nirik> #info smooge and relrod got all the new download servers in place and working 18:22:35 <smooge> now I am dealing with hardware problems on the RDU download servers 18:22:38 <relrod> that was mostly smooge 18:22:47 <smooge> relrod, you did the ansible stuff 18:23:18 <nirik> ok, any other sysadmin side items to mention? 18:23:21 <abadger1999> nirik: question re: app servers ; are the last things simply shifting as a group to new hosts or are they shifting to separate hosts? 18:23:47 <nirik> abadger1999: most of the last things are moving to the sundries servers... the big apps already moved to their own things 18:23:53 <abadger1999> <nod> 18:23:57 <relrod> smooge: doing ansible stuff is easier than looking at a screen and trying to hit f12 in a vnc window within 5 seconds ;) 18:23:58 <nirik> the last thing is freemedia. which is just a php/cgi 18:24:24 <nirik> I was thinking once I get that moved to power them off for a few days... see if anything breaks or still depends on them. 18:24:36 <abadger1999> nirik: Cool. One thought about that -- we may want to upgrade to rhel7 before mirrormanager is ported away from tg1. 18:24:48 <smooge> current items on my task list: fix download RDU server hardware issues, rebuild RDU servers to be ansible, build new log server, get stuff off old log server, and move virthost box over to cloud 18:25:05 <abadger1999> nirik: So we may want to split that away from the other sundries stuff i nthe future. 18:25:36 <abadger1999> (we can cross that bridge when we start thinking about rhel7 migration, though :-) 18:26:08 <nirik> abadger1999: ok. yeah. MM has 3 parts: mirrorlists (already moved to their own instances), mirrormanager adminwebapp (moved to sundries) and backend/cron stuff thats still on bapp02... still need to move that. 18:26:19 <henderbj> Hello. Sorry to be late! 18:26:33 <nirik> henderbj: no worries. welcome. 18:26:38 <abadger1999> (for those who haven't followed -- I'm not planning on maintaining TurboGears1 on EPEL7. So mirrormanager nad FAS will be stuck on RHEL6 until we port them to a newer framework). 18:27:08 * oddshocks here late 18:27:13 * oddshocks roommate troubles 18:27:20 <nirik> abadger1999: we will have lots of other things to migrate, so we can save those for last. 18:27:23 <nirik> welcome oddshocks 18:27:29 <abadger1999> <nod> 18:27:36 * mapyth is pissed by my troublesome internet connection 18:28:22 <ootbro> (went business-class for my home connection and it's usually very stable) 18:28:46 <nirik> ok, lets see how nagios treated us this last week... 18:28:53 <nirik> #topic nagios/alerts recap 18:28:54 * threebean cringes 18:29:01 <nirik> https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=5&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=5&eday=15&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=2&hoststates=3&servicestates=56&limit=25 18:29:09 <nirik> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=5&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=5&eday=15&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=2&hoststates=3&servicestates=56&limit=25 18:29:10 <zodbot> nirik: http://tinyurl.com/q8j48o9 18:29:33 <nirik> yeah, the new fedmsg monitoring was a bit shouty. ;) 18:30:04 <nirik> but I think we have that mostly tuned better now? 18:30:10 <henderbj> mirrorlist-serverbeach is always swaping? 18:30:20 <threebean> yeah.. there were actual problems it was reporting in the beginning.. those seem mostly worked out, but the periodic UNKNOWNs from badges-backend01 is still a mystery. 18:30:23 <nirik> ha. Just got another feedmsg alert. ;) 18:30:57 <nirik> henderbj: yeah, it's proving troublesome. I am not sure why it's having trouble where the other instances aren't. ;( I guess I could just destroy it and make one somewhere else. 18:31:23 <nirik> I've tried various things to make it happier. (Fewer threads, etc) 18:32:10 <henderbj> nirik: Maybe some processes are running there that are not present on those others 18:32:14 <nirik> and the telia stuff is typical. Our phx2 main datacenter to telia often has network issues. 18:32:33 <henderbj> nirik: maybe some backups? 18:32:40 <nirik> henderbj: very unlikely. They are configured from the same ansible playbook, so they should be pretty much identical. 18:32:49 <henderbj> nirik: ok 18:33:24 <nirik> The main difference is that is in another datacenter on different hardware. 18:33:33 <henderbj> Ah... i finished (at least i think) work for ticket #4325: https://fedorahosted.org/fedora-infrastructure/ticket/4325 18:33:37 <smooge> and the serverbeach hardware can at times get odd 18:33:39 <nirik> so it could be that that hw/network just sucks. ;( 18:33:57 <nirik> henderbj: cool. ;) I saw, but haven't had time to look yet. 18:34:35 <henderbj> nirik: Please check it and if it works, then we are done and can close the ticket 18:35:10 <nirik> so, on nagios: tune fedmsg alerts more and figure out badges-backend timeout, move or do something with mirrorlist-serverbeach, and sigh at telia. ;) 18:35:16 <nirik> it's on my list, yep. 18:35:38 <nirik> #topic Upcoming Tasks/Items 18:35:38 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:35:47 <nirik> anyone have upcoming items they want to note or schedule? 18:36:27 <threebean> Oh, I'll be out on vacation tomorrow. 18:36:27 <nirik> we have our FAD coming up in just a few weeks... 18:36:36 <nirik> https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014 18:36:38 <threebean> also Monday is a US holiday, so things might be quiet around then too. 18:37:04 <nirik> oh yeah, true... 18:37:36 <threebean> i'm not actually travelling, so I might poke my head into channel here and there.. :) 18:38:00 * nirik should be around, but might be playing video games or watching movies or whatever. ;) 18:38:04 <nirik> #topic Open Floor 18:38:24 <nirik> anyone have any items for open floor? questions? comments? 18:38:34 <henderbj> nirik has time to play video games??? 18:38:40 <ootbro> (crickets chirping) 18:38:42 <nirik> sometimes. ;) 18:38:59 <nirik> on weekends. If I get to the gaming system before my GF does. ;) 18:38:59 <henderbj> ansible, and nagios wonderfull ;) 18:39:17 <ootbro> maybe you need two gaming systems? 18:39:41 <nirik> that would need 2 tv's... 18:39:55 <ootbro> space would be a problem with that 18:40:01 <henderbj> ho... or use multiseat consoles ;) 18:40:10 <nirik> yeah. Where's our virtual reality headsets! 18:40:22 <henderbj> i have a multiseat fedora machine... that's great thing to have ;) 18:40:42 <nirik> anyhow, lets continue over in #fedora-admin, #fedora-apps and #fedora-noc... Thanks for coming everyone! 18:40:44 <nirik> #endmeeting