19:00:03 <nirik> #startmeeting Infrastructure (2014-02-06) 19:00:03 <zodbot> Meeting started Thu Feb 6 19:00:03 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:03 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:03 <nirik> #meetingname infrastructure 19:00:04 <nirik> #topic welcome everyone 19:00:04 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 19:00:04 <zodbot> The meeting name has been set to 'infrastructure' 19:00:04 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 19:00:13 * abadger1999 here 19:00:20 * johe is here 19:00:22 * relrod here, but sick. :( 19:00:25 <mariocav> hi 19:00:31 <janeznemanic> hi guys 19:00:39 * ausmarton here 19:00:39 <nirik> relrod: :( 19:00:40 * willo says gidday 19:00:55 * lmacken 19:01:51 <nirik> #topic New folks introductions and Apprentice tasks 19:02:02 <nirik> any new folks like to introduce themselves? 19:02:08 <nirik> or apprentices with questions or comments? 19:02:43 <nirik> threebean and pingou are traveling today... 19:02:53 * maayke is here 19:03:57 <nirik> welcome maayke 19:04:33 * oddshocks here 19:05:10 <nirik> #topic Applications status / discussion 19:05:19 <nirik> any applications news or plans? 19:05:38 * nirik doesn't think too many of the applications folks are able to make it today 19:05:40 <lmacken> I'm wrapping up a few bug fixes now, but I plan on deploying a new bodhi release this week 19:06:06 <lmacken> fixing some of the odd editing race conditions :) 19:06:19 <nirik> awesome. 19:06:36 <nirik> lmacken: oh, I was going to make a bodhi-dev cloud instance wasn't I? I can do that today... for bodhi2 19:07:08 <lmacken> nirik: cool, sounds good. then we can test out the ansible playbook that janeznemanic wrote 19:07:45 <nirik> lmacken: yeah, I added some more comments there too... I guess we need to finish that up, then move on to releng stuff so we can move releng04/repel01 to ansible... 19:07:58 <nirik> there's lots of things on those that aren't really directly bodhi... 19:08:47 <lmacken> yeah, other than bodhi it'll be a bunch of cron jobs right? 19:08:56 <lmacken> some signing stuff maybe 19:09:13 <relrod> I'm hoping to have the Fedora Mobile side of FMN notifications done by the end of this weekend, provided I feel good enough to do anything :( 19:09:26 <nirik> yeah, cron jobs, signing stuff (client end), nfs 19:09:42 <nirik> relrod: cool. 19:09:54 <lmacken> relrod: very cool. I need to get my SDK setup again with my n5 so I can start playing with it again 19:10:18 <lmacken> I'm really excited to start getting fedmsgs to my phone :) 19:10:28 <relrod> :) 19:10:32 <nirik> so one thing I was wondering with FMN... but likely should ask threebean when he's around... 19:11:22 <nirik> could we use it for larger announcements someday? ie, push an email via it to notify everyone about some big issue? or to reset their passwords or something? 19:11:38 <nirik> just an idle thought 19:11:58 <lmacken> I know there is a plan to be able to have a certain group of people be able to send announcements 19:12:23 <nirik> cool. 19:12:54 <nirik> ok, any other applications news? 19:13:22 <nirik> #topic Sysadmin status / discussion 19:13:27 <lmacken> nirik: looks like fedmsg-announce already exists :) https://github.com/fedora-infra/fedmsg/blob/53e851f68f106bff919a591a3516f26d5b07c375/fedmsg/commands/announce.py 19:13:29 <nirik> so, on the sysadmin side... 19:13:39 <nirik> sweet 19:14:24 <nirik> I've been working to clean up all our ansible playbooks some more... we now have a daily cron job that runs --check --diff on all playbooks, so I have been working to move that output to 0. 19:14:44 <nirik> ie, make sure all playbooks are idempotent and hosts are synced up to them. 19:15:25 <nirik> if anyone wants to look, our ansible stuff is logging everything... on lockbox use '/srv/web/infra/ansible/scripts/logview -d today -s CHECK_DIFF:CHANGED -s CHECK_DIFF:FAILED | less' 19:15:57 <nirik> smooge will be in phx2 datacenter next week. 19:16:12 <smooge> yay me! 19:16:13 * ianweller joins from class 19:16:23 <willo> thks nirik, I just learnt a new word "idempotent". 19:16:23 <nirik> smooge: we should come up with a list/agenda for you... 19:16:37 <nirik> willo: yeah, it's a weird word, but important. ;) 19:16:46 <smooge> yes please.. otherwise I will just fight the alligators and forget the swamp. 19:17:13 <nirik> smooge: do you know if any new hw will get there in time for you being there? (I think it unlikely, but... ) 19:17:54 <sart> (I missed the introductions due to a phone call, but I'm here, by the way) 19:19:05 <nirik> hey sart. Welcome. :) are you more interested in sysadmin or application devel type things? or both? 19:19:07 <smooge> nirik, no idea 19:19:20 <smooge> I am expecting it to all arrive after I leave 19:19:33 <sart> both, but I'm somewhat less of a noob in sysadmin 19:19:49 <nirik> smooge: yeah, me too, thats the way it goes. ;( 19:20:05 <nirik> sart: great. see me in #fedora-admin after the meeting and I can help get you started. 19:20:37 <sart> awesome, thanks for the welcome nirik 19:20:58 <nirik> no problem. :) 19:21:14 * SmootherFrOgZ here 19:21:21 <nirik> hey SmootherFrOgZ. ;) 19:21:50 <nirik> so, on other sysadmin stuff, we have some new hardware that will be wandering in over the next few weeks... 19:22:46 <nirik> Oh, and I tried tuning our pkgdb and elections stuff recently to stop us getting alerts on it all the time 19:22:54 <nirik> so far it seems to be working. Knock on wood. 19:23:01 <smooge> heck I am not sure the PPC hardware will get there when I am to be there 19:23:16 <abadger1999> cool. 19:23:17 <nirik> smooge: you might take some dvd's there to leave... 19:23:57 <nirik> oh, this is likely not an easyfix, but would be great if someone could come up with it: we need a glusterfs role for ansible. ;) 19:25:03 <nirik> Anyhow thats all that leaps to mind.. 19:25:17 <nirik> #topic Upcoming Tasks/Items 19:25:17 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 19:25:26 <nirik> any upcoming items folks would like to note or schedule? 19:25:48 <nirik> There may be netapp work next tuesday. They have pushed it back twice tho, so it could get pushed again. 19:27:08 <nirik> #topic Open Floor 19:27:11 <nirik> anything for open floor? 19:27:18 <SmootherFrOgZ> yep 19:27:21 <johe> me too 19:27:28 <nirik> Oh, one thing if we had enough folks interested, we could talk about hotfixes in ansible sometime. 19:27:35 <nirik> SmootherFrOgZ: go ahead and then johe. ;) 19:27:37 <johe> +1 19:28:09 <willo> +1 hotfixes in ansible 19:28:21 <mariocav> +1 19:28:50 <SmootherFrOgZ> I started working on FAS 3.0. You can check the branch on github. and also in a process of writing down everything note (technical & functional) I had to people can have a look on what's coming 19:28:59 <SmootherFrOgZ> this is for end of this week 19:29:16 <webpigeon> link? :) 19:29:27 <nirik> SmootherFrOgZ: cool! 19:29:35 <lmacken> SmootherFrOgZ: excellent 19:29:58 <nirik> https://github.com/fedora-infra/fas/tree/FAS_3.0 19:29:59 <SmootherFrOgZ> I choosed pyramid as framework 19:30:08 <SmootherFrOgZ> choose* 19:30:09 <webpigeon> thanks nirik :) 19:30:49 <lmacken> SmootherFrOgZ: cool. It'll be nice to have a test suite for fas this time around ;) 19:30:56 <SmootherFrOgZ> So if you guys have any ideas of features you would like to talk/add/whatever, go to the issues' page 19:31:20 <SmootherFrOgZ> lmacken: definitely! 19:31:31 <nirik> will do. ;) 19:32:07 <nirik> SmootherFrOgZ: thanks. Anything else on this? 19:32:50 <SmootherFrOgZ> yeah, last flock we talked about permission for specific access from apps, fas 3.0 will be (i guess) the first one to have it :) 19:33:17 * SmootherFrOgZ done 19:33:32 * relrod was one thing - today is my two-year anniversary at Red Hat. I just want to give a huge thank you to the entire team, you've been awesome, and I could not ask for a better job. 19:33:34 <SmootherFrOgZ> for anuthing else come talk to me 19:33:35 <relrod> s/was/has/ 19:33:36 <nirik> nice. 19:33:46 <SmootherFrOgZ> anything* 19:33:51 <nirik> relrod: congrats! very nice to have you around! 19:33:59 <nirik> johe: you had an item? 19:34:01 <SmootherFrOgZ> relrod: congrats! 19:34:03 <johe> well, i thought some weeks ago, someone told he would like to do a network map or layout, any news on that? i would find it useful to see how machines interact, like what is redundant, how is it balanced, and so one 19:34:05 <willo> relrod: awesme 19:34:06 <relrod> thanks :) 19:34:10 <johe> relrod: gz 19:34:39 <willo> johe: that was me 19:34:48 <willo> still working on it. 19:35:04 <johe> willo: ah, okay, do you need some help? 19:35:09 <nirik> :) there is the nagios map, but yeah, it's not so good in some senses. 19:35:26 <nirik> I'd be happy to try and explain things after meeting (time permitting) 19:35:42 <johe> for example last week or so i came across hosted03 when it swapped, according to nagios there is a hosted04 but i dont know if the are redundant or standalone 19:36:04 <johe> nirik: okay 19:36:26 <mariocav> nirik: are the chats of #fedora-admin logged and publicly available? 19:36:27 <nirik> hosted04 is a 'warm spare' (it syncs data from 03 via rsync). So, if 03 died we could bring up 04 pretty quickly (hopefully) 19:36:48 <nirik> mariocav: there's no public logs that I know of. Some people might log in their clients... 19:37:00 <johe> nirik: but if i would restart apachte on 03 , 04 would take over? 19:37:10 <johe> -t 19:37:19 <nirik> johe: nope, the process is manual. It needs someone to switch dns to 04. 19:37:20 <willo> johe: sure, but I really wanted to something up first for others to then add to 19:37:39 <johe> nirik: willo okay 19:37:42 <nirik> They are in different datacenters. ;) 19:37:56 <willo> :) 19:37:57 <johe> i see 19:38:28 <willo> nirik: do you use nagios to trigger events on host down 19:38:43 <willo> such as do that dns switch for you automatically 19:39:04 <nirik> willo: not currently no. It could be pretty complex to make that happen... 19:39:10 <johe> such an automatication is not always useful 19:39:21 <nirik> and what happens if it drops and comes back up 30sec later? 19:39:27 <nirik> or was always up, but unreachable. 19:39:35 <nirik> (from just noc01) 19:39:41 <willo> yeah, true 19:39:55 <nirik> we do have nagios doing a few triggered actions... 19:40:01 <willo> interesting point that 19:40:05 <nirik> it can restart httpds in some cases. 19:40:06 <willo> have you heard of shinken 19:40:30 <nirik> not off hand 19:40:38 <willo> nagios rewrite in python 19:40:52 <nirik> huh, cool. 19:40:56 <willo> deals with unreachable from noc01 issue 19:41:09 <willo> check can be run distributed 19:41:40 <nirik> sure, there's always ways and tradeoffs. 19:41:55 <willo> so, you can check a host from another host that better aligns with the way your traffic flows 19:42:14 <nirik> but then you have to run a monitoring agent on it and allow access, etc... 19:42:58 <johe> could turn out in a very long discussion 19:43:06 <willo> true, and true 19:43:33 <nirik> anyhow, anything else general for the meeting? or shall we continue in #fedora-admin, #fedora-noc and #fedora-apps? 19:43:37 <willo> used nagios for years, now a big fan of shinken 19:44:00 * nirik is waiting for http://linux-ha.org/source-doc/assimilation/html/index.html to be usable. ;) 19:44:22 <willo> are drinks a regular occurence 19:44:28 <willo> I came prepared 19:44:47 <webpigeon> willo: I can't fine any licence infomation on the site? 19:44:51 <nirik> I'm having some lovely vanilla coffee myself. ;) 19:45:39 <nirik> webpigeon: AGPL it looks like 19:45:44 <willo> webpigeon: https://github.com/naparuba/shinken 19:45:51 * relrod should make himself some tea with honey and lemon and hope that it helps re: being sick :/ 19:45:54 <nirik> http://www.shinken-monitoring.org/participate/ (look down at the bottom) 19:46:27 <webpigeon> nirik/willo: thanks :) 19:46:39 * nirik notes it's not packaged in fedora/epel that I see. 19:46:57 <willo> it is, or maybe was 19:47:08 <nirik> anyhow will close out the meeting in a minute if nothing else... 19:47:11 <webpigeon> couldn't see COPYING or LICENCE in source :P 19:47:33 <willo> Slow Screw up against a Red Wall -- http://www.webtender.com/db/drink/6130 19:48:08 <nirik> alright, thanks for coming everyone! 19:48:11 <nirik> #endmeeting