19:00:18 <nirik> #startmeeting Infrastructure (2011-06-09)
19:00:18 <zodbot> Meeting started Thu Jun  9 19:00:18 2011 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:18 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:18 <nirik> #meetingname infrastructure
19:00:19 <nirik> #topic Robot Roll Call
19:00:19 <nirik> #chair goozbach smooge skvidal codeblock ricky nirik abadger1999
19:00:19 <zodbot> The meeting name has been set to 'infrastructure'
19:00:19 <zodbot> Current chairs: abadger1999 codeblock goozbach nirik ricky skvidal smooge
19:00:26 * ricky 
19:00:26 * skvidal is here
19:00:34 * tideline here
19:00:36 * pingou around
19:00:38 * StylusEater is here
19:00:41 * CodeBlock silently wonders if nirik's lowercase 'codeblock' still makes me a chair.
19:00:46 <goozbach> present
19:00:48 <goozbach> http://goo.gl/doodle/rsSl
19:00:49 * athmane hello everyone
19:00:53 <LoKoMurdoK> hello
19:02:10 <nirik> morning all
19:02:56 <abadger1999> hello
19:03:01 <nirik> #topic New folks introductions / apprentice stuff
19:03:38 <nirik> anyone want to say hi?
19:03:45 <pingou> hi :)
19:03:47 <nirik> any apprentice thoughts?
19:03:57 <tideline> is there a current list of hosts managed by this team somewhere one the wiki
19:04:21 <ricky> It's all in puppet, look at puppet/manifests/nodes
19:04:35 <LoKoMurdoK> Hi, people I'm New Apprentice
19:04:44 <CodeBlock> ^ or nagios or virthost-list
19:04:56 <CodeBlock> virthost-lists.out*
19:05:06 <nirik> welcome LoKoMurdoK
19:05:07 <tideline> on which hosts?
19:05:09 <athmane> hello, another one from fedora-qa
19:05:31 <LoKoMurdoK> I am happy to belong to the team
19:05:35 <LoKoMurdoK> tks
19:05:43 <CodeBlock> tideline: /var/log/virhost-lists.out and what ricky said are both puppet01, nagios is http://admin.fpo/nagios
19:05:56 <tideline> CodeBlock: thanks
19:06:04 <skvidal> CodeBlock: virthost-lists doesn't cover physical hosts which are NOT xen/kvm hosts
19:06:17 <CodeBlock> truedat.
19:06:28 <nirik> there's a sort of list in the mass update wiki page too...
19:06:35 <skvidal> tideline: /tmp/complete-minion-list on puppet1
19:06:39 <skvidal> is the total list
19:06:39 <nirik> but ideally we would get rid of that in favor of something thats auto-generated. ;)
19:06:42 <skvidal> as of 30s ago
19:06:48 <skvidal> nirik: wilco
19:06:54 <skvidal> nirik: when infra.fp.o is hot
19:06:55 <athmane> I've updated the wiki page about Nagios alert access
19:07:04 <skvidal> we should just have a location it is available in /srv/infra/
19:07:07 <skvidal> that you can simple ls
19:07:16 <nirik> yeah.
19:07:19 <LoKoMurdoK> I am currently working with the disclaimer of the planet, I'm a systems administrator in Panama, and thanks for the welcome.
19:07:27 <CodeBlock> Heh my clone died/they finally killed that vps that I stopped paying for like 2 months ago?
19:07:28 <skvidal> LoKoMurdoK: you're Luis?
19:07:30 <LoKoMurdoK> yes
19:07:33 <LoKoMurdoK> Luis
19:07:58 <LoKoMurdoK> irc: LoKoMurdoK , FAS. lbazan,
19:08:02 <nirik> NOTE to apprentice folks: I am going to go thru the group later today/tonight and remove folks who didn't reply to my email on the 2nd. :) Nothing personal, and we can re-add folks as they get time to work on things....
19:09:02 <tideline> nirik: have you send the email?
19:09:08 <tideline> I didn't get ine
19:09:10 <tideline> on*
19:09:10 <pingou> nirik: 2nd of June ?
19:09:11 <nirik> tideline: I sent email on the 2nd...
19:09:21 <nirik> yes. If you were added after that, no problem.
19:09:25 <goozbach> nirik: I'm still in said group, don't think I replyed to the email thought :)
19:09:26 <tideline> I dont think I was in there on the 2nd
19:09:33 <tideline> nirik: cool
19:09:58 <nirik> goozbach: please do if you get a chance. ;)
19:10:05 <athmane> the same here (< 1 week)
19:10:05 <nirik> Subject was: "June Status update for Fedora Infrastructure Apprentices"
19:10:12 <LoKoMurdoK> skvidal, yes I'm Luis
19:10:14 <nirik> so, no worries... you will get one in July. ;)
19:10:29 <nirik> ok, any more apprentice/new folks business? any questions? or shall we move on to the next topic?
19:10:39 <pingou> nirik: infra list ?
19:11:01 <nirik> pingou: directly to fi-apprentice-members
19:11:06 <pingou> ah ok
19:11:26 <skvidal> LoKoMurdoK: nod - cool
19:11:28 <nirik> ok, moving on.
19:11:41 <nirik> #topic Upcoming Events / Tasks
19:12:11 <nirik> I'm just going to info all these:
19:12:41 <nirik> #info 2011-06-09 Remove inactive fi-apprentice people.
19:12:41 <nirik> #info 2011-06-14 or so: post release housecleaning tasks.
19:12:42 <nirik> #info 2011-06-14 Class B mass reboots
19:12:42 <nirik> #info 2011-06-15 Class A mass reboots.
19:12:42 <nirik> #info 2011-06-17 FPCA drop dead.
19:12:42 <nirik> #info 2011-07-01 mail fi-apprentice folks.
19:12:44 <nirik> #info 2011-07-01 BLOGS closing time.
19:12:46 <nirik> #info 2011-07-11 - 14: smooge and nirik at phoenix
19:13:06 <nirik> Does anyone have questions on those or other upcoming items we should plan for?
19:13:12 <skvidal> 2010-07-11-14: Seth On PTO
19:13:13 <skvidal> :)
19:13:29 <skvidal> actually....
19:13:34 <nirik> nice. ;)
19:13:36 <skvidal> is that going to be a problem?
19:13:43 <nirik> I don't think so off hand.
19:13:49 <skvidal> on the plus side y'all will be RIGHT NEXT to most of the servers
19:13:55 <skvidal> on the minus side you might not be on hand on irc, etc
19:14:00 <nirik> yeah.
19:14:07 <CodeBlock> I should be around.
19:14:10 * ricky will be around
19:14:12 <skvidal> CodeBlock: cool.
19:14:12 <nirik> as long as some other main folks are around I think we will be ok
19:14:16 <skvidal> good
19:14:19 <skvidal> b/c it's eunice's b-day
19:14:20 <smooge> well then skvidal you are allowed to go :)
19:14:25 <skvidal> and I don't htink I can get out of it :)
19:14:29 <skvidal> not w/o a lot of pain
19:14:29 <skvidal> :)
19:14:33 <smooge> 2010? or 2011
19:14:39 <CodeBlock> haha
19:14:54 <CodeBlock> smooge: He's going to go back in time, take PTO, and then rejoin us :)
19:14:58 <pingou> one year past already
19:15:03 <nirik> :)
19:15:05 <smooge> ah man that is bad
19:15:33 <skvidal> wow
19:15:33 <skvidal> yah
19:15:34 <skvidal> 2011
19:15:37 <pingou> ^^
19:15:37 <skvidal> <sigh>
19:15:52 <smooge> well have to head out now bbl
19:15:58 <CodeBlock> have fun
19:16:00 <nirik> have fun smooge
19:16:00 <ricky> Enjoy :-)
19:16:04 <nirik> #topic Meeting tagged tickets:
19:16:05 <nirik> https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assigned&status=reopened&group=milestone&keywords=~Meeting&order=priority
19:16:18 <nirik> any meeting tagged tickets folks would like to talk about?
19:16:33 <ricky> .ticket 2517
19:16:34 <zodbot> ricky: #2517 (Need mod_evasive for EL6) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2517
19:16:40 <ricky> can we just close that, since we've been doing fine without?
19:16:41 <abadger1999> One thing on FPCA -- skvidal, wanna send out another nag mail?
19:16:54 <skvidal> abadger1999: do we want to send out another one?
19:16:59 * skvidal looks for spot
19:16:59 <nirik> ricky: fine with me.
19:17:01 <skvidal> I'm fine with it
19:17:14 <abadger1999> We're much improved now but still over a thousand packages will be orphaned.
19:17:16 * skvidal was not replying to mod_evasive
19:17:19 * skvidal was replying to abadger1999
19:17:22 <skvidal> sorry
19:17:37 <skvidal> abadger1999: wow, sounds like a win
19:17:37 <skvidal> :)
19:17:43 <abadger1999> http://toshio.fedorapeople.org/fpca3/
19:17:46 <abadger1999> skvidal: haha
19:17:46 <skvidal> abadger1999: b/c those 1K pkgs are obviously not maintained :)
19:18:07 <nirik> it looks like about 2/3rds of people are signed now. (fedorapeople accessing folks that is)
19:18:07 <abadger1999> True dat
19:18:38 <nirik> athmane: you wanted to talk about ticket 308 some?
19:18:46 <athmane> nirik, yes
19:18:51 <nirik> #action send out another fpca nag... can do out of band
19:19:58 <skvidal> abadger1999: it's http://toshio.fedorapeople.org/fpca3/union_important_users.txt right?
19:20:05 <abadger1999> skvidal: Correct
19:20:10 <skvidal> abadger1999: I can send out the same email I did the other day, if spot is okay w/ir
19:20:13 <skvidal> s/ir/it/
19:20:26 <abadger1999> skvidal: Perfect.  Thanks
19:20:34 <nirik> athmane: ok, so you have a proposed policy, etc...
19:20:44 <skvidal> abadger1999: I'll email spot and cc you
19:20:53 <athmane> yes, tested with Drupal but not yet with MW
19:21:12 <athmane> also when i looked into puppet
19:21:15 <nirik> athmane: yeah, we will want to possibly next roll out to stg and test all the apps?
19:21:39 <athmane> each httpd has mod_sec but disabled (rules commented)
19:21:47 <skvidal> abadger1999: I just want spots okay b/c it has his name on it :0
19:22:04 <ricky> Yeah, we actually ran into issues with mod_security the last time we enabled it
19:22:18 <ricky> Which is why we want to be really careful with the rules this time - start them very permisive
19:22:38 <athmane> nirik, the default rules-set are PITA
19:22:43 <ricky> Will mod_security live on the app or proxy servers?
19:23:02 <skvidal> :)
19:23:19 * skvidal smiles at nothing in particular (sorry)
19:23:19 <athmane> proxies are haproxy based ?
19:23:34 <ricky> skvidal: No need to be sorry for being happy :-)
19:23:43 <nirik> yeah, I would like to see a cautious approach... test in stg and see everything looking ok, then deploy on ONE proxy/app and test live for a while before rolling to the others.
19:23:48 <ricky> athmane: The proxies run apache with varnish and haproxy in front of them (yeah, it's a little copmlex)
19:24:31 * ricky will say that he personally is not a fan of stuff like mod_security, but thanks for working on it nevertheless :-)
19:24:55 <athmane> ricky, me too :)
19:25:22 <skvidal> what problem is mod_security solving for us currently?
19:25:22 <StylusEater> ricky: this probably isn't the forum but I'd like to hear why ... maybe in admin later?
19:25:35 <skvidal> s/currently/in the future/
19:25:49 <athmane> maybe 0-days in some apps ?
19:25:52 <ricky> StylusEater: Sure, although it seems like skvidal is kind of getting at it too
19:26:07 <skvidal> we don't need to discuss it here
19:26:07 <skvidal> sorryt
19:26:21 * nirik hasn't used mod_security too much...
19:26:37 <nirik> yeah, looks like it could help us with new attacks/0-days/things our apps are not yet hotfixed for.
19:26:39 <ricky> As a blacklisting-type thing, there are always known ways to bypass it floating around - it's not even too hard when you can see the rules.  At the same time, false positives create pain for admins
19:27:11 <ricky> So that's why I'm not crazy about it - I try to fix problems at the core (hotfixing) when I see them.
19:27:38 <ricky> Does mod_security have a logging-only configuration?
19:27:41 <skvidal> zodbot: .whoowns mod_security
19:27:42 <ricky> That I wouldn't mind seeing.
19:28:04 <StylusEater> ricky: can't we protect the list by making it "private" ? maybe that flies in the face of what we do, but ...
19:28:06 <ricky> (Definitely as a required first step if we plan on deploying it further, I think)
19:28:09 <athmane> ricky, yes you can make default action to log
19:28:39 <skvidal> hmm
19:28:52 <skvidal> so mod_security is one of the pkgs impacted if its owner doesn't sign the fpca soon
19:29:09 <ricky> StylusEater: Eh, I'm not crazy about the security of it depending on the configs being private - attackers have all the time in the world to try permutations to get past the rules.
19:29:36 <ricky> I wouldn't mind seeing mod_security + a plan where we actually monitor the logs from it
19:30:12 <ricky> We currently don't do a great job of monitoring and addressing problems in logs...  as a result, some of our logs are like, 50% tracebacks (just made that number up)
19:30:25 <ricky> Which is largely my fault with FAS's case, but just saying :-)
19:30:41 <skvidal> so
19:30:49 <skvidal> we've talked about this at fudcon and in here
19:31:07 <skvidal> but ironing out these issues are all pieces of the whole problem
19:31:52 * athmane notes that mod_sec logs are huge
19:32:15 <skvidal> do we need mod_security immediately? is there a known threat?
19:32:26 <nirik> yeah, handling logging would be nice.
19:32:41 <nirik> skvidal: no, other than it's something athmane knows and was willing to work on.
19:32:58 <nirik> perhaps we could refocus on log manageing/reporting?
19:33:01 <athmane> not sure but you should ask mmcgrath (he opened that ticket)
19:33:08 <skvidal> okay - my concern is that it sounds like mod_security is going to need a lot more effort to do properly
19:33:19 <skvidal> and a bunch of other infrastructure needs to be better prepared for it
19:33:28 <skvidal> does that sound about right?
19:33:32 <StylusEater> skvidal: yes
19:33:49 <nirik> yeah.
19:33:59 <ricky> Yup, log processing being the biggest one
19:34:16 <skvidal> right
19:34:28 <StylusEater> ricky: we currently point rsyslog to a db?
19:34:37 <skvidal> it's not at  a db, is it?
19:34:40 <ricky> Nope, rsyslog to flat files
19:34:40 <skvidal> it's to disk
19:34:48 <skvidal> ricky: well a hierarchy of them
19:34:53 <skvidal> not all glommed together.
19:35:02 <skvidal> s/not all/not only all/
19:35:06 <ricky> Oops, yeah, misused the word "flat" there.
19:35:33 <athmane> other idea is to use a local log scanner, like http://code.google.com/p/apache-scalp/
19:35:42 <athmane> and php-ids defs
19:35:56 <skvidal> so it feels a bit like we're off in the weeds here
19:36:04 <nirik> yeah, how about this:
19:36:20 <nirik> - lets defer mod_security for now until we have more handle on log management.
19:36:32 <StylusEater> nirik: good idea
19:36:37 <athmane> works for me
19:36:41 <nirik> - lets file a ticket/see if athmane or others are willing to work on log management/reporting. ;)
19:36:54 <nirik> possibly working on epylog, or other similar things.
19:37:14 <marchant> i do log maintenance and reporting as part of my $dayjob
19:37:21 <skvidal> marchant: what do you use?
19:37:23 <marchant> but env is very different
19:37:25 <nirik> athmane: thoughts?
19:37:29 <marchant> splunk
19:37:39 <nirik> marchant: cool. :)
19:37:39 <skvidal> yah - that's not gonna fly
19:37:43 <athmane> marchant, the same here
19:37:45 <marchant> right
19:38:01 <marchant> it is a great tool, but obviously not for fpo
19:38:08 <marchant> the concept is great
19:38:12 <athmane> I can do regulat checks etc (httpd logs, clamav, rkhunter etc..)
19:38:18 <athmane> **regular
19:38:26 <marchant> and it puts all logs into a central db for foresics and reporting
19:38:34 <skvidal> understood
19:38:38 <skvidal> it's not free software
19:38:41 <skvidal> so running it on our boxes
19:38:44 <skvidal> is a non-starter
19:38:46 <marchant> no it is very expensive
19:38:50 <nirik> right. we need a free solution. ;)
19:39:02 <marchant> i definitely understand that
19:39:06 <skvidal> so - epylog is the only semi-maintained solution that I know of that's not full of pain and agony
19:39:12 <skvidal> s/full/completely full/
19:39:22 <nirik> so, would you guys be willing to work on this? we can discuss details out of meeting?
19:39:31 <athmane> mod_sec log analyzer tool also not foss
19:39:38 <marchant> i am definitely willing
19:40:13 <marchant> i am still learning
19:40:14 <athmane> nirik, ok
19:40:22 <nirik> skvidal: can you file a ticket (although, do we already have one?) with epylog info, etc?
19:40:32 <skvidal> I'll look if we have one
19:40:35 <skvidal> other wise, yes
19:40:57 <nirik> thanks.
19:41:06 <nirik> ok, any other meeting tickets?
19:41:13 <nirik> (that we want to discuss?)
19:41:40 <marchant> can i ask what will probably be mundate questions about the easyfix tickets in fedora-noc during the day?
19:41:52 <nirik> absolutely... or fedora-admin...
19:41:54 <ricky> Yes, feel free :-)
19:42:00 <nirik> #topic Open Floor
19:42:05 <nirik> anything for open floor?
19:42:09 <marchant> i hate to get in the way of stuff that you are working on during the day
19:42:24 <marchant> wasn't sure if that was a good platform
19:42:28 <nirik> marchant: no problem... if people are busy they likely just won't answer. ;)
19:42:36 <marchant> is there an apprentice irc
19:42:37 <nirik> and then can answer when they get time.
19:43:02 <skvidal> marchant: ask in #fedora-admin if you have things
19:43:10 <skvidal> I thin kit is entirely welcome and appropriate
19:43:17 <nirik> oh, for main folks... if anyone wants to help with class "C" host updates, please see me... it would be good to get those done as time permits before the other ones next week.
19:43:32 * nirik seconds skvidal's response.
19:43:33 <skvidal> nirik: puppet changes/moving things around - smooge seems down w/it - are you okay w/me moving servergroups into services?
19:43:44 <nirik> yep. Make it so
19:43:45 <StylusEater> nirik: class C .. low impact hosts ... no?
19:44:07 <nirik> StylusEater: right. low impact or some measure of HA, so no end user impact
19:45:04 <nirik> ok, lets continue over in #fedora-admin or #fedora-noc... thanks for coming everyone!
19:45:13 <nirik> #endmeeting