18:59:59 <nirik> #startmeeting Infrastructure (2013-03-07)
18:59:59 <zodbot> Meeting started Thu Mar  7 18:59:59 2013 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:59:59 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:59:59 <nirik> #meetingname infrastructure
18:59:59 <zodbot> The meeting name has been set to 'infrastructure'
18:59:59 <nirik> #topic welcome everyone
18:59:59 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
19:00:00 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
19:00:21 <nirik> who all is around for a infrastructure meeting?
19:00:23 * lmacken 
19:00:35 * cyberworm54 here
19:00:37 * abadger1999 here
19:00:40 <blackdeerranger> here
19:00:41 * maayke here
19:00:48 <KasumiNinja> KasumiNinja here
19:01:11 * threebean is here
19:01:24 * nirik will wait another min for folks to wander in.
19:02:03 * pingou 
19:02:22 <nirik> ok, I guess we can go ahead and dive in. ;)
19:02:28 <nirik> #topic New folks introductions and Apprentice tasks
19:02:42 <nirik> any new folks want to introduce themselves? or apprentices with questions?
19:02:50 <blackdeerranger> me
19:02:52 <KasumiNinja> I'm new
19:03:06 <ssmoogen> here
19:03:12 * SmootherFrOgZ is here
19:03:21 <mdomsch> hola
19:03:27 <KasumiNinja> I work as  a sysadmin and like to help fedora with sysadmin tasks
19:03:34 <nirik> blackdeerranger / KasumiNinja: welcome. ;)
19:03:39 <blackdeerranger> :)
19:03:47 <KasumiNinja> :-)
19:04:03 <nirik> blackdeerranger: are you also interested in sysadmin side of things? or more application development?
19:04:07 <KasumiNinja> I have no programming experience
19:04:21 <blackdeerranger> I typed my introduction here: http://paste.fedoraproject.org/4574/80565136/
19:04:45 <blackdeerranger> I´m interested in sysadmin tasks
19:04:59 <nirik> one thing I'd like to mention here... I sent out my regular fi-apprentice feedback email and got some email back that highlighted how self driven we require people to be...
19:05:02 <blackdeerranger> I´m not a software developer
19:05:37 <nirik> cool.
19:06:13 <cyberworm54> http://paste.fedoraproject.org/4574/80565136/
19:06:19 <cyberworm54> oops sorry :)
19:06:22 <nirik> I'm wondering if we want to try and change that focus any, or if there are ways to better communicate it to new folks so they don't get confused when no one is assigning them specific tasks and asking for updates all the time.
19:07:16 <nirik> perhaps I will start a discussion on the list around that, just wanted to mention it here.
19:07:18 <blackdeerranger> As I unterstand it is a good staring point to join "fi-apprentice "
19:07:22 <maayke> nirik: I will send you email tomorrow :)
19:07:28 <nirik> blackdeerranger: yeah.
19:07:33 <blackdeerranger> good
19:08:09 <nirik> the thing is that we are not setup to do heavy mentoring... so joining usually means the new person has to be very focused on just going and doing things and bugging people.
19:08:22 <KasumiNinja> works fine for me
19:08:25 * samkottler is just getting here
19:09:00 <skvidal> something worth mentioning
19:09:06 <nirik> anyhow, will post a thread to the list about it. ;) just brainstorming.
19:09:08 <skvidal> given where fedora infra is going with config mgmt and c&c
19:09:15 <skvidal> anyone interested in the sysadmin side of things
19:09:27 <skvidal> would be best served by going to ansible.cc
19:09:35 <skvidal> and trying it out for themselves to be familiar
19:09:44 <nirik> absolutely. ;)
19:09:54 <skvidal> we have lots of examples at:  http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/
19:10:08 <skvidal> but our examples are going to increasingly get more complicated, I suspect :)
19:10:28 <nirik> they always do.
19:10:29 <skvidal> so KasumiNinja, blackdeerranger: probably worth familiarizing yourself with this tool
19:10:55 <blackdeerranger> sounds interesting - thanks
19:11:09 <nirik> #info new folks would be advised to look into ansible.
19:11:16 <nirik> #topic Applications status / discussion
19:11:29 <KasumiNinja> great I'll look into it
19:11:29 <nirik> any exciting application side news this week or upcoming?
19:11:38 * pingou has mainly be bothering threebean
19:11:42 <nirik> #info fas-openid is now in production. Thanks puiterwijk. :)
19:11:43 <pingou> been*
19:11:51 <skvidal> KasumiNinja, blackdeerranger: great
19:11:53 <pingou> \ó/ well done puiterwijk
19:12:06 <threebean> we *just* got our first fedmsg message out of the secondary arch compose process
19:12:10 <abadger1999> Three cheers for puiterwijk :-)
19:12:16 <threebean> puiterwijk++
19:12:30 <nirik> #info fedmsg starting to be enabled for secondary arch compose processes.
19:12:48 <abadger1999> puiterwijk and I also discussed oauth to some length.  I'm organizing that into a message to send to the list.
19:12:57 <lmacken> I announced the fedmsg-notify today, so we'll hopefully see a lot more consumers
19:13:08 <abadger1999> An oauth server is what I think his next project is going to be.
19:13:19 <lmacken> I also fixed some long-standing push ordering bugs in the bodhi masher last week
19:13:36 <nirik> lmacken: yeah. Great post. ;)
19:13:41 <lmacken> thanks ☺
19:13:48 * Adran is here (late)
19:14:03 <nirik> #info fedmsg-notify blog post out there, hopefully more consumers
19:14:07 <pingou> nice lmacken
19:14:12 <nirik> lmacken: do we have any way to tell how many consumers there are?
19:14:18 <lmacken> https://admin.fedoraproject.org/haproxy/proxy01
19:14:19 <lmacken> and 02
19:14:19 <pingou> #link http://lewk.org/blog/fedmsg-notify.html
19:14:28 <lmacken> fedmsg-raw-zmq-outbound
19:14:29 <nirik> cool.
19:14:48 <pingou> lmacken: where is your blog plugin?
19:14:56 <lmacken> pingou: hrm?
19:15:02 <nirik> abadger1999: related. Have we given any thought about further expanding our 2 factor stuff to web apps or other places? Or haven't really explored that yet.
19:15:07 <abadger1999> mdomsch: Question -- is python-GeoIP still needed for mirrormanager?  I noticed that it was orphaned in Fedora/EPEL.
19:15:11 <pingou> lmacken: I saw tour twitter feed on your blog and I went... :)
19:15:45 <abadger1999> nirik: yes -- SmootherFrOgZ has been working up a patch to do 2-factor login to fas and support in python-fedora to take advantage of that.
19:15:53 <mdomsch> abadger1999: yep - largely used. I'll have to go adopt it
19:15:58 <abadger1999> nirik: I think that'll give us the first step.
19:16:28 <nirik> abadger1999: ah ha. I was confused as to what that work was about. Good.
19:16:38 <abadger1999> mdomsch: Cool.  If you need help, feel free to add me as a comaintainer and I'll put it on my once a cycle, look for updates list.
19:16:43 <nirik> note that also fas-openid has that PAPE plugin we could use for openid consuming applications.
19:16:49 <abadger1999> nirik: His work will be a first step.
19:17:03 <abadger1999> There's lots of other things that need to happen before it's "real"
19:17:21 <abadger1999> ie: at first we'll have both the single factor and 2-factor login.
19:17:28 <nirik> sure, yeah
19:17:36 <abadger1999> so you could circumvent by simply going to the other login page.
19:17:58 <abadger1999> But first we get it working, then we make it mandatory for some people.
19:18:08 * nirik nods.
19:18:16 <abadger1999> So -- continuing to make progress :-)
19:18:20 <lmacken> not sure if it was talked about last week, but we hit yet an issue with our tg1 apps where they would just silently block all requests. It happened after some dns change, but hopefully this change will fix it in the future. https://github.com/fedora-infra/python-fedora/pull/18
19:18:20 <nirik> also, that would only be yubikey?
19:18:49 <abadger1999> nirik: yeah -- since fas only knows about yubikey at this point.
19:18:54 <nirik> lmacken: ah yes, thanks for finding that. ;) We should try and get that fix rolled out to production...
19:18:58 <abadger1999> It's another thing we'll have to do to get this good to go.
19:19:13 <Adran> abadger1999: I might poke you later, I'd be interested in seeing if Google Authenticator could work, maybe I can work on something.
19:19:44 <abadger1999> Adran: cool.  It should.  We just have to integrate it more tightly with fas (we have basically a separate google authenticator setup right now)
19:19:54 <Adran> Ah.
19:20:01 <skvidal> Adran: could work where? we have support for google auth in totpcgi - it's available - just not integrated to fas
19:20:05 <skvidal> Adran: yes - what abadger1999 said :)
19:20:22 <nirik> so, next week is fudcon... how many of you will be out there?
19:20:24 <Adran> skvidal: Right. Maybe it can be integrated? :)
19:20:51 <threebean> nirik: s/fudcon/pycon/
19:20:58 <threebean> I'll be there :)
19:21:00 <nirik> yeah, sorry, brain failure. ;)
19:21:01 <lmacken> me too
19:21:50 <nirik> so, suggestion: we may want to be cautious about application changes tomorrow/the weekend... just so things are not possibly unstable with all you folks gone. ;)
19:22:37 <nirik> and hopefully you all have safe travels. :)
19:22:55 <nirik> any other application news?
19:23:10 <threebean> pingou started a really good conversation with hughsie about integrating tagger with AppStream
19:23:30 <threebean> and he's already written some code for it today.
19:23:38 <threebean> Pretty exciting.  :)
19:23:40 <nirik> cool. I saw some of that.
19:23:46 <nirik> I haven't looked at appstream yet much...
19:24:17 <nirik> #topic Sysadmin status / discussion
19:24:25 <abadger1999> I'll be there
19:25:03 <nirik> so, lets see... not too much on the sysadmin side that I can recall in detail. ;)
19:25:14 <skvidal> nirik: a few things
19:25:18 <nirik> #info ssmoogen reinstalled proxy01 as x86_64...
19:25:27 <nirik> (which is good, since it now matches all the other proxies)
19:25:32 <ssmoogen> and I think it worked
19:25:32 <skvidal> 1. we've been moving ahead on the ansible conversion
19:25:46 * mdomsch is going to need mmbapp01 to be x86_64 too due to s3cmd memory usage
19:25:59 <skvidal> 2. I've added a path lookup plugin to ansible that will allow us to have lookups for staging then production like we do now in puppet
19:26:19 <threebean> nice!
19:26:31 <pingou> very nice
19:26:36 <nirik> skvidal: yep. :) we need to test out some workflow there/make some simple playbooks for simple hosts to test things out.
19:26:48 <nirik> mdomsch: :( oh well... we can do that.
19:26:51 <skvidal> 3. more things moving into openstack and we're already running into limits  of our available resources
19:27:11 <pingou> oh :(
19:27:14 <skvidal> I have 4 more instances to finish transitioning and then we can start moving systems over to increase the available resources
19:27:30 <skvidal> which is great b/c that should double our available resources
19:28:01 <skvidal> nirik: for later - we might consider tinkering with multiple availability zones and osuosl02
19:28:04 <nirik> and add cinder volumes from each of the compute nodes.
19:28:15 <nirik> yep. That would be great.
19:29:02 <nirik> so, that bug that lmacken mentioned earlier... I was wondering, should we make app servers depend on their local proxy ?
19:29:19 <nirik> right now they hit admin.fedoraproject.org, which is dns round robin for all of them.
19:29:28 <nirik> but if they talked to their local proxy it might be faster.
19:29:36 <nirik> and also if that proxy is down, then likely they are too.
19:29:59 <lmacken> so appX->proxyX?
19:30:34 <nirik> yeah.
19:30:38 <ssmoogen> #info smooge is waiting to find out if physical memory arrived in PHX2 so we can give it to systesm
19:30:42 <nirik> app01/02/03/04 would hit proxy01
19:31:32 <nirik> anyhow, it's a thought. I don't think it's urgent.
19:31:46 <skvidal> nirik: query on this
19:32:04 <skvidal> would it make more sense for us to pursue the above? or for us to pursue breaking all the apps out?
19:32:36 <nirik> well, the above is pretty trivial if we want to do it. ;) Breaking apps out is still a longer term thing we really must do, IMHO.
19:32:54 <skvidal> nirik: fair enough..
19:33:02 <skvidal> if the proxies were running next to the app servers
19:33:07 <skvidal> like on the same hw
19:33:10 <skvidal> I';m inclined to say yes
19:33:22 <nirik> and I think breaking apps out kinda wants ansible to be ready to handle those apps.
19:33:25 <skvidal> but we don't really want an outage on proxy01 to kill the app servers
19:33:35 <nirik> yeah, true.
19:33:50 <nirik> well, lets leave it for now...
19:34:17 <nirik> #topic Private Cloud status update / discussion
19:34:20 <abadger1999> About the private cloud -- does this mean we're going to just use openstack going forward?
19:34:24 <nirik> we already hit on some of this above...
19:34:33 <skvidal> abadger1999: so right now here is what we have discussed
19:34:43 <skvidal> abadger1999: 1. moving the instances we have over to openstack
19:34:54 <skvidal> 2. moving 2 of the compute nodes over to openstack for additional resources
19:35:04 <skvidal> 3. taking the remaining 2 machines for other prototyping/testing
19:35:22 <skvidal> whether 3 is of openstack or of $something_else is really up to later discussion/decision
19:35:32 <abadger1999> <nod>
19:36:03 <ssmoogen> would adding another proxy to phx2 help?
19:36:07 <abadger1999> Okay so it seems like our "production" services are moving openstack but we're still testing out the alternatives ?
19:36:13 <nirik> #info moving things into openstack cloudlet
19:36:53 <nirik> #info will move some compute nodes from the other cloudlet over to it, and then have 2 nodes to continue testing other things with.
19:37:08 <nirik> abadger1999: yeah, or possibly we will use the other 2 to test the next openstack version...
19:37:14 <abadger1999> <nod>
19:38:08 <nirik> I think we can knock a bunch of things off our 'need to figure out before production' list now too.
19:38:24 <nirik> I'll look at cleaning up the wiki page on that, since I think we solved or decided many of them
19:38:50 <nirik> anything else on cloudlets?
19:39:04 <skvidal> no - but I had something for openfloor when that happens
19:39:28 <nirik> ok
19:39:33 <nirik> #topic Upcoming Tasks/Items
19:39:38 <nirik> #info 2013-03-07 remove inactive apprentices.
19:39:39 <nirik> #info 2013-03-12 to 2013-03-21 pycon
19:39:39 <nirik> #info 2013-03-19 to 2013-03-26 - koji update
19:39:39 <nirik> #info 2013-03-29 - spring holiday.
19:39:39 <nirik> #info 2013-04-02 to 2013-04-16 ALPHA infrastructure freeze
19:39:40 <nirik> #info 2013-04-16 F19 alpha release
19:39:41 <nirik> #info 2013-05-07 to 2013-05-21 BETA infrastructure freeze
19:39:43 <nirik> #info 2013-05-21 F19 beta release
19:39:45 <nirik> #info 2013-05-31 end of 1st quarter
19:39:47 <nirik> #info 2013-06-11 to 2013-06-25 FINAL infrastructure freeze.
19:39:49 <nirik> #info 2013-06-25 F19 FINAL release
19:39:51 <nirik> anything anyone would like to schedule or note?
19:40:11 <nirik> we have a little less than a month until alpha freeze.
19:40:38 <nirik> #topic Open Floor
19:40:42 <nirik> skvidal: you had something?
19:40:45 <skvidal> yah
19:40:53 <skvidal> something I was thinking about
19:40:53 <pingou> I sent somethings about jenkins, feedbacks welcome :)
19:41:01 <skvidal> in any given week we're all working on a billion things
19:41:06 <nirik> pingou: yeah, I have it marked to reply to. ;) Thanks.
19:41:15 <skvidal> and I was wondering if there was any thought to doing something like a virtual fad week
19:41:26 <skvidal> where we intend to focus on a few tasks and get them done
19:41:31 <nirik> skvidal: focusing on one area?
19:41:33 <pingou> sounds nice
19:41:34 <nirik> yeah.
19:41:35 <skvidal> right
19:41:39 <skvidal> we all are on irc
19:41:42 <skvidal> and most of the time all day long
19:41:55 <nirik> I think that kind of thing is very effective if we plan exactly what we want to try and do.
19:41:55 <skvidal> it seems like it would be very possible to schedulea vfad
19:41:57 <pingou> and most of us in a close timezone :)
19:42:00 <nirik> and make sure the needed people are available
19:42:09 <skvidal> hell, tie us toigether using a google hangout if need be
19:42:18 * threebean nods
19:42:20 <skvidal> but announce that we will be focused on one thing
19:42:28 <skvidal> and unavailable for random-ass pings
19:42:41 <nirik> #info idea: do some vfads and focus on specific areas to get them done.
19:42:48 <skvidal> I liked that we were able to knock out a specific problem last year for the 2fa fad
19:42:57 <skvidal> and I think we  should try it w/o the relocation
19:43:10 <skvidal> so what subjects would be things we could knock out?
19:43:20 <nirik> application logging?
19:43:27 * lmacken was just about to say that :P
19:43:30 <pingou> 2fa app wise
19:43:30 <threebean> perfect
19:43:53 <lmacken> breaking apps out
19:43:53 <skvidal> what else?
19:44:13 <threebean> ansible migration
19:44:43 <pingou> CLI logins for our web-app
19:44:47 <nirik> fedorahosted-ng
19:44:58 <pingou> but that comes back to the discussion abadger1999 wants to start :)
19:45:15 <lmacken> IDS
19:45:26 <pingou> IDS?
19:45:31 <lmacken> intrusion detection system
19:45:46 <pingou> ah, cool
19:45:48 * skvidal is making a list
19:45:51 <nirik> identify and plan how to get rid of our SPOF.
19:45:56 <skvidal> the db?
19:46:04 <skvidal> db-replication
19:46:11 <nirik> db's are a big one.
19:46:18 <pingou> spof?
19:46:19 <nirik> there might be other things tho.
19:46:24 <skvidal> single point of failure
19:46:26 <abadger1999> <nod>  I'd love to have some FADs on 2-fa and oauth.
19:46:27 <nirik> sorry, single point of failure.
19:46:34 <abadger1999> oauth needs some discussion first.
19:47:08 <abadger1999> 2-fa is closer to having a solid plan where a FAD type setting would really help
19:47:42 <skvidal> okay that's a good list start
19:47:43 <abadger1999> lmacken, for app logging -- do we know how to fix the problems we have?
19:48:04 * abadger1999 still hasn't found any reason we aren't getting all tracebacks in our logs.
19:48:35 <lmacken> abadger1999: I didn't know tracebacks were not appearing. Could be a few things.
19:48:49 <lmacken> I'd love to experiment with https://github.com/ryanpetrello/canary
19:49:12 <abadger1999> lmacken: that's like my number 1 problem with app logging since 2007 or so :-)
19:49:18 <pingou> spof?
19:49:20 <pingou> sorry
19:49:25 <lmacken> hmm, probably a simple ini or wsgi config tweak honestly
19:49:36 <nirik> getting all the data is a good first step, then to fix/reduce so it only tells us about real errors...
19:49:36 <ssmoogen> single point of failure
19:49:41 <skvidal> pingou: places where if that server/service dies then everything dies
19:49:52 <skvidal> nirik: I'll add another item to our list
19:49:53 <skvidal> nirik: nagios
19:50:01 <nirik> I know some apps really send a ton of stuff... I think fas sends to error_log on every successfull login.
19:50:18 <nirik> yeah. I have done some work on nagios, but there is more to plan out and do.
19:50:25 <pingou> skvidal: I hit arrow up/enter in the wrong window, sorry for the noise
19:50:37 <skvidal> pingou: oh - I thought you were still wondering what that means
19:50:57 <abadger1999> lmacken: I would think so... but I know you've looked at it a ton of times nad we've only succeeded in over logging things that we don't care about :-/
19:51:47 <nirik> I'd love to get to the point where someone says "hey, I just hit $app and got a 500" and we can easily see a traceback to attach to a ticket about it. ;)
19:52:02 <skvidal> so
19:52:05 <lmacken> abadger1999: potential pycon hackfest item :)
19:52:05 <skvidal> wrt app logging
19:52:10 <skvidal> it seems, to me, somewhat obvious
19:52:14 <skvidal> that if we break out app servers
19:52:18 <skvidal> logging becomes A LOT simpler
19:52:21 <nirik> yep.
19:52:22 <skvidal> doesn't it?
19:52:34 <nirik> if the log is from foobarapp01 it's likely caused by foobarapp
19:52:34 <threebean> yup
19:52:35 <skvidal> since isolating the logs for tagger won't involve sifting through a bunch of other logs
19:52:41 <abadger1999> skvidal: for some definition of a lot.
19:52:44 <abadger1999> yeah
19:52:45 <skvidal> (tagger was just an example)
19:52:51 <abadger1999> that's about 1/3 of the problem i think.
19:53:09 <skvidal> abadger1999: what are the other 2/3rds?
19:53:12 <lmacken> centralizing
19:53:13 <lmacken> analyzing
19:53:30 <abadger1999> another 1/3 is getting logs consolidated per service rather than per host.
19:53:32 <lmacken> realtime notifications of Bad Errors
19:53:46 <lmacken> trending
19:53:53 <abadger1999> ie: fas is its on boxes but it's still hard to search for the traceback because it could be on fas1,2,3
19:54:08 <abadger1999> *on its own boxes
19:54:20 <skvidal> okay - a couple of things to note - with our existing logging configuration on log02
19:54:26 <skvidal> right now we have 2 major logging groups
19:54:27 <pingou> load balancing makes it harder for sure
19:54:27 <skvidal> per-host
19:54:28 <skvidal> and merged
19:54:38 <skvidal> there is nothing stopping us from grouping those logs, too
19:54:43 <skvidal> ie: fas
19:54:46 * nirik nods
19:54:47 <skvidal> apps
19:55:02 <lmacken> hookup up the SyslogHandler for each app is still on the TODO
19:55:04 <skvidal> so you'd end up with consolidated syslogs/app logs int /var/log/groups/fas/
19:55:49 <abadger1999> skvidal: It's syslog based for the apache logs?  So we can have a single log file for a service?  Because that would be really nice.
19:55:59 <skvidal> abadger1999: well we have the app log now
19:56:34 <skvidal> which is only for apps which are using it
19:56:42 <nirik> which isn't many
19:56:42 <lmacken> bodhi in stg atm, iirc
19:56:43 <skvidal> istr it is local4
19:56:54 <skvidal> (that's the log facility)
19:56:57 * lmacken hasn't had cycles to wrap that up
19:57:09 <skvidal> yes local4
19:57:16 <skvidal> abadger1999: so we have 2 options there
19:57:25 <lmacken> (documented here: https://fedoraproject.org/wiki/Infrastructure/AppBestPractices#Centralized_logging)
19:57:41 <skvidal> 1. setup apache on the app servers to emit all error logs to logger on local4
19:57:52 <skvidal> 2. figure out a nicer way to setup apache/our apps
19:58:00 <skvidal> (or something in between)
19:58:08 <skvidal> I'd like to suggest one more avenue
19:58:11 <skvidal> that will require testing
19:58:13 <skvidal> and discussion
19:58:24 <skvidal> but is this - if there is a non-syslog mechanism for getting apache logs off of systems
19:58:33 <skvidal> let's hear about it
19:59:05 <nirik> we could also work on reducing our syslogs....
19:59:21 <skvidal> for example if there is a way to use 0mq to emit logs sanely
19:59:27 <skvidal> and to integrate it at the apache layer
19:59:37 <skvidal> I'd be inclined to pay attention to it, personally.
19:59:50 <skvidal> but
19:59:52 <skvidal> 1. it needs to work
19:59:57 <lmacken> where sanely == reliably w/o risk of losing messages
19:59:58 * puiterwijk is finally home and online
19:59:58 <skvidal> 2. be fairly reliable under load
20:00:05 <skvidal> lmacken: :)
20:00:09 <nirik> yeah.
20:00:23 <skvidal> lmacken: it can lose a few - but ideally the ring buffer that rsyslog uses would be the most desireable
20:01:00 <lmacken> sounds like a logging vfad would be a good idea ☺
20:01:01 <nirik> skvidal: so, can you post your vfad list and thoughts around those to the list and we can look at picking one and scheduling it?
20:01:13 * nirik nods. logging seems popular
20:01:17 * lmacken going to Monitorama after PyCon, so may have more ideas later this month
20:01:22 <skvidal> nirik: yes
20:01:34 <nirik> thanks.
20:01:34 <abadger1999> nirik: careful though -- logging is only popular because it's such a pain :-)
20:01:39 <nirik> indeed.
20:01:55 <skvidal> it's only a pain for you crazy kids and your new-fangled webapps ;)
20:02:00 <skvidal> :)
20:02:00 <nirik> ok, any other open floor items before we close out?
20:02:23 <threebean> hm, I'd put a vote in for splitting appservers first.  It might make fixing logging easier.
20:02:26 <abadger1999> I think for logging -- we should have a plan (like:  reconfigure all apps to log to syslog local4).  Then the vfad concentrates on doing that plan.
20:02:57 <nirik> yeah.
20:02:58 <pingou> +
20:03:38 <nirik> ok, thanks for coming everyone. Do continue on #fedora-admin, #fedora-apps, and #fedora-noc.
20:03:41 <nirik> #endmeeting