20:02:37 <mmcgrath> #startmeeting Infrastructure
20:02:37 <zodbot> Meeting started Thu Jan 14 20:02:37 2010 UTC.  The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:02:37 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:02:39 <mmcgrath> damnit
20:02:41 <mmcgrath> Who's here?
20:02:47 * a-k is
20:02:48 * wzzrd is here
20:02:57 * nirik is hanging around in the back
20:02:58 * PhrkOnLsh .
20:03:07 * skvidal is here
20:03:28 <mmcgrath> Ok, lets get started
20:03:32 <mmcgrath> #topic Infrastructure -- Tickets
20:03:51 <mmcgrath> Looks like no meeting notes so that's good.
20:04:03 <mmcgrath> I'll go over just a couple of things that happened this week worth mentioning
20:04:07 <mmcgrath> #topic Fedora hosted
20:04:11 <mmcgrath> hosted got it's new memory replaced.
20:04:22 <mmcgrath> it wasn't as smooth as it could have been but it did get done and within the outage window time.
20:04:30 <mmcgrath> #topic SPOF for koji and bastion
20:04:36 <mmcgrath> koj and bastion are still SPOF right now.
20:04:47 <mmcgrath> mostly because we've not re-configured heartbeat.
20:04:54 <mmcgrath> but I'm also trying to re-think our vpn setup.
20:05:07 <mmcgrath> to be more robust with outage scenarios.
20:05:14 <mmcgrath> not allowing outbound udp makes that difficult.
20:05:32 <mmcgrath> though I suspect specific outound udp requests could be allowed.
20:05:37 <mmcgrath> #topic /mnt/koji
20:05:57 <dgilmore> im waiting on the try-n-buy to be approved
20:05:57 * ricky is here for a bit
20:05:59 <mmcgrath> So this one's kind of happened behind closed doors, not intentional just lots of "do you think this would work?"  "how much budget?" etc, etc.
20:06:15 <mmcgrath> but yeah, dgilmore's been working on trying to get what will be the new /mnt/koji
20:06:25 <mmcgrath> and I've been working to figure out what will be the new backup of /mnt/koji.
20:06:31 <mmcgrath> this is all good for several reasons.
20:06:52 <mmcgrath> 1) it'll allow us to use all of our tapes for *everything* else and do daily, weekly and monthly backups as we should be doing.
20:06:52 <skvidal> mmcgrath: what's the plan for the storage for the backup? Still tape or are you switching to disks?
20:06:56 <mmcgrath> 2) /mnt/koji will hopefully be much faster.
20:07:16 <mmcgrath> skvidal: switching to disks.  I'm hoping we'll be able to make snapshots of it for testing.
20:07:35 <mmcgrath> but dgilmore and I also talked about using bacula to backup to disk.
20:07:52 <mmcgrath> in the end though I think it will be a better solution because if /mnt/koji does die right now a horrible death.
20:07:57 <mmcgrath> it'd take a long time to get a new /mnt/koji up
20:08:10 <mmcgrath> whereas if we have disks to backup to I think the impact would be lessoned.
20:08:21 <mmcgrath> I do, however, have concerns just because it's so easy to wipe disks.
20:08:31 <dgilmore> mmcgrath: i agree
20:08:33 <mmcgrath> blowing away the /mnt/koji backup right now is a significantly more difficult task I think.
20:08:41 <mmcgrath> but at the end of the day I think it'll be worth it to us.
20:09:12 <mmcgrath> we'll be spending something like 5X what we spent for the /mnt/koji now so hopefully it'll all work out :)
20:09:26 <mmcgrath> ehh actually that's not quite true, probably closer to 4X
20:09:30 <mmcgrath> but still.
20:09:36 <dgilmore> and ive requested extra budget for next yera to grow it and hopefully help it more
20:09:39 <mmcgrath> here's to hoping it's all fast and usable and safe and backed up :)
20:09:59 <mmcgrath> anyone else have any questions or concerns on that?
20:10:29 <mmcgrath> alllrighty
20:10:34 <mmcgrath> #topic ssh_known_hosts
20:10:42 <mmcgrath> smooge: around to talk about this?
20:11:18 <mmcgrath> the smooge is likely busy :)
20:11:32 <mmcgrath> After the move our ssh_known_hosts got way out of wack and instead of fixing it, he's been redoing it from scratch
20:11:41 <mmcgrath> and including more possible names so it should be even more useful then it was.
20:11:48 <smooge> here
20:11:49 <smooge> sorry
20:11:55 <smooge> one sec
20:12:04 <mmcgrath> smooge: no worries, just wanted a quick blurb about ssh_known_hosts so people know we're working on it.
20:12:16 <mdomsch> do our shells automatically use it, or do we need to do something in .ssh/config for it?
20:12:28 <smooge> ok sorry. I have updated ssh_known_hosts to all systems I could find in DNS and
20:12:32 <mmcgrath> mdomsch: they'd use it automatically but you should remove your .ssh/known_hosts
20:12:36 <smooge> removed ones that were no longer available
20:12:56 <mmcgrath> mdomsch: worst case, you'll get a conflict in known_hosts and you can just remove it.
20:13:07 <mdomsch> ok
20:13:12 <smooge> I have tested it on app01.stg and was able to go to hosts inside fedora. I have commited and pushed
20:13:16 <mmcgrath> but the search order is ~/.known_hosts then /etc/ssh/ssh_known_hosts
20:13:21 <smooge> The only things that aren't in it are IPV6 addresses
20:13:26 <mmcgrath> smooge: excellent, thanks.
20:13:37 <mmcgrath> smooge: have you updated the SOP to match the new search order?
20:13:49 <smooge> new search order?
20:13:57 <smooge> and which SOP
20:14:04 <dgilmore> thats the default search order
20:14:09 <mmcgrath> http://fedoraproject.org/wiki/Infrastructure/SOP/ssh_known_hosts
20:14:19 <mmcgrath> smooge: I think right now we just have short hostname, and actual IP of the host.
20:14:36 <smooge> ah ok will fix
20:14:39 <mmcgrath> in the SOP I mean, you expanded on that so it'd be good to get it in the example :)
20:14:42 <mmcgrath> smooge: thanks
20:14:46 <mmcgrath> Ok, any other questions on that?
20:15:02 <mmcgrath> zodbot: hey look, fedbot's a big quitter.
20:15:02 <mmcgrath> ok
20:15:06 <mmcgrath> #topic Search Engine
20:15:08 <mmcgrath> a-k: take it!
20:15:19 <a-k> The two search candidates I wanted to put in public test are Xapian and Nutch
20:16:00 <a-k> Xapian installed fine and crawled the wiki for 90 minutes, then died
20:16:00 <a-k> I think it's fixable, so I still have some hope for Xapian
20:16:01 <mmcgrath> a-k: has the pt instance worked out for you so far?
20:16:01 <a-k> For Nutch, I've got Tomcat installed, but not configured yet
20:16:01 <a-k> This is pt3, BTW
20:16:10 <a-k> I haven't checked if Tomcat's ports are open
20:16:14 <mmcgrath> If you need more ram or disk let me know, sometimes we're stuck but sometimes not
20:16:31 <a-k> Yeah, pt is working fine
20:16:44 <a-k> I intended to reverse proxy Tomcat through Apache, so no need for extra open ports
20:16:46 <mmcgrath> a-k: I've never used any of them, what are the key differences?
20:17:25 <a-k> Xapian is pretty flexible and customizable, so the custom keyword requirement should be satisfied, wherever that goes
20:17:38 <mmcgrath> then died a good death or a bad death?
20:17:46 <a-k> Nutch is very pre-packaged, so not so flexible
20:18:05 <a-k> Xapian didn't like a long URL, plus it may not like non-UTF8 so much
20:18:13 <mmcgrath> ahhh
20:18:20 <mmcgrath> interesting
20:18:26 <a-k> The URL thing I think is fixable with how I do the crawl
20:18:37 <mmcgrath> <nod>
20:18:45 <a-k> Plus, I still intend to keep looking at other candidates on my list
20:18:51 <mmcgrath> maybe not the non-UTF8 thing though huh?
20:18:57 <a-k> So these aren't necessarily the only two choices
20:19:01 <mmcgrath> sure
20:19:05 <abadger1999> Both java solutions?
20:19:07 * mmcgrath is happy to hear progress is being made.
20:19:15 <dgilmore> a-k: did we look at the one archive.org uses
20:19:16 <dgilmore> ?
20:19:23 <a-k> Yeah, UTF-8 could be the thing that kills most of the candidates, if that's going to be a real requirement
20:19:27 <mmcgrath> a-k: yeah what was the wiki link again?
20:19:52 <skvidal> why wouldn't utf-8 be a real requirement?
20:19:53 <a-k> dgilmore: archive.org uses arhiving, not indexing
20:20:04 <abadger1999> Or xapian is the C++ solution.  /me was thinking lucene.
20:20:08 <mmcgrath> skvidal: I think non-utf-8 is the possible requirement
20:20:13 <a-k> .link http://fedoraproject.org/wiki/Infrastructure/Search
20:20:24 <a-k> #link http://fedoraproject.org/wiki/Infrastructure/Search
20:20:32 <mmcgrath> a-k: thanks
20:20:36 <dgilmore> a-k: they have a crawler http://crawler.archive.org/
20:21:00 <a-k> Xapian is C, with a little Perl
20:21:08 <a-k> Nutch is Java, hence Tomcat
20:21:25 <a-k> dgilmore: I can look at archive.org again
20:21:47 <a-k> I think that's it, unless there are more questions
20:21:54 <mmcgrath> a-k: just because I know people will ask, can you make sure that those that have been eliminated have a specific "eliminated because: " section?
20:22:14 <a-k> I can do that
20:22:30 * nirik wonders if any solutions here will tie into the mailman/lists archives?
20:22:38 <smooge> eliminated because: It kills kittens and makes you eat them
20:22:40 <nirik> to replace pipermail.
20:22:57 <smooge> why would a search engine replace pipermail?
20:23:07 <skvidal> smooge: he means a new archiver
20:23:20 <skvidal> nirik: I don't think any of the others are any more maintained
20:23:26 <mmcgrath> a-k: if you want additional help feel free to ask on the list and recruit :) This is a pretty massive project.   Especially for your first one for Fedora.  If you need anything feel free to ask :)
20:23:46 <a-k> Sure.  Thanks.
20:23:53 <mmcgrath> anyone have anything else right now?
20:24:02 <nirik> sure, personally I think pipermail is ok, but there is a ticket wanting us to replace it with monharc or something.
20:24:20 * dgilmore is ok with pipermail
20:24:46 <skvidal> nirik: mhonarc is more trouble than its worth, ime
20:25:03 <skvidal> mmcgrath: logging? anyone interested in working on it?
20:25:08 * nirik nods. perhaps 3.0 will come out someday with pipermail improvements.
20:25:19 <wzzrd> I'm kinda working on it, fwiw
20:25:20 <skvidal> nirik: perhaps monkeys will fly out of my arse
20:25:36 <smooge> skvidal, I am interested in doing it
20:25:46 <wzzrd> the logging that is
20:25:46 <smooge> I was starting to do it after I finished DNS/NTP
20:25:52 <mmcgrath> skvidal: sure
20:25:56 <mmcgrath> #topic logging
20:26:03 <smooge> wzzrd, what are you doing
20:26:05 <skvidal> smooge: okay
20:26:07 <smooge> skvidal, what did you see
20:26:11 <skvidal> there are a number of things to work on
20:26:13 <mmcgrath> So logging is a pretty large topic with lots of sub parts
20:26:16 <skvidal> first - cleaning up the logs we have
20:26:19 <smooge> skvidal, or would I be too many cooks
20:26:24 <mmcgrath> lets start on what I gave wzzrd a week or two back.
20:26:29 <skvidal> okay
20:26:33 <mmcgrath> which is log analysis.
20:26:40 <skvidal> which is the wrong place to start
20:26:41 <mmcgrath> wzzrd: have you had a chance to look at some of the suggestions on th elist?
20:26:44 <skvidal> until you have logs under control
20:26:55 <mmcgrath> skvidal: none of which will get done while this meeting is going on :)
20:27:15 <wzzrd> i've checked out epylog a bit further
20:27:16 <skvidal> umm - if you don't want to discuss logging, that's fine
20:27:26 * skvidal is sorry for making the meeting longer
20:27:54 <mmcgrath> skvidal: keep  your pants on we're talking about logging right now, I'm just trying to do so in a way that doesn't discourage wzzrd, a new potential sysadmin member.
20:28:20 <mmcgrath> wzzrd: any luck with it?
20:28:24 <wzzrd> but i want to raise the matter of realtime parsing vs. cron-based once-a-day parsing, before i dive into something head over heels
20:28:35 <mmcgrath> I have very little experience with it but I know skvidal has used it as has Jeff_S and some others.
20:28:45 <mmcgrath> I don't think we need realtime parsing at this time.
20:28:51 <skvidal> wzzrd: realtime parsing and analysis are seldom done by the same tool
20:28:51 <mmcgrath> at least I don't think it would buys us much.
20:29:11 <wzzrd> epylog is nice, but i think it woulde require the logs from a group of servers going into *one* file on the loghost if you want a single report for that group
20:29:17 <skvidal> wzzrd: and realtime parsing is handy if there are specific triggers you know to look for - but only useful insofar as they can raise a warning in nagios
20:29:21 <mmcgrath> but yeah, I haven't heard anyone really argue for realtime so we can just assume non-realtime for the moment :)
20:29:38 <wzzrd> mmcgrath, skvidal: ok, no real-time
20:29:40 <wzzrd> just making sure
20:29:40 <mmcgrath> epylog can't look at multiple files?
20:29:50 <skvidal> mmcgrath: yes, it can - but it requires editingits configs
20:30:06 <skvidal> mmcgrath: remember what I was saying on the infrastructure list about mimicing the file structure of /var/log
20:30:11 <skvidal> mmcgrath: I wasn't making that up :)
20:30:12 <mmcgrath> skvidal: just edits though?  not like some crazy bastadrization?
20:30:21 <skvidal> mmcgrath: significant edits
20:30:26 <mmcgrath> skvidal: I still have no idea what you're talking about with that?
20:30:37 <skvidal> look in /var/log on your laptop/desktop
20:30:38 <mmcgrath> you mean I should have a /var/log/messages that has all messages from all of our hosts going int o it?
20:30:43 <skvidal> no
20:30:53 <mmcgrath> so I should have a /var/log/hosts/bastion/messages?
20:31:01 <mmcgrath> I know what is in /var/log/ on my laptop
20:31:03 <skvidal> you understand there are certain files that commonly exist in /var/log
20:31:05 <skvidal> good
20:31:10 <mmcgrath> yeah?
20:31:12 <mmcgrath> like funcd
20:31:16 <mmcgrath> which won't exist on log1
20:31:21 <skvidal> and those files are expected to havecontent consistentwith a lot of log parsers
20:31:35 <wzzrd> skvidal: epylog.conf doesn't allow it afais, and I don't think it comes with a module that allows for the parsing of multiple files
20:31:36 <skvidal> funcd doesn't log via syslog in that way
20:31:45 <mmcgrath> isn't that creating just the opposite of what wzzrd is talking about though?
20:31:45 <skvidal> wzzrd: It really does.
20:31:48 <skvidal> mmcgrath: no
20:31:54 <mmcgrath> you're talking about creating more files and he's needing less
20:31:59 <skvidal> no
20:32:01 <skvidal> I'm not
20:32:09 <skvidal> if y'all would let me explain
20:32:19 <skvidal> instead of peppering me with remarks
20:32:21 <skvidal> It might help
20:32:31 * wzzrd shuts up
20:32:31 <skvidal> we want logs per-host
20:32:42 <skvidal> but we also want logs per-by-service/group
20:32:54 <skvidal> so let's say all of the app servers belong to the appgroup
20:33:47 <skvidal> we can setup rsyslog so that if a log comes in from app01 (for example) that it gets sent to /var/log/hosts/app01/2010/01/14/ AND to /var/log/groups/app-servers/2010/01/14
20:34:10 <skvidal> then if we want to do log analysis for the appservers we tell epylog to look at /var/log/groups/app-servers/2010/01/14
20:34:32 <skvidal> if we want it to do analysis for a specific app server we tell it to look at: /var/log/hosts/app01/2010/01/14/
20:34:47 <mmcgrath> epylog doesn't understand /var/log/hosts/app* ?
20:34:52 <skvidal> inside each of those dirs will be the syslog files normally generated by /var/log
20:35:23 <skvidal> mmcgrath: not when it is parsing log files - it expects the log files to be in the normal location relative to the base log path
20:35:52 <mmcgrath> I guess I'm not understanding how what is in my laptop in /var/log isn't what's in /var/log/hosts/app01/2010/01/14
20:36:02 <skvidal> okay
20:36:05 <skvidal> let's look at an example
20:36:29 <skvidal> login to bastion
20:36:41 <skvidal> cd /var/log
20:36:43 <skvidal> ls *log
20:36:47 <skvidal> ls *log
20:36:47 <skvidal> anaconda.log     boot.log  ha-log   ldirectord.log  sa-update.log  yum.log
20:36:47 <skvidal> anaconda.syslog  faillog   lastlog  maillog         tallylog
20:37:14 <skvidal> the files syslog is writing are the only ones we can deal with for a remote logging server
20:37:33 <skvidal> so messages, maillog, spooler, boot.log and cron
20:37:38 <skvidal> that's all we have access to
20:37:41 <skvidal> now
20:37:47 <skvidal> if you look on log1
20:37:51 <skvidal> in
20:37:54 <skvidal> /var/log/hosts/bastion01/2010/01/14
20:37:55 <skvidal> for example
20:38:00 <skvidal> cron.log  kernel.log  mail.log  messages.log  secure.log
20:38:04 <skvidal> you have all of those files
20:38:21 <skvidal> do you see how those files do not match the filenames and separation that are normally in /var/og?
20:38:26 <skvidal> ie mail.log vs maillog
20:38:30 <skvidal> kernel.log existing AT ALL
20:38:36 <skvidal> messages.log vs messages
20:38:47 <smooge> oi
20:38:51 <mmcgrath> yeah, so you're talking, mostly, about renaming the files?
20:38:52 <skvidal> that's what I mean by the difference
20:39:04 <mmcgrath> k
20:39:09 <skvidal> mmcgrath: and what content goes into them
20:39:19 <skvidal> ie: kernel.log shouldn
20:39:22 <skvidal> 't exist at all really
20:39:28 <skvidal> it's content should be in messages
20:39:28 <wzzrd> skvidal: crap you were right, i was looking in the wrong place... i didn't quite grasp how epylog's internals worked yet, i suppose...
20:39:32 <mmcgrath> k, I don't see any problem with that.
20:39:47 <skvidal> that's what I mean about fixing the structure of our remote logs
20:39:51 <skvidal> then once we do that
20:40:01 <skvidal> and we log by 'type of server/service'
20:40:04 <mmcgrath> wzzrd: k, you want to continue working and learning epylog?
20:40:12 <skvidal> then we can run generic log tools like epylog and generate lovely results
20:40:19 <wzzrd> mmcgrath: sure, eager to help out
20:40:33 <skvidal> w/o having to beat our brains out modifying epylog to access logs we don't want
20:40:42 <mmcgrath> skvidal: so in theory the amount of logs we store is going to about double.  Which do you think we should keep longer?  the host level logs or the service level logs?
20:40:54 <skvidal> mmcgrath: it's not going to double
20:41:08 <skvidal> right now we've made the mistake of doing *.* from syslog.conf on our logclients
20:41:13 <skvidal> instead of trimming the crap out
20:41:22 <skvidal> no one needs spooler.debug sent remotely
20:41:23 <mmcgrath> how do you know what to include and what not to?
20:41:31 <skvidal> years of experience?
20:41:32 <skvidal> :)
20:41:44 <skvidal> seriously-  you keep warning and above
20:41:51 <skvidal> and drop a lot of the info and debug crap
20:41:59 <mmcgrath> skvidal: but still, lets say we only sent what warning and above right this second.
20:42:01 <skvidal> we can get rid of a lot of crap that's not helpful
20:42:09 <mmcgrath> when we start storing services too, we're storing all logs twice right?
20:42:25 <skvidal> <shrug> sure but it's just not that much content
20:42:26 <mmcgrath> skvidal: so, for example, we wouldn't be sending mail logs?
20:42:37 <skvidal> I think we should send maillogs
20:42:56 <skvidal> unless you want to do mailog analysis ON the mailservers
20:43:06 <skvidal> which seems like a bad use of their cpu time
20:43:09 <mmcgrath> I've never setup a central logger that didn't store everything, skvidal do you happen to want to take lead on trimming that stuff down?
20:43:15 <dgilmore> i should just stop getting email and there would be alot less log data
20:43:17 <mmcgrath> <nod> I'd prefer to keep log analysis on the logger.
20:43:24 <mmcgrath> dgilmore: that's very true :)
20:43:28 <skvidal> mmcgrath: sure
20:43:35 <skvidal> mmcgrath: smooge you wanna work w/me on this?
20:43:45 <smooge> I am happy to. I love log analysis
20:43:51 <mmcgrath> skvidal: yup yup.
20:43:52 <skvidal> smooge: I'm glad I'm not alone :)
20:44:09 <mmcgrath> so on this same topic... there's still one thing I'd like to get converted.
20:44:14 <mmcgrath> most of our hosts are still not using rsyslog
20:44:22 <mmcgrath> I'd like to convert them to rsyslog, some have been but not most.
20:44:25 <smooge> mmcgrath, on my list of things to fix after ntpd
20:44:25 * mmcgrath is just mentioning that.
20:44:29 <skvidal> mmcgrath: for rhel5?
20:44:31 <mmcgrath> it could be as easy as yum install rsyslgo
20:44:35 <skvidal> did rhel5 switch to rsyslog?
20:44:36 <mmcgrath> skvidal: yeah
20:44:48 <skvidal> okie doke
20:44:48 <smooge> mmcgrath, skvidal it is basically 5 commands
20:44:49 <mmcgrath> skvidal: it does for fresh installs, but if you updated, it didn't do a replace.
20:44:58 <smooge> 1) yum install rsylog
20:45:04 <mmcgrath> as of 5.3 I think.
20:45:05 <skvidal> smooge: no problem - I just hadn't heard the switch was official in rhel5
20:45:14 <smooge> its an alternate
20:45:32 <smooge> syslogd is still prefered because of age
20:45:51 <smooge> oh wait.. I missed the new isntall part
20:46:14 <mmcgrath> smooge: is this something that's going to be possible / easy in puppet or are we goign to have to get func involved?
20:46:42 <smooge> func migth be easiest
20:46:47 * mmcgrath thinks perhaps he's been overthinking it.
20:46:54 <mmcgrath> smooge: what about new installs?
20:47:04 <mmcgrath> well, we can figure that after the meeting I guess :)
20:47:08 <smooge> yeah..
20:47:20 <mmcgrath> wzzrd: any other questions on your side?
20:47:31 <mmcgrath> does anyone have any thoughts about exactly what we're looking for in these reports?
20:47:47 <wzzrd> well, im not sure whether this ok to ask
20:48:14 <mmcgrath> you can ask whatever you want.  If it's for the root password though we probably won't answer.
20:48:31 <wzzrd> i think it would be easy to have some sort of mentor,
20:48:40 <wzzrd> you know, to ask some questions to
20:48:41 <skvidal> mmcgrath: for mail- I'm looking for errors and to make sure we don't have too much overrun/disk/cpu issues, for the rest of systems I'm looking to start getting a baseline on what 'normal' looks like and then fixing up problems
20:48:44 <skvidal> wzzrd: ask me
20:49:01 <wzzrd> skvidal: great, thanks!
20:49:07 <skvidal> I'm around often and I know a fair bit about the epylog code base
20:49:18 <skvidal> and I know the author of epylog personally and am willing to annoy him
20:49:19 <wzzrd> skvidal: you seem to be pretty well informed in this logging business :)
20:49:36 <wzzrd> lol
20:49:43 <mmcgrath> wzzrd: yeah for epylog ask skvidal I have no experience with it.  If you have questions about fedora or how we're doing something just ask anyone in #fedora-admin, skvidal smooge and I are almost always in there.
20:49:56 <wzzrd> ok appreciate it
20:50:07 <dgilmore> wzzrd: skvidal, mmcgrath, myself, ricky, smooge  and others will be more than happy to answer questions
20:50:20 <mmcgrath> Ok, anyone hav any other questions on logging?
20:51:01 <smooge> not me
20:51:08 <mmcgrath> ok, with that I'll open the floor for anything and everything
20:51:18 <mmcgrath> #topic Infrastructuer -- Open FLoor
20:51:21 <mmcgrath> jds2001: you around?
20:51:27 <mmcgrath> our newest -main member has been pretty quiet.
20:51:49 <dgilmore> mmcgrath: i wore him out on Saturday
20:52:01 <mmcgrath> hehe
20:52:03 <mmcgrath> oh!  i know one thing.
20:52:04 <skvidal> dgilmore: umm - that sounds
20:52:05 <skvidal> umm
20:52:06 <skvidal> wrong
20:52:26 <mmcgrath> I'm still in the process of getting our secondary,alt,archive stuff to download.fedora.redhat.com
20:52:27 <dgilmore> skvidal: sure.
20:52:32 <skvidal> :)
20:52:40 <dgilmore> skvidal: i made him work alot on saturday while migrating the lists
20:52:41 <dgilmore> ;)
20:52:43 <mmcgrath> After the move we have root squashed for all /pub content (which is good)
20:52:45 <dgilmore> that better
20:52:53 <mmcgrath> but now I can't do wnything with my dirs that are there becaues they're root owned.
20:53:14 <ricky> Heh, yow :-)
20:53:23 <dgilmore> opps
20:53:27 <mmcgrath> yeah
20:53:33 <mmcgrath> I do still have several concerns.
20:53:36 <mmcgrath> same as smooge
20:53:41 <mmcgrath> will the disks be able to keep up?
20:53:59 <mmcgrath> will the snap-mirror process work correctly with the new load?
20:54:45 <mmcgrath> Ok, anyone have anything else?
20:54:51 <mmcgrath> If not we'll close the meeting in 30
20:54:52 <smooge> concerns? I have no concerns.. I have reality
20:55:01 <smooge> cold clear that it probably wont keep up
20:55:07 <mmcgrath> smooge: yeah, I have a feeling we're in for a ride here.
20:55:13 <mmcgrath> but who knows, we might get surprised :)
20:55:17 <nirik> mmcgrath: will this change what I have to do to push the nightly live composes?
20:55:24 <smooge> yeah.. I am buying lotto tickets just in case
20:55:55 <mmcgrath> nirik: it'll just change were you write to
20:56:07 <mmcgrath> nirik: hopefully for the better because it'll be all mirrored and stuff
20:56:11 <nirik> cool.
20:56:47 <mmcgrath> ok,  with that!
20:56:49 <mmcgrath> #meetingend
20:56:53 <mmcgrath> #endmeeting