19:00:17 <nirik> #startmeeting Infrastructure (2011-09-01)
19:00:17 <zodbot> Meeting started Thu Sep  1 19:00:17 2011 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:17 <nirik> #meetingname infrastructure
19:00:17 <zodbot> The meeting name has been set to 'infrastructure'
19:00:17 <nirik> #topic Robot Roll Call
19:00:17 <nirik> #chair smooge skvidal codeblock ricky nirik abadger1999 lmacken
19:00:17 <zodbot> Current chairs: abadger1999 codeblock lmacken nirik ricky skvidal smooge
19:00:23 * abadger1999 here
19:00:40 * athmane is kinda here
19:01:06 <CodeBlock> here
19:01:26 * herlo here
19:01:37 * mahrud here
19:01:49 * ricky listens in
19:02:15 <nirik> cool. Lets go ahead and dive in.
19:02:22 <nirik> #topic New folks introductions and apprentice tasks/feedback
19:02:43 <nirik> any new folks want to say hi? or apprentice folks want to talk about/look at any tickets or feedback?
19:03:27 <nirik> guess not. ;)
19:03:39 <nirik> #topic Upcoming new machine plans.
19:03:49 <nirik> I posted some upcoming new machine/migration plans to the list.
19:03:59 <LoKoMurdoK> here
19:04:17 <nirik> Feedback welcome there. Assistance building machines from sysadmin-main folks welcome. ;)
19:04:31 <nirik> I am planning on trying to migrate bastion02 and db02 next week.
19:04:39 <nirik> I will be sending out an outage notice on that.
19:04:49 * CodeBlock was going to look at working on some of those this afternoon with smooge
19:05:06 <nirik> I'm not sure if it makes sense to do the outage early in the day, or later at night after us folks are gone home.
19:05:16 <nirik> CodeBlock: appreciated. ;)
19:06:19 <nirik> On the new releng boxes I am waiting for a few tickets to get done... we need to move bvirthost04's vlan on secondary interface, and the netapp needs to allow them to mount it.
19:06:30 <nirik> once those get done, we can migrate some releng stuff too.
19:07:49 <nirik> anyone have thoughts on the db02/bastion02 outage timing? It will basically take most things out, as it takes down the vpn and fas.
19:08:48 <nirik> is this thing on? :)
19:09:30 * skvidal is here
19:09:31 <skvidal> sorry
19:09:33 <mahrud> isn't there any backup for bastion02?
19:09:37 * skvidal had someone at the dor :)
19:09:39 <skvidal> err door, even :)
19:10:11 <mahrud> nirik: you said the problem is with selinux, right?
19:10:32 <nirik> sure, we can go to bastion01... it would still be a blip in vpn tho.
19:10:38 <nirik> mahrud: which problem?
19:10:49 <nirik> but if we are migrating db02, fas will be down.
19:11:21 <mahrud> not sure, i think you said some host has problem with selinux ...
19:11:31 <mahrud> and you wanted too reboot it
19:11:36 <nirik> oh, fas01.stg? that is fixed. ;)
19:11:46 <mahrud> aha, ok
19:12:12 <nirik> I guess I should do a dump/reload on db01 of db02 content and see how long it takes. :)
19:12:42 <nirik> #topic Upcoming Tasks/Items
19:12:54 <nirik> Any other upcoming items folks are working on or want to talk about.
19:13:08 <nirik> note that the beta freeze is coming up:
19:13:10 <nirik> 2011-09-13 - 27: Beta change freeze
19:13:24 <mahrud> nirik: between, I want to work on ticket 1084, but not sure about what are some hosts doing exactly, also, not sure about which hosts do we have!
19:13:40 <abadger1999> I'm going to be upgrading python-fedora soon -- lots of changes to the fas auth providers for both TG1 and TG2 apps.
19:13:42 <nirik> .ticket 1084
19:13:45 <zodbot> nirik: #1084 (Fix proxy -> app docs) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1084
19:14:29 <nirik> mahrud: yeah, I cleaned up logs on log02, so that should look much nicer. ;) as for the apps/proxies you will have to dig around some and ask questions...
19:14:39 <nirik> and thanks for looking at updating those docs.
19:15:06 <nirik> abadger1999: cool. when might this land? before freeze?
19:15:29 <abadger1999> nirik: I'm going to try to push to stg tomorrow (possibly tonight) and prod on Monday.
19:15:44 <abadger1999> nirik: I need the update for the raffle app which I'm trying to deploy to prod before freeze.
19:16:02 <nirik> cool.
19:16:07 <mahrud> nirik: thanks for cleaning log02, it much better, but yet if you change a hostname or ... you'll need to update log02
19:16:10 <nirik> lmacken: you were going to push out a bodhi update soon too?
19:16:21 <mahrud> nirik: i was thinking about some script based on ip addresses
19:16:50 <nirik> well, as long as we change hosts / add new ones consistently moving forward we should be fine.
19:17:02 <skvidal> nirik: this is another place where I think we need to be autogenerating some info for other hosts based on the info in puppet/func
19:17:03 <nirik> There are also some hosts not logging to log02 yet, which we need to add
19:17:25 <mahrud> nirik: we have vpn on 192.168.0.0/16 (right?), and --i think-- phx2 hosts are on 10.5.125-126-127.0/24
19:17:41 <skvidal> nirik: I'm not sure, yet, how we should go about deploying autogenerated information to other systems off of lockbox01, though
19:17:48 <lmacken> nirik: yeah, i'm trying to wrap up a couple of fixes and will hopefully push a new release out soonish
19:18:01 <skvidal> mainly b/c there is no way to nicely automatically check in something to puppet as a non-user
19:18:40 <nirik> mahrud: yep. 192.168.x.x is vpn. 10.5.125.0/24 is builder network, 10.5.126.0/24 is main network, 10.5.127.0/24 is storage network, 10.5.124.0/24 is qa/community network.
19:18:58 <mahrud> nirik: then we can write a script to scan the net and report online hostnames
19:19:01 <nirik> yeah, we should leverage infra-hosts as much as we can. ;)
19:19:14 <nirik> mahrud: well, sure, but lets back up a second.
19:19:57 <nirik> I think we should use infra-hosts for this... should it not contain all hosts?
19:20:04 <skvidal> nirik: it does, yes
19:20:12 <skvidal> nirik: but again -we have to check things in there, too, right?
19:20:23 <nirik> yeah, true.
19:20:27 <skvidal> nirik: oh you were replying to mahrud
19:20:32 <skvidal> so I have a couple of thoughts on that
19:20:38 <skvidal> we could use infrahosts as we have been
19:20:40 <nirik> yes, but I agree if it was automated it could be nice too. ;)
19:20:49 <skvidal> but generate some lists to another location
19:20:52 <skvidal> that is still a git repo
19:20:55 <mahrud> in case of any hostname change, we need something to automatically list hosts
19:21:00 <mahrud> nirik: is 10.5.*.* only for phx2 or all of them?
19:21:05 <skvidal> but t is just a single committer
19:21:08 <skvidal> so we can tell what changes
19:21:16 <skvidal> but not change it ourselves
19:21:19 <skvidal> if you see what I mean
19:21:19 <nirik> mahrud: only phx2.
19:21:51 <nirik> well, it seems like we are creating lots of repos... more means it's difficult to remember which one to do what in...
19:22:07 <nirik> we really shouldn't be renaming stuff much if at all should we?
19:22:49 <skvidal> nirik: right
19:22:55 <skvidal> I'm not suggesting we make this a new repo
19:22:56 <skvidal> for US
19:23:10 <skvidal> I'm saying we make it so a root/system process commits changes to it
19:23:17 <skvidal> every hour
19:23:38 <nirik> ok, as a dump of func hosts?
19:23:59 <mahrud> skvidal: you mean an script to commit its changes to repo?
19:24:12 <skvidal> mahrud: yes
19:24:18 <skvidal> the point of the commit
19:24:21 <mahrud> on each host?
19:24:23 <skvidal> is just so we can walk back through waht it changed
19:24:27 <mahrud> that sounds nice ...
19:24:29 <skvidal> not _on_ each host no
19:24:41 <skvidal> it would be located on lockbox01
19:24:42 <mahrud> hmm, so where?
19:24:47 <nirik> into infa-hosts? :)
19:25:09 <skvidal> nirik: right - that's the problem
19:25:15 <skvidal> it is sorta like this
19:25:18 <skvidal> hmm
19:25:22 <skvidal> maybe we can do this with infra hosts
19:25:23 <Southern_Gentlem> hmmm can someone see  whats up with fedorapeople.org
19:25:36 <skvidal> Southern_Gentlem: looks fine from here
19:26:02 <skvidal> nirik: I really just want a 'space where info is autogenerated' and a 'space where admins edit things' that has the same list of hosts
19:26:19 <nirik> I know! lets make it branches in the same repo!
19:26:21 * nirik runs away
19:27:21 <skvidal> nirik: you know how to hurt a guy, don't you?
19:27:37 <nirik> anyhow, lets discuss this out of meeting? and come up with a plan/
19:27:40 <skvidal> yah
19:27:42 <skvidal> sounds fine w/me
19:27:46 <nirik> we could do it like virt-hosts
19:27:46 <mahrud> ok
19:27:57 <nirik> hosts and it mails when it changes.
19:28:16 <skvidal> nirik: nod - thats sorta what I was thinking - but we could put the results into either a repo or into a path accessible on infrastructure.fp.o
19:28:25 <skvidal> nirik: so then other hosts could use/mine the information for their own processes
19:28:52 <nirik> yeah.
19:29:11 <nirik> ok, any other upcoming work people are looking at?
19:29:28 <mahrud> ow
19:29:36 <mahrud> nirik: about that certificate ...
19:29:48 <nirik> mahrud: the koji one?
19:29:54 <mahrud> yeah
19:30:16 <nirik> I'd prefer if we get dgilmore to change that and pkgs... just in case there are things we are not thinking of. ;)
19:30:35 <nirik> what was the ticket # on that one?
19:30:54 <mahrud> umm
19:31:04 <nirik> .ticket 1929
19:31:06 <zodbot> nirik: #1929 (https://koji.fedoraproject.org server certificate is signed with MD5) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1929
19:31:30 <nirik> yeah, I will see what we can do there. If you could add your testing and thoughts to the ticket that would be great.
19:31:47 <mahrud> ok
19:32:09 <nirik> #topic Meeting tagged tickets:
19:32:10 <nirik> https://fedorahosted.org/fedora-infrastructure/report/10
19:32:17 <nirik> I cleaned up our meeting tagged tickets.
19:32:21 <nirik> we currently have 0. ;)
19:32:36 <nirik> if anyone has a specific ticket they want to bring up moving forward, add the 'meeting' keyword to it.
19:33:33 <nirik> #topic Open Floor
19:33:39 <nirik> Anyone have anything for open floor?
19:33:54 <abadger1999> I opened a new ticket for fi-apprentice to look at.
19:34:03 <abadger1999> Just wanted to check that it's what we want to happen.
19:34:11 <nirik> abadger1999: did you stick the easyfix keyword on it?
19:34:27 <abadger1999> the new_repo script for fedorapeople repos take a "group or user to own the repo"
19:34:32 <abadger1999> nirik: yeah, I did
19:34:39 <nirik> cool.
19:34:42 <abadger1999> https://fedorahosted.org/fedora-infrastructure/ticket/2931
19:34:58 <abadger1999> currently, that's just a freeform string.
19:35:03 <mahrud> easyfix
19:35:07 <mahrud> not EasyFix :D
19:35:09 <nirik> cool.
19:35:17 <abadger1999> I think we should make that confirm that it's either a username on fedorapeople or a group on fedorapeople.
19:35:28 <abadger1999> Sound good to everyone else?
19:35:44 <nirik> yep. Sounds good to me.
19:36:05 <jsmith> +1
19:37:01 * abadger1999 updates ticket.
19:37:02 <nirik> ok, anything else? or shall we close on up and get back to work?
19:37:12 <mahrud> hmm
19:37:16 <mahrud> before that
19:37:16 <nirik> oh, FYI, all servers should have the updated httpd and have been restarted...
19:37:41 <CodeBlock> oh, cool
19:37:48 <mahrud> can I ask to put audit log on log02 too?
19:38:12 <skvidal> is audit logged via syslog?
19:38:19 <skvidal> I thought it logged directly for some reason
19:38:29 <nirik> mahrud: well, I think we have talked about that... but yeah, it does it's own logging.
19:38:44 <nirik> but I agree it would be good to get going on there too...
19:38:49 <mahrud> no it isn't via syslog, but there must be some way ...
19:39:53 <mahrud> in the worst case, a script to read it and send it with nc should work :)
19:39:54 <nirik> yeah, I think it's possible/doable... we can look into doing so.
19:40:50 <nirik> #info look into audit logging to log02
19:41:23 <nirik> also, I think I might like a log03 to have a sync/backup copy of all logs thats ro/locked down. Just to have another copy in the audit trail. ;)
19:42:01 <nirik> anyhow, thanks for coming everyone!
19:42:22 <herlo> nice meeting.
19:42:27 <herlo> short and sweet
19:42:41 <nirik> oh, hey herlo. Any news on paste? ;)
19:42:53 <nirik> and I forgot news on ask.
19:43:04 <nirik> #topic quick RFR roundup
19:43:20 <nirik> #info ask is making some last minute packaging changes to make it easier to deploy.
19:43:40 <nirik> #info There's a puppet commit I have been getting ready to add ask01.stg once thats done.
19:43:43 <herlo> nirik: no, been crazy busy the past couple weeks. I will probably have some time in sept though
19:44:00 <nirik> no worries at all.
19:44:19 <nirik> ok, thanks again for coming everyone.
19:44:22 <nirik> #endmeeting