19:00:03 <nirik> #startmeeting Infrastructure (2014-03-06)
19:00:03 <zodbot> Meeting started Thu Mar  6 19:00:03 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:03 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:03 <nirik> #meetingname infrastructure
19:00:03 <nirik> #topic welcome everyone
19:00:03 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
19:00:03 <zodbot> The meeting name has been set to 'infrastructure'
19:00:03 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
19:00:15 * lmacken 
19:00:22 <nirik> hey everyone.
19:00:31 * johe is here
19:01:01 * pingou 
19:01:04 * tflink is lurking
19:01:09 <axil42> hello :)
19:01:49 <webpigeon> Hello :)
19:01:57 <nirik> ok, lets go ahead and dive in...
19:01:58 <janeznemanic> hi
19:02:06 <nirik> #topic New folks introductions and Apprentice tasks
19:02:09 <purplex> Hello everyone!
19:02:15 <nirik> any new folks want to introduce themselves?
19:02:22 <nirik> Or apprentices with questions or comments?
19:02:33 <axil42> yeap :)
19:02:42 <purplex> Introducing myself
19:02:47 * relrod here, sorry
19:03:06 * abadger1999 extricates from other meeting
19:03:07 <nirik> welcome axil42 and purplex. Do give a short sentence introduction if you can...
19:03:11 <smooge> hello
19:03:12 <purplex> I'm Bhumish, currently a student. I'm into system admin and a Fedora-lover.
19:03:50 <pingou> abadger1999: good luck :)
19:04:15 <axil42> I'm Achilleas also student, currently Fedora packager and want to get into sysadmin tasks :)
19:04:24 <nirik> purplex / axil42: are you more interested in sysadmin type tasks or application development/coding?
19:04:32 * mirek-hm joins
19:04:51 <purplex> I'm interested in sysadmin tasks. Currently I'm exploring FUNC.
19:04:53 * willo here, apologies
19:04:59 <axil42> me more on sysadmin, like I said in my introduction mail I'd like to know more about ansible
19:05:06 <nirik> cool. See me after the meeting in #fedora-admin and I can make sure you are added to our apprentice sysadmin group and pointed at tickets and such. ;)
19:05:13 <nirik> and welcome again!
19:05:20 <axil42> cool :)
19:05:30 <purplex> sure! :)
19:06:05 <nirik> #topic Meeting time (DST)
19:06:15 <nirik> so the us goes to daylight savings time next week...
19:06:36 <nirik> I'd like to move the meeting forward an hour to be at the same time... so 20UTC... is that ok for everyone?
19:06:54 <nirik> (at least if i got that right, I hate timezones and dst)
19:07:01 <purplex> yes the time will be alright.
19:07:29 <relrod> worksforme
19:07:29 <johe> +1
19:07:38 <pingou> so for couple of weeks it's gonna be at 21:00 (9pm) instead of 20:00 (8pm) for western europ
19:07:53 <webpigeon> that's fine for me
19:08:19 <nirik> ok, cool. :)
19:08:32 <pingou> wfm :)
19:08:36 <nirik> #info meeting to move ahead an hour next week
19:08:46 <pingou> for those that changes to DST :)
19:08:52 <nirik> #topic Applications status / discussion
19:08:58 <pingou> otherwise it moves one hour back :)
19:08:59 <nirik> any applications status or news?
19:09:06 <nirik> pingou: right, true.
19:09:08 <pingou> there is a new FAS release, 0.9.0
19:09:12 <nirik> dst gives me a headache.
19:09:23 <pingou> https://admin.stg.fedoraproject.org/accounts stg is updated
19:09:33 <pingou> pkgdb2-cli is almost ready
19:09:45 <abadger1999> yay!
19:09:48 <smooge> yay!
19:09:50 <pingou> I might still tweak a little the api for pkgdb2
19:09:51 <smooge> psych
19:10:02 <pingou> but basically I'm hoping to push pkgdb2 to stg next week
19:10:10 <abadger1999> pingou: Do we need a new python-fedora release to fgo with that?
19:10:17 <pingou> then push it to prod, then take, I don't know, a month off? :)
19:10:44 <pingou> abadger1999: I went the way that pkgdb-cli has the login mechanism embedded, until we can figure out python-fedora
19:10:49 <abadger1999> <nod>
19:11:06 <pingou> so pkgdb2 can move forward :)
19:11:14 <nirik> I see 3 fas hotfixes. I'll ping SmootherFrOgZ to make sure they are rolled into upstream.
19:11:37 <pingou> cool thanks
19:11:56 <pingou> abadger1999: if you could ping fchiulli about the pending pr on elections2
19:12:13 <pingou> ?
19:12:17 <pingou> pretty please :)
19:12:18 <nirik> #info new fas release out soon, stg updated
19:12:25 * pingou cats eyes mode off
19:12:39 <nirik> #info pkgdb2-cli is close to ready and pkgdb2 hopefully to stg next week
19:12:43 <pingou> note for the new fas release, there is a db change
19:13:00 <nirik> ok, so we will likely need an outage for it... for prod
19:13:04 <pingou> https://github.com/fedora-infra/fas/blob/develop/updates/fas2-0.8.9.1_to_0.8.10.sql
19:13:10 <abadger1999> pingou: will do.
19:13:16 <pingou> and the script name doesn't reflect the version change
19:13:19 <pingou> abadger1999: thanks :)
19:14:07 <pingou> oh and there is a fedocal 0.5.1 in stg
19:14:16 <pingou> that I'll push to prod some time soon :)
19:14:41 <nirik> cool. Just some bugfixes? or any new goodies?
19:14:57 <pingou> a little UI improvement and bugfix in the reminder email
19:15:21 <pingou> https://apps.stg.fedoraproject.org/calendar/fedora-meeting/ check the month calendar on the side
19:15:29 <pingou> (Oo the db has been synced from prod to stg?)
19:15:47 <nirik> nice
19:16:32 <nirik> oh...
19:16:46 <nirik> #info bodhi01 / 02 production instances added to the bodhi app pool
19:17:03 <nirik> if everything looks ok, we will drop the app01-04 next week
19:17:10 <nirik> and another thing will be off the app servers.
19:17:25 <lmacken> yay!
19:17:29 <pingou> cool
19:17:52 <abadger1999> Cool.
19:18:26 <nirik> I'll possibly file some more tickets on other things to move off.
19:19:26 <smooge> what is left on the app boxes?
19:19:47 <nirik> whats left: wiki, mirrormanager admin interface, pkgdb (will be done when we move to 2), elections (will be done with the move to new), freemedia, geoip, busmon
19:20:20 <nirik> those last 3 perhaps can go to the same pair of new ones... they are all small.
19:21:08 <nirik> ok, anything else on the application devel side of things?
19:21:17 <willo> just quickly, geoip and freemedia are not on puppettoansiblemigration
19:21:22 <willo> should they be?
19:21:33 <nirik> willo: yeah, we should add them... as apps I guess?
19:21:38 <willo> Will do
19:21:45 <nirik> they are both really simple... one is a wsgi and the other is a cgi I think?
19:21:56 <nirik> #topic Sysadmin status / discussion
19:22:00 <nirik> Speaking of which...
19:22:15 <nirik> willo put together a page for us on puppet-> ansible migration. ;)
19:22:20 * lbazan here
19:22:42 <nirik> https://fedoraproject.org/wiki/Infrastructure/PuppetToAnsibleMigration
19:22:54 <pingou> nice work :)
19:23:58 <nirik> In the last week or two I have also taken work from janeznemanic and adimania and migrated smtp-mm servers and paste over.
19:24:09 <nirik> Thanks for all the help everyone, we are on a roll. ;)
19:24:13 <willo> wiki page updated with geoip and freemedia
19:24:42 <nirik> #info paste and smtp-mm migrated over recently
19:24:42 <willo> they weren't on that list because they weren't on app.fp.o
19:24:53 <willo> should they be there as well
19:25:36 <nirik> willo: I guess we could add freemedia... geoip isn't really something users directly go to. It's used by anaconda to tell timezone/lang on installs and by gnome for maps location or the like...
19:25:56 <nirik> I mean they do directly query it, but you wouldnt normally go there in a web browser.
19:25:58 <willo> i also don't know the URL for those two apps.  so, if someone cam update the page, then that'll be awesome
19:26:49 <nirik> https://fedoraproject.org/freemedia/FreeMedia-form.html is freemedia
19:28:02 <nirik> lets see...
19:28:02 <willo> Ok, I'll add freemedia to apps.fp.o.
19:28:14 <nirik> #info updates applied all around yesterday/day before.
19:28:29 <nirik> we have some new machines rolling in soon... smooge: any news on those?
19:28:33 <smooge> I think I have found a couple of 'solutions' for the download servers
19:28:52 <nirik> smooge: I saw that email with tuning... it's worth trying, but I think we already tried similar.
19:28:54 <smooge> the guys are stuck on another project in PHX1 so haven't had time. I have an open ticket on the new boxes
19:29:03 <nirik> ok
19:29:24 <nirik> #info ansible 1.5 came out recently. Some nice improvements.
19:29:35 <smooge> nirik, this option is mount level and basically tells the nfs side to actually do what the flags we set did on local disks
19:30:09 <smooge> I found another option which is for read-only mounts that don't change a lot. I am not sure about how we fit in that
19:30:10 <nirik> smooge: ok. Do give it a whirl. Anything we can do to make it better without impacting our tier1 mirrors is good.
19:30:57 <smooge> well at least I got something productive this morning... :)
19:31:08 <nirik> FYI, for anyone lost here... we have been having problems with the netapp storage behind our main download servers. It's been very busy. We are trying to make it cache better/behave nicer... while waiting for them to move us to new hardware.
19:32:04 <pingou> eta on the new hardware?
19:32:24 <nirik> hard to say. "soon" they keep telling us. ;)
19:32:48 <mirek-hm> Q: who is sysadmin capable of fixing stuff in fedora cloud in europe timezone?
19:33:21 <smooge> no one?
19:33:45 <smooge> puiterwijk, is in europe timezone but he is not a fulltime sysadmin
19:34:07 <mirek-hm> ok
19:34:12 <nirik> actually I think SmootherFrOgZ has access to the nodes...
19:34:30 <nirik> mirek-hm: I'd be ok with givig you ssh access to the nodes if it would help?
19:35:03 <nirik> I really want to make a new clean cloud on the new machines we get... migrate everything to it, then redo this cloud clean too...
19:35:11 <nirik> so it's nice and redeployable and such.
19:35:59 <smooge> yes.
19:36:11 <mirek-hm> nirik: *nod* I just want to avoid in future, what happen today when FC was dozen hours non-functional and waiting just on nirik
19:37:09 <nirik> mirek-hm: yeah. :( I can add your key to the fed-cloud02.cloud node (it's the controller). Do be very carefull there tho... it's older and manually setup so if we break things it may be hard to fix. ;)
19:37:31 <smooge> s/hard/impossiblish/
19:37:31 <nirik> and sadly it's not often clear what the cause of issues is. :9
19:37:57 <smooge> oh I like that emoticon :9 when I am confused and sad
19:38:36 <nirik> I meant :), but it came out ;9
19:38:43 <pingou> :)
19:38:51 <nirik> anyhow, any other sysadmin related stuff?
19:38:58 <johe> by the way, what type of cloud software do we use?!?
19:39:05 <nirik> johe: openstack
19:39:13 <johe> makes sense :-)
19:39:59 <nirik> oh, related to both ansible and openstack/cloud: I fixed our playbooks for temp cloud instances the other week... they should work nicely now to just spin up something to test with.
19:40:02 <smooge> which explains why my euca notes didn't work
19:40:40 <nirik> I am happy I figured out I could make the second play run as the 'fedora' user with sudo and avoid the anoying root cannot login thing.
19:40:46 <pingou> nice
19:41:05 <nirik> #topic Upcoming Tasks/Items
19:41:05 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
19:41:15 <nirik> any upcoming items folks would like to schedule or note?
19:42:06 <nirik> I did a tiny bit more tweaking on the FAD page...
19:42:16 <nirik> https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014
19:42:26 <nirik> still needs a bunch more work before we can move forward with it.
19:43:04 <danofsatx-work> just for the record, I am here - missed the roll call.
19:43:17 <nirik> hey danofsatx-work. :)
19:44:17 <nirik> #topic Open Floor
19:44:36 <nirik> Anyone have items for open floor? questions? comments? favorate colors?
19:44:57 <smooge> I am rebooting dl05 at the moment. stupid mount
19:45:05 <nirik> fun.
19:45:33 <nirik> ok, if nothing else, will close out in a minute...
19:45:34 <smooge> so when we redo the cloud... will they be in a different ansible tree or the main one?
19:45:46 <nirik> smooge: well, ideally they would be in our main ansible repo.
19:46:00 <nirik> but I can't find a good workable ansible playbook for making them.
19:46:21 <nirik> so, we may end up using some other tool to install/deploy and configure on top of that with ansible.
19:46:21 <smooge> the cloudlets ro the servers on top?
19:46:45 <nirik> installing the actual openstack cloud on the hardware.
19:46:53 <nirik> Managing it after it's up in much easier.
19:47:57 <nirik> but we can figure something out I'm sure...
19:48:20 <smooge> I was thinking we could take one fo the boxes that are under warranty but being replaced and put them up as a lockbox -like box on the network
19:48:50 <nirik> thats not really the problem tho. We can reach them via ssh to configure, it's just that setting up openstack is a pain in the rear.
19:49:00 <nirik> it has 50 components and databases and ...
19:49:25 <tflink> IIRC, it took me a day or so to set up my single-node openstack system
19:49:45 <smooge> oi
19:49:47 <tflink> and that's without a distributed fs or the other components that a multi-node system needs
19:50:11 <nirik> there's 19 services on fed-cloud02 set to start on boot with 'openstack' in their names.
19:50:27 <pingou> 'nice'
19:50:51 <danofsatx-work> what services are those?
19:50:54 <nirik> anyhow, we will get there. I might just use some other scripting and get it initally setup, then use ansible on top
19:51:32 <nirik> http://paste.fedoraproject.org/83130/94135484/
19:51:38 <danofsatx-work> don't need much more than keystone and nova, unless it's a storage node
19:51:57 <nirik> it is. also network, image store, etc.
19:52:23 <nirik> anyhow... we will sort out something.
19:52:29 <danofsatx-work> oh, I see openstack is getting bit by the creep bug, too.
19:52:36 * danofsatx-work sighs
19:52:51 <nirik> danofsatx-work: it's always had a bunch of components. With silly names. ;)
20:09:35 <nirik> #endmeeting