19:00:13 <nirik> #startmeeting Infrastructure (2014-01-16)
19:00:13 <zodbot> Meeting started Thu Jan 16 19:00:13 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:13 <nirik> #meetingname infrastructure
19:00:13 <nirik> #topic welcome all planets of the solar federation
19:00:13 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
19:00:13 <zodbot> The meeting name has been set to 'infrastructure'
19:00:13 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
19:00:26 * relrod waves
19:00:42 * lmacken 
19:00:44 * abompard is here
19:00:46 * threebean is here
19:00:50 * adimania is here
19:01:04 * johe_ (new one) is here
19:01:36 * jsmith is lurking
19:01:37 <dgilmore> hola
19:01:42 * jlar is new here
19:01:47 <zemmiph0bia> morning :) (New)
19:01:50 * willo is here
19:02:16 * tflink is lurking
19:02:16 <nirik> cool. Nice crowd today. ;)
19:02:30 <nirik> #topic New folks introductions and Apprentice tasks
19:02:46 <nirik> any new folks like to give a short bio/intro? or apprentices with questions or comments?
19:02:54 * nirik sees a number of new folks today
19:03:04 <johe_> okay
19:03:39 <johe_> i introduced already to the list, so hi again, i am a 34 year old systemadministrator and would like to help
19:03:47 <johe_> :-)
19:03:57 <zemmiph0bia> I'm Elizabeta, developer on ratticdb and recovering sysadmin. Wanting to get added to the fi-apprentice fas group.
19:04:11 <nirik> welcome johe_ and zemmiph0bia!
19:04:19 * threebean waves "hello"
19:04:29 <adimania> Hi. I am Aditya. I graduated to sysadmin-badges a few months ago but had to take a break.
19:04:38 <nirik> happy to add you folks to our apprentice group... just see me in #fedora-admin after the meeting.
19:04:39 <threebean> adimania: welcome back :)
19:04:45 <adimania> Will get back in action and start contributing again. :)
19:04:51 <nirik> hey adimania. :) Welcome back
19:05:08 <adimania> threebean, nirik, thanks :)
19:05:25 <jlar> Hi, I'm Jon, I'm looking to keep my skills sharp as my current job doesn't have me doing a lot of linux stuff. I want to help out any way I can.
19:05:30 <nirik> johe_: you are more interested in sysadmin tasks? or application development? or both?
19:05:42 <nirik> zemmiph0bia: you are more interested in sysadmin I guess?
19:05:49 <nirik> welcome jlar.
19:05:58 <smooge> hwelcome
19:06:01 <johe_> nirik, well, the syadmin tasks would fit my skills i guess, as i am doing it every day :-)
19:06:24 <johe_> but i could/would like to take a look on development also
19:06:26 <zemmiph0bia> nirik, both. sysadmin is certainly what I'm better with but I'd like to get more into the development
19:06:35 <nirik> jlar: great. See me also after the meeting in #fedora-admin and we can help get you started. ;)
19:06:40 <nirik> zemmiph0bia: excellent.
19:06:50 <jlar> nirik: sounds good!
19:07:13 <nirik> #fedora-admin is our main channel, we also have #fedora-apps for development related stuff and #fedora-noc for sysadmin/outages/etc
19:07:34 <willo> nirik: I've got a Q when the intros are done
19:07:48 <nirik> willo: fire away.
19:07:57 <willo> I found a link to this page:
19:07:58 <nirik> any other new folks?
19:08:01 <willo> http://infrastructure.fedoraproject.org/csi/security-policy/en-US/html-single/
19:08:15 <willo> from the wiki.  Can't recall exactly where it was.
19:08:37 <willo> There's an associated fedora hosted project - CSI
19:08:46 <nirik> indeed there is. ;)
19:09:08 <nirik> also, we folded it into the main docs too: http://docs.fedoraproject.org/en-US/Community_Services_Infrastructure/1/html-single/Security_Policy/index.html
19:09:18 <willo> Is this current policy on how the servers should be setup?
19:09:41 <willo> Or any other guidance on the SHOULD's, SHALL's and MUST's?
19:09:41 <nirik> in theory. It's something I've really not had time to keep current...
19:10:06 <nirik> if there's people interested in working on it again we could revive it somewhat
19:10:56 <nirik> there's good info in there (although probibly dated now) about how to setup your client machine as an admin... and how to handle security incidents, etc.
19:11:00 <johe_> nirik, i would join in :-)
19:12:32 * nirik ponders the best place to discuss changes there... I guess we could just use the main infra list at least to start.
19:13:03 <johe_> +1
19:13:22 <willo> I droped from channel for last 2 mins
19:13:25 <nirik> so, yeah, read thu and propose changes. ;) it would be great to get it updated.
19:13:27 <nirik> willo: oops. ;(
19:13:31 <nirik> let me repeat:
19:13:39 <nirik> in theory. It's something I've really not had time to keep current...
19:13:44 <nirik> if there's people interested in working on it again we could revive it somewhat
19:13:48 <nirik> there's good info in there (although probibly dated now) about how to setup your client machine as an admin... and how to handle security incidents, etc.
19:14:16 <willo> Cool, great.
19:14:37 <nirik> great topic. Thanks for bringing it up willo
19:15:00 <nirik> #topic Applications status / discussion
19:15:06 <willo> I was thinking that I could wrte some SCAP contentbased on tat
19:15:11 <nirik> any applications news or plans?
19:15:25 <threebean> willo: that would be pretty sweet
19:15:30 <nirik> willo: I think there was some talk about SCAP recently... perhaps ask Sparks when he's around?
19:15:39 <willo> will do
19:16:17 <threebean> on applications new, a new datagrepper and a new fedora-packages got deployed at the end of last week
19:16:22 <threebean> https://apps.fedoraproject.org/datagrepper
19:16:28 <threebean> https://apps.fedoraproject.org/packages
19:16:35 <nirik> I have one bit of application news. :) I've been trying to get the new version of askbot working in our staging setup.
19:16:42 <nirik> threebean: any big changes?
19:16:58 <threebean> enhancements to both.  charul's been working on datagrepper...
19:17:13 * LoKoMurdoK here
19:17:16 <threebean> she added that counter to the frontpage and a new html representation of messages if you look at queries in your browser.
19:17:34 <relrod> threebean: What is the goal date for deploying fmn to prod? Is there one?
19:17:55 <threebean> fedora-packages got a search bugfix - it accepts slashes correctly now.
19:18:02 <threebean> it also displays EPEL bugs in the bugs list (finally)
19:18:03 <nirik> currently: I have askbot working at https://ask.stg.fedoraproject.org/ except for the email sending. Need to figure that our and get it working, then we can repave it and redeploy and make sure it's still good and then look at updating production
19:18:11 <nirik> welcome LoKoMurdoK
19:18:24 <dgilmore> buenos dias LoKoMurdoK
19:18:42 <dgilmore> nirik: i got some emails yesterday
19:18:49 <threebean> relrod: no concrete goal date for fmn -- we never set one (although I think we just said "january")
19:19:13 <nirik> dgilmore: yeah, there was a flood of old ones. I wiped those, but it's not sending any new ones for answers, etc that I can see. ;(
19:19:19 <threebean> pkgdb2 depends on it, but it looks like pingou will be busy with other stuff before he can get around to the pkgdb2 deployment.
19:19:44 <willo> nirik: I was also looking around at askbot in stg.  You saw sme of the results in fedora-noc channel.  I'll try and avoid tripping the orbital defence lasers in the future.  :)
19:20:26 <nirik> note to new folks: do feel free to ask questions about what things are or what someone means. ;) It's always good to explain in more detail if folks want...
19:20:35 <nirik> willo: :) no worries.
19:20:54 <willo> I saw trying to find out how my public key gets from the staging fas into my home dir's authkeys file. Apologies.
19:21:52 <nirik> willo: we can sort it out. I think it's all ok now, but if you still see any issues please let me know.
19:22:28 <relrod> heh
19:22:57 <nirik> oh, in other application news, abompard and I have been exchanging some emails on hyperkitty roll out. I'm likely going to make some production instances soon to start figuring out setup.
19:23:23 <nirik> the biggest problem is going to be setting up just single lists to forward to the new setup from the old... or otherwise make sure they don't interfere.
19:23:42 <nirik> for both web and smtp.
19:23:53 <smooge> cool
19:24:05 <threebean> abompard: how's the caching situation look these days?  iirc, when we first imported messages from the old lists into the staging hyperkitty instance
19:24:09 <nirik> so, we may need folks to put on their thinking caps as to how to do that.
19:24:16 <willo> Dropped again, but just from meeting channel.  No message, just gets quite and then all the nicks disappear.  I'll read what I missed in the meeting notes.
19:24:25 <threebean> .. well, you were looking at caching stuff, no?
19:24:35 <abompard> Caching in hyperkitty is waaaay better now
19:24:41 <threebean> \ó/
19:24:50 <nirik> staging instance: https://lists.stg.fedoraproject.org/archives/
19:24:51 <abompard> I'm using memcached
19:25:02 <threebean> oh yeah.  that's quick :)
19:25:26 <nirik> #info new packages app ( epel bugs and enhancements) and datagrepper (new frontpage and fixes) rolled out last week.
19:25:50 <abompard> threebean: actually the best improvement is when archive are imported
19:25:51 <nirik> #info new askbot in stg is almost working. need to fix email sending then we can schedule prod upgrade
19:26:23 <nirik> #info hyperkitty roll out moving forward. Will make prod instances soon and start figuring out forwarding setup.
19:26:27 <abompard> because I was using some tables as a cache, which needed rebuilding. Now the cache is populated ondemand, from the requests.
19:27:07 <abompard> well there are quite a few non-visible improvements these days. And nirik is rebuilding my test server, which will be useful
19:27:36 <nirik> abompard: lists-dev should be redone, just needs some tweaks on the ansible playbook to finish
19:27:44 <abompard> nirik: yeah, will do soon
19:28:03 <abompard> nirik: I just finish my big commit here, and I'll do that
19:28:23 <nirik> excellent.
19:28:29 <threebean> cool :)
19:29:05 <nirik> abompard: I know our schedules are different, but perhaps next week we could meet up and sort out production forwarding... or perhaps it would be good for me to write up for the list the way things are now so we can figure it there.
19:29:22 <abompard> nirik: sure
19:29:32 <nirik> let me try and do a list post and we can go from there.
19:29:43 <abompard> nirik: great
19:30:04 <nirik> I am sure we can get it working, just will be a matter of the best place to splice in the new one without messing with the old.
19:30:38 <nirik> ok, any other applications news?
19:31:03 <nirik> #action nirik to write up a infra list post about how mailing lists are setup now, so we can decide how to add in mailman3/hyperkitty.
19:31:15 <nirik> oh, another thing:
19:31:24 <nirik> #info new coprs rolled out today
19:31:41 <threebean> relrod: do you have a preference on an fmn deployment goal date?
19:31:58 <relrod> threebean: Not particularly
19:32:15 <threebean> ok, we'll sort it out in #fedora-apps then I guess.
19:32:19 <relrod> threebean: By the end of January works for me, if that is still your goal
19:32:58 <nirik> for new folks: http://copr.fedoraproject.org/ is copr... fmn is a messaging preference application, only in staging so far: https://apps.stg.fedoraproject.org/notifications
19:34:29 <nirik> ok, moving along then...
19:34:32 <nirik> #topic Sysadmin status / discussion
19:34:39 <nirik> lets see...
19:35:07 <nirik> not too much going on right now.
19:35:14 <nirik> I need to finish the ansible cron job...
19:35:26 <nirik> It sounds like folks wanted it to just run --check --diff each day
19:35:46 <nirik> I might try and migrate some more things from puppet to ansible soon.
19:36:12 <nirik> we have setup for those mailman01/02 in production pending
19:36:17 * adimania thinks that is interesting
19:36:45 <nirik> I might file a ticket asking for a script to help you make new instances in ansible.
19:36:57 <nirik> it's just a matter of copying some templates and editing them...
19:37:28 <zemmiph0bia> Are there tickets for the remaining stuff that's left in puppet that needs to get migrated?
19:37:55 * willo was wondering the same thing
19:38:02 <nirik> zemmiph0bia: there's not. it's just been 'as we get to it'. I posted a thing to the list a few weeks ago about the plan moving forward...
19:38:05 <adimania> I'll give it a shot. I understand puppet but I have very basic understanding of ansible.
19:38:18 <zemmiph0bia> adimania, ansible is exceptionally easy.
19:38:21 <nirik> if folks want to help out migrating that would be lovely.
19:38:25 <smooge> I am going to order some hardware tonight (whether it gets approved is another matter). I am doing some certs for coprs and I am catching up with other stuff from extra long vacation/disease marathon
19:38:31 <zemmiph0bia> I'd be happy to help with that, help my partner write a book on it :p
19:38:42 <nirik> zemmiph0bia: awesome!
19:38:59 <nirik> smooge: any news on that hw order? or same old same ol?
19:39:05 <willo> I'm keen to assist in the migration
19:39:28 <willo> Ansible learning experience
19:39:28 <smooge> nirik, I have heard nothing so I am going to do it anyway.
19:39:52 <nirik> some things are pretty trivial to move, just takes a bit of time... some things (like our proxy setup thats in puppet) will likely require more brainpower. ;(
19:40:02 <nirik> smooge: :(
19:40:50 <willo> So, I'm guessing as things are migrated, they're moved out of puppet pretty quickly
19:40:51 <nirik> so folks interested in ansiblle migrations bug me and I can try and get you started. :)
19:41:01 <nirik> willo: yep. We are only adding new stuff to ansible...
19:41:09 <threebean> question -> we still don't have a release schedule for Fedora 21, correct?  which means we don't have a foreseeable infrastructure freeze on the horizon, correct?
19:41:12 <nirik> but we had a lot in puppet, so it's not a quick process
19:41:26 * adimania will go through http://docs.ansible.com/intro_getting_started.html before bugging nirik
19:41:27 <willo> So, anything that I can see in puppet is a list of things that need to be migrated, correct
19:41:28 <nirik> threebean: correct. we are no freezes until we know more.
19:41:33 * threebean nods
19:41:44 <threebean> that's a good thing for the migration effort ;)
19:41:56 <nirik> willo: yep. although... there's also some old junk in puppet that wasn't properly removed when things were retired...
19:42:11 <nirik> for example, I think our asterisk server is still in there, even tho it went away about 5 years ago
19:42:41 <nirik> anyhow, we will get there i hope. ;)
19:42:41 <dgilmore> there is a ton of legacy things in puppet
19:42:48 <willo> :)  just to keep it interesting
19:42:52 <nirik> dgilmore: yeah. just old stuff.
19:43:35 <nirik> I still do want to do a FAD sometime (Fedora activity day, where we get folks to gather in the same physical location and work on one problem). finishing ansible migration is one possible topic.
19:43:47 <nirik> #topic Upcoming Tasks/Items
19:43:47 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
19:43:55 <willo> nirik: will hit you up in fedora-admin later tonight or the weekend.
19:44:01 <nirik> Anyone have upcoming tasks or items they want to schedule or note?
19:44:05 <nirik> willo: sounds good.
19:44:49 <nirik> okey dokey.
19:44:50 <threebean> #info goal: finish round 1 of development and deploy fmn to production by the end of January.
19:44:59 <nirik> sounds good. ;)
19:45:43 <nirik> #topic Open Floor
19:45:48 <nirik> any items for open floor?
19:46:00 <nirik> questions, comments, ideas, youtube videos?
19:46:23 <janeznemanic> nirik: may I ask you what is the purpose of jenkins in our cloud
19:46:51 <nirik> https://fedoraproject.org/wiki/Jenkins@infra
19:47:05 <nirik> see the 'what is' there. ;)
19:47:20 <janeznemanic> great
19:47:25 <willo> I'm a picture person.  Are there currently any diagrams showing interactions between systems?
19:47:44 <willo> I understand that it would be complex, I suppose, I'm not sure just how complex
19:48:12 <zemmiph0bia> http://fedoraproject.org/wiki/Infrastructure/Architecture there is a high overview
19:48:18 <nirik> well, there's: https://fedoraproject.org/wiki/Infrastructure/Architecture talking about our proxy setup, but ti might be pretty out of date
19:48:24 <nirik> yeah. :)
19:48:26 <zemmiph0bia> :p
19:48:29 <willo> Yeah, saw that one
19:48:38 <willo> Would any more diagrams like this be useful?
19:48:48 <nirik> there was someone interested in making a network diagram a while back, but they just wandered off.
19:48:57 <willo> Or is it changing to quickly to keep up to date
19:48:59 <nirik> if someone would like to make one, it could be useful yeah
19:49:01 <threebean> heh, there's a fedmsg diagram -> http://www.fedmsg.com/en/latest/topology/
19:49:05 <threebean> also somewhat out of date
19:49:12 <willo> :)
19:49:27 <danofsatx> Since Nagios is running, can't it spit out a diagram?
19:49:36 <willo> well network is defintely up my alley, I could start there
19:49:40 <nirik> danofsatx: it can yeah.
19:49:51 <nirik> https://admin.fedoraproject.org/nagios/
19:50:15 <johe_> on nagios again? wasnt it zabbix?
19:50:42 <nirik> we were never fully on zabbix. There was an attempted migration, but it was scrapped.
19:50:43 <willo> Looked at nagios before and I got password prompt
19:50:56 <adimania> won't it depend on the parents which we defined?
19:50:58 <willo> guess fi-apprentice doesn't give me the power...:_
19:51:14 <johe_> nirik, well i was on the noc team when they did the switch, was just a bit confused
19:51:35 <adimania> the map at https://admin.fedoraproject.org/nagios/ looks too cluttered :(
19:51:40 <nirik> willo: you should be able to use your fedora account login/pass there.
19:51:52 <willo> I think anything out of nagios is only as good as your service checks and dependancies
19:51:58 <nirik> adimania: yes, it's based on what nagios has... but I think it's actually reasonably accurate now.
19:52:12 <willo> Which can be hard to describe a lot of interactions about data flow
19:52:17 <nirik> johe_: yeah, that was before I was too involved...
19:53:28 <nirik> anyhow, does that answer the question?
19:54:03 <janeznemanic> it's complicated and big
19:54:19 <nirik> any other items for open floor?
19:54:23 <danofsatx> Since I'm here, I'd like to give a quick intro of myself to the infra team. I haven't sent the email to the list yet, though.
19:54:31 <willo> nirik: ah, ok cool.
19:54:36 <kamina1> Hi, I missed the introduction session. So, may I introduce myself?
19:54:36 <johe_> janeznemanic, true, but nagios cant display the relation, such maps are always huge
19:54:54 <nirik> danofsatx / kamina1: welcome. Please do.
19:54:59 <willo> Didn't try that after my mention in fedora-noc :)
19:55:24 <danofsatx> My name is dan, fas dmossor - I am a 20+ year veteran of the IT feild, and maintain three separate mixed CentOS/Fedora/Windows networks in my day job.
19:56:06 <kamina1> Thanks. I am Kamina. Currently, I live in Germany. I am fresh college grad, looking for improving my skills in coding and contribute to open source
19:56:17 <danofsatx> Technically, I am a Cisco certified network engineer, but I can work with anything
19:56:33 <nirik> excellent. Welcome to both of you.
19:56:38 <kamina1> Thanks
19:56:44 <nirik> are you more interested in sysadmin or application development ? or both?
19:56:50 <danofsatx> about the only thing I can't do in IT is coding - my job is to make sure the coders can do theirs.
19:57:04 <kamina1> I am interested in Application Dev
19:57:19 <danofsatx> and I in sysadmin/NOC work
19:57:24 <nirik> cool.
19:57:40 <willo> danofsatx: welcome from a fellow network engineer
19:57:52 <danofsatx> thanks willo ;)
19:58:01 <nirik> for sysadmin, see me after the meeting in #fedora-admin and we can get you added to the apprentice group, for applications, see #fedora-apps and we can get you pointed in the right direction. ;)
19:58:20 <nirik> ok, we are almost out of time... anything else before we close out the meeting?
19:58:33 <threebean> busy meeting today!
19:58:37 <threebean> good to see so many new people :)
19:58:39 <nirik> yeah, great stuff.
19:58:44 <nirik> always good seeing new folks...
19:58:49 <danofsatx> I'll tackle the diagram, in consultation with willo
19:59:14 <relrod> \o/
19:59:16 <nirik> great. Happy to assist.
19:59:22 <nirik> ok, thanks for coming everyone!
19:59:30 <nirik> #endmeeting