infrastructure
LOGS
18:00:11 <nirik> #startmeeting Infrastructure (2014-08-28)
18:00:11 <zodbot> Meeting started Thu Aug 28 18:00:11 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:11 <nirik> #meetingname infrastructure
18:00:11 <zodbot> The meeting name has been set to 'infrastructure'
18:00:11 <nirik> #topic aloha
18:00:11 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:11 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:00:17 * pingou here
18:00:21 * bwood09 is here
18:00:32 * oddshocks here
18:00:34 * danielbruno here
18:00:35 * lmacken 
18:00:51 * mirek-hm is here
18:00:58 <Neldogz> neldogz is here
18:01:30 * threebean is here
18:01:36 <nirik> cool. Lets go ahead and dive on in then...
18:01:38 <nirik> #topic New folks introductions and Apprentice tasks.
18:01:45 <nirik> any new folks want to intrduce themselves?
18:01:53 <nirik> or apprentices with questions/comments/ideas?
18:02:05 <smooge> here
18:02:21 <nirik> hey smooge
18:02:31 <danielbruno> I have some questions about the Apprentice tasks
18:02:35 * danofsatx present for accounting purposes
18:02:38 <nirik> danielbruno: fire away. ;)
18:02:41 <Neldogz> hello by any chance are there any openings in the Fedora Infrastructure team for volunteers? I have a few hours a week that I would love to dedicate to this team and  the Fedora Project.
18:03:02 * lanica is here for the infra meeting.
18:03:29 <nirik> Neldogz: see http://fedoraproject.org/wiki/Infrastructure/GettingStarted and yes, always looking for folks to help out... are you more interested in sysadmin tasks? or application development?
18:03:30 <danielbruno> so, I read the documentations.. and I would like to know  a good point to start
18:03:45 <danielbruno> if I have to ask in the channel
18:04:30 <Neldogz> hi nirik; mostly in systems administration. I do have experience with RHEL and Fedora. I would love to learn more about how these work in a corporate environment. I am pretty good with monitoring, documentation and up for anythign that is needed.
18:04:34 <nirik> danielbruno: well, best bets are to look at the easyfix tickets and pick something you are interested in and start working on it... and/or hang out in our channels and when you see an interesting to you discussion chime in and ask if you can help out. ;)
18:05:14 <nirik> if there's specific areas you are interested in we might be able to come up with some more targeted things... :)
18:05:38 <danielbruno> great! :)
18:05:45 <nirik> Neldogz: ok, welcome. we have an apprentice program (see link off that getting started page) if you are interested in it, I can add you after the meeting... see me in #fedora-admin.
18:06:06 <Neldogz> great! would love to join fi-apprentice
18:06:12 <Neldogz> see you in #fedora-admin
18:06:27 <nirik> cool. ;)
18:06:58 <nirik> I will also see about filing some more easyfix tickets... haven't done any new ones for a bit.
18:07:28 <danielbruno> nirik, I have looked at the easyfix and i'm interested in some tasks
18:07:28 <vgologuz> hi all
18:08:08 <nirik> danielbruno: cool. Do comment in those tickets and/or ask the person who filed them if they are around on irc...
18:08:15 <nirik> morning vgologuz
18:08:34 <vgologuz> I've recently joined Copr project, and would like (in some time) to learn about fedora infrastructure
18:08:41 <danielbruno> nirik, ok.. I will :)
18:09:01 <nirik> vgologuz: great. :) I can also add you to our apprentice group if you like... see me after the meeting in #fedora-admin.
18:09:37 <vgologuz> nirik, great )
18:10:05 <mirek-hm> I will try to learn vgologuz as much as I know about ansible.git, playbooks and FedoraCloud
18:10:25 <mirek-hm> s/learn/teach/
18:12:04 <nirik> great.
18:12:06 * nirik just had a laptop lockup. sorry for the delay.
18:12:14 <nirik> #topic Freeze reminder
18:12:44 <nirik> So, just a reminder that we are freeze now. Any frozen hosts should not be modified without a patch with 2 +1s from sysadmin-main or releng.
18:12:58 * pingou was the first one to forgot about it
18:13:04 <pingou> it has been so long since the last one :)
18:13:37 <nirik> yeah, I know. ;)
18:13:44 <nirik> see the mail on the list for more details. ;)
18:13:52 <nirik> #topic Applications status / discussion
18:14:04 <nirik> any applications news/status this week?
18:14:08 <lanica> Everything is frozen besides stg?
18:14:10 <lanica> Sorry for the lateness
18:14:34 <pingou> new fedocal released earlier this week, bug fix mainly
18:14:50 <pingou> new pkgdb2 released as well, bugfix again
18:14:59 <pingou> new nuancier released, quite some changes on that one :)
18:15:15 <oddshocks> Fedimg is installed on fedimg01.stg. Working out some Comcast internet problems this moment (I'm currently tethered to my phone 'net), but once they're resolved I'm gonna find out if it's properly doing the thing
18:15:20 <pingou> and elections just got a new election type: irc which offers +1/0/-1
18:15:29 <pingou> work from nerdsville
18:15:35 <oddshocks> (finally got permissions stuff with my sudo access to fedimg01.stg worked out.. I think)
18:15:47 <oddshocks> /s/i think//
18:16:01 <pingou> oddshocks: ansible?
18:16:50 <nirik> lanica: there's a script in ansible called 'freezelist' it will show the ones frozen/not.
18:17:02 <nirik> there are some non stg hosts that also aren't frozen.
18:17:03 <lanica> nirik: thanks!
18:17:16 <nirik> #info fedocal, pkgdb2 bugfix releases this week
18:17:27 <nirik> #info nuancier release with lots of changes.
18:17:39 <nirik> #info fedimg work ongoing
18:17:41 <oddshocks> pingou: what about it? I'm not 100% clear on what the issue was, nirik and smooge resolved it for me
18:18:07 <oddshocks> pingou: i didn't install fedimg with ansible yet, if that's what you're asking. just `rpm -i`'d the rpm
18:18:19 <pingou> oddshocks: that was my question :)
18:18:21 * relrod checks in late, sorry
18:18:27 <oddshocks> I need to figure this ansible stuff out though. I have little to no experience with it sad to say
18:18:31 <oddshocks> relrod: hey
18:18:39 <pingou> oddshocks: installing it via a playbook might have solved the problem there :)
18:18:40 <nirik> oddshocks: we can walk you through it.
18:18:45 <oddshocks> nirik: awesome, thanks :)
18:18:47 <oddshocks> pingou: ahh
18:19:00 <smooge> oddshocks, none of us know how it works. I have 3 voudou dolls which help me
18:19:05 <oddshocks> nirik: pingou: TBH I'm not even 100% clear on which things I should be doing with ansible, and what it can do for me
18:19:06 <pingou> oddshocks: feel free to ping wrt ansible
18:19:09 <oddshocks> smooge: haha
18:19:14 <oddshocks> pingou: cool
18:19:19 <pingou> oddshocks: install rpm, configuration file
18:19:41 <pingou> oddshocks: basically everything that's needed to drop the VM and build a new one working out-of-the-box
18:20:05 <pingou> oddshocks: getting one of those woudou dolls from smooge is a good idea ;-)
18:20:28 <nirik> basically if we had to reinstall the machine today, ansible would let us get it back to the same exact working state.
18:20:33 <oddshocks> pingou: ah that makes sense.
18:20:44 <oddshocks> nirik: right, so yeah just installing the fedimg rpm and putting fedimg.cfg in /etc/
18:20:54 <pingou> nirik: are we doing cloud on kvm? :)
18:20:55 <oddshocks> and likely starting the fedimg relay if it doesn't do so already
18:21:10 <oddshocks> nirik: I'll also ping you soon this week about internal openstack uploading if you'd like
18:21:13 <pingou> oddshocks: sounds nice and simple, shouldn't be too hard to do with ansible :)
18:21:17 <nirik> oddshocks: sounds good.
18:21:24 <nirik> pingou: our cloud uses kvm, yes?
18:21:55 <mirek-hm> yes, OS use kvm
18:21:56 <pingou> nirik: I meant more our stg/prod boxes, we're always "ready" to turn one down and rebuild one, sounds cloudy :)
18:22:16 <nirik> pingou: kinda, but more persistent I think...
18:22:29 <pingou> hopefully :)
18:22:35 <nirik> and more decentralized
18:22:48 <pingou> that's very true
18:23:06 <nirik> any other application news?
18:23:24 <nirik> I'm gonna try and ping abompard next week when he's back about moving more forward on hyperkitty
18:23:35 * pingou looks forward
18:23:42 <nirik> #topic Sysadmin status / discussion
18:23:50 <nirik> lets see...
18:24:19 <nirik> I got bochecha's pkgs migration to ansible commited and made a new pkgs01.stg
18:24:31 <nirik> There's more work we need to do on it, but I think it's coming along nicely.
18:24:53 <nirik> #info pkgs01.stg now in ansible and running rhel7
18:25:08 <pingou> w/ it we're moving to gitolite3
18:25:39 * nirik nods.
18:25:40 <pingou> is that the remaining work you were thinking about or are there also other things?
18:25:52 <nirik> #info fixed a bunch of hosts monday that are still in puppet that were stuck. ;(
18:26:01 <nirik> yeah, mostly gitolite3 and lots of testing.
18:26:08 <pingou> +1 on testing
18:26:29 <nirik> cgit isn't working right for some reason yet.
18:26:30 <pingou> note: the gitolite configuration cron/task isn't on (or working) atm
18:27:02 <nirik> yeah.
18:27:25 <nirik> So, given the freeze we can work on stg stuff and documentation and planning. ;)
18:27:30 <pingou> \ó/
18:27:39 <nirik> #topic nagios/alerts recap
18:27:40 <mirek-hm> I proceed with Openstack installatation. networking is hopefully fixed (although after reset it need manual intervention via console :( ) and I'm currently fighting with one problem in packstack.
18:27:59 <nirik> mirek-hm: great. ;) so it's getting close?
18:28:05 <mirek-hm> yes
18:28:19 <pingou> I've been working with releng to drop the pkgdb2branch script
18:28:20 <mirek-hm> it is my priority now
18:28:59 <nirik> awesome. we still need to hook in storage... but hopefully that won't be too bad
18:29:12 <pingou> the idea is to replace that script by another that checks all git for their branches vs the branches they should have according to pkgdb and adjust the git accordingly
18:29:30 <pingou> so that's one more thing changing with the move of pkgs01 to el7 :)
18:30:06 <nirik> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=8&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=8&eday=28&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=3&hoststates=7&servicestates=120&limit=25
18:30:14 <zodbot> nirik: http://tinyurl.com/m3v3a5x
18:30:59 <nirik> most of our alerts continue to be network related... but I've found that it's likely openvpn getting saturated.
18:31:13 <nirik> I've not isolated what traffic is causing it yet, but will keep looking.
18:31:21 <pingou> threebean: and I are thinking about giving a trial for datanommer/datagrepper on a mongodb backend, that might help with datagrepper's response time
18:32:17 <nirik> ok. I wonder also if moving that db to rhel7 and newer postgres could help
18:32:29 <pingou> could be also
18:33:17 <nirik> #topic Upcoming Tasks/Items
18:33:17 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:33:26 <nirik> anything upcoming anyone would like to schedule or note?
18:34:00 <smooge> not from me
18:34:10 <threebean> pingou: yeah, we should really follow up with yograterol on that.  he has actual experience with mongo and is interested in helping with FI
18:34:57 <nirik> we should make sure we know what we are getting into before we move to it, but trying things seems like a great first step .
18:35:30 <pingou> threebean: +1 but I haven't seen him on irc recently :(
18:36:03 <nirik> #topic Open Floor
18:36:10 <nirik> anyone have anything for open floor?
18:36:20 <nirik> questions, ideas, favorate cartoons?
18:37:38 <oddshocks> I like The Boondocks
18:37:42 <lanica> noc01/nagios mod_auth_openid issue -- looks to be a cobination of a bug in mod_auth_openid and libopkele.  One patch, already in existance for mod_auth_openid fixes part of the issue.  Once I can find some time, I can do some more testing and validate the fix for libopkele.  I probably need to submit a bug report for libopkele if it does not already exist.
18:37:50 <lanica> DuckTales perhaps?
18:38:10 <nirik> lanica: awesome. Thanks for tracking that down... it's an anoying bug. ;)
18:38:17 * nirik likes bob's burgers.
18:39:07 <nirik> alright, will close out in a minute if nothing more.
18:39:49 <nirik> Thanks for coming everyone!
18:39:52 <nirik> #endmeeting