17:59:55 <nirik> #startmeeting Infrastructure (2014-08-14)
17:59:55 <zodbot> Meeting started Thu Aug 14 17:59:55 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:59:55 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:59:55 <nirik> #meetingname infrastructure
17:59:55 <nirik> #topic aloha
17:59:55 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
17:59:55 <zodbot> The meeting name has been set to 'infrastructure'
17:59:55 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:00:03 <smooge> aloha
18:00:07 <nirik> :)
18:00:08 <abadger1999> Greetings
18:00:09 <bochecha> hi everyone
18:00:20 * lmacken 
18:00:28 <adionditsak> Greetings - i will copy paste a small introduction in 1 minute :-)
18:00:59 <adionditsak> I am a very new (1 week old) contributor/ambassador of Fedora, and i look forward to get started for real. I just joined the FI apprentice group, 1 hour ago, and i am excited to dig into stuff. My name is Anders Aarvik, i am from Denmark, and i am a young IT guy with 10 years of Linux usage, lately professionally as a sysadmin (developer earlier). Tell me if you would like to know something else about me. L
18:01:00 <adionditsak> ook forward to talk with you in the future.
18:01:00 <adionditsak> Sincerely,
18:01:01 <adionditsak> Anders Aarvik (adionditsak)
18:01:03 <adionditsak> https://fedoraproject.org/wiki/User:Adionditsak
18:01:05 <adionditsak> https://www.linkedin.com/pub/anders-aarvik/26/133/47b
18:01:20 * lanica is here for the infrastructure meeting.
18:01:46 * adimania_ is here
18:01:53 * mirek-hm is here
18:02:03 * danielbruno here
18:02:07 <ootbro> here
18:02:09 <oddshocks> here
18:02:10 * nmzaheer is here
18:02:12 * relrod here
18:02:28 <nirik> adionditsak: welcome. ;)
18:02:51 <nirik> #topic New folks introductions and Apprentice tasks
18:03:14 <ootbro> From the apprentice front ....   the second diagram is now ready for review at http://www.jt-sw.com/docs/index.nsf/docs/fpo-proxy
18:03:24 <nirik> adionditsak: were you more interested in sysadmin tasks or development
18:03:31 <adionditsak> Thanks :-) Inserted a small intro further up
18:03:48 <adionditsak> Mostly system administration, but maybe a bit of development in the future aswell.
18:03:57 <danielbruno> Hello guys, I contribute to Fedora for 5 ~ 6 years and I intend to contirbute on infra group now
18:04:16 <nirik> ootbro: looks accurate to me. ;)
18:04:33 <nirik> adionditsak: cool. We have #fedora-apps for our application developers too.
18:04:35 <ootbro> thanks.  this version has the extra notes that weren't in the early preview I e-mailed you last week.
18:04:38 <nirik> danielbruno: welcome. :)
18:04:49 <danielbruno> nirik, thank you :)
18:05:08 <threebean> (here)
18:05:23 <adionditsak> Great :-) I will hang out in both channels then. What is the fedora-ops for? Saw it under administration aswell.
18:05:54 <bochecha> adionditsak, operators of IRC channels
18:05:58 <nirik> adionditsak: thats unrelated to infrastructure. thats for irc channel operators.
18:06:10 <adionditsak> Ah.
18:06:12 <danielbruno> more info about me: https://fedoraproject.org/wiki/User:Dbruno and http://danielbruno.eti.br
18:06:16 <nirik> adionditsak: we also have #fedora-noc for sysadmin type things. It also gets alerts and outage info, etc.
18:06:37 <adionditsak> Great - i will make a note on this
18:07:02 <nirik> danielbruno: you were interested in sysadmin tasks? or development? or both? :)
18:07:18 <nirik> danielbruno: I can add you to our apprentice group to look around after the meeting... see me in #fedora-admin.
18:07:39 <danielbruno> nirik, sysadmin tasks
18:08:07 <nirik> ok, great.
18:08:41 <nirik> welcome again to all the new folks. Do hang out in our channels and ask questions anytime...
18:08:58 <adionditsak> Thank you :-)
18:09:07 <nirik> any other new folks?
18:09:51 <nmzaheer> Hi guys..i am just observing for now
18:10:06 <nirik> nmzaheer: ok. :) Feel free.
18:10:09 <nirik> #topic Applications status / discussion
18:10:26 <nirik> Any application development news/plans/discussion this week?
18:10:47 <mirek-hm> I upgraded Copr to new version. Main feature is that we know have queues separate for each architecture, so Copr code is completly ready for new arches. And adding those ARM is just matter of creating playbook for spinning up and terminating (or reprovision) a machine.
18:10:53 <nirik> lmacken: what all do we need to do to get bodhi2 in staging? shouldn't be too far should it?
18:10:58 <mirek-hm> s/know/now/
18:11:02 <threebean> I'm working on some further optimizations to fmn, https://apps.fedoraproject.org/notifications
18:11:04 <nirik> mirek-hm: awesome!
18:11:12 <threebean> it's getting faster but still not yet where I'd like it to be.
18:11:16 <lmacken> nirik: we'll need to port the ansible playbooks, since right now bodhi.dev is setup by hand in a virtualenv with paster
18:11:34 <lmacken> bodhi.dev is also dead in the water right now with an expired koji cert, and I haven't figured out how to fix it
18:11:41 <nirik> #info copr update to seperate queues per arch, hopefully arm support before too long.
18:11:54 <nirik> #info more optimizing on FMN ongoing
18:12:21 <nirik> lmacken: ok. Happy to help with that. I think we can redo bodhi01.stg with bodhi2 as soon as it's ready for it.
18:12:58 <lmacken> nirik: cool, sounds good. we're almost ready to dive into that port, I just want to polish up the masher service and systemdify it
18:13:07 <nirik> fair enough.
18:13:18 <lmacken> or I guess we can just piggy-back on the moksha-hub service instead
18:13:19 <nirik> mirek-hm: did you get any further with cloud networking?
18:14:11 <mirek-hm> nirik: nope. I had to catch up my normal work after Flock (rebuilding SCLs for el7)
18:14:17 <mirek-hm> hopefuly next week
18:14:26 <nirik> ok, no worries.
18:14:30 <mapyth> Hello Everybody!
18:14:30 <mapyth> I would like to update on the status of bugspad. As decided, I was focussing on making the currently implemented
18:14:30 <mapyth> features more robust, and as bug free as possible. In the mean time, my mentor has started doing a parallel work
18:14:31 <mapyth> in python. This would allow to get a performance metric, for Python vs Golang. :)
18:14:43 <nirik> happily it seems like the old cloud has been reasonably stable for copr recently.
18:14:51 <mirek-hm> yes
18:15:23 <threebean> mapyth: cool :)
18:15:23 <nirik> mapyth: cool. I still need to arrange a demo/talk with your mentor. Hopefully tomorrow or early next week.
18:15:36 <threebean> I forget if we've asked already.. but is there a demo instance of bugspad somewhere that we could play with?
18:15:52 <nirik> threebean: I think there is in the cloud... mapyth ?
18:16:05 <mapyth> nirik, threebean Please please check out, http://209.132.184.128/ (my instance)
18:16:28 <mirek-hm> nirik: last day on Flock I discovered https://github.com/openstack-ansible/openstack-ansible-modules , which should easy writing our playbook for our OS instance. asamalik tested it yesterday and it works fine. jruzicka promised that he may try to package it for Fedora
18:16:41 <mapyth> nirik: Currently only basic features are done, and I am testing it for robustness.
18:17:18 <nirik> mirek-hm: oh nice. I think I found that a while back and it wasn't active, but looks like it has been recently now.
18:17:26 <nirik> mapyth: great
18:17:40 <mapyth> nirik: :)
18:18:19 <nirik> #info some progress on new private openstack cloud. Hopefully more next week.
18:18:26 <nirik> any other application type news?
18:18:39 <threebean> mapyth: neat :)
18:18:53 * nirik was hoping abompard would be here and we could talk hyperkitty plans, but will catch him when he's around.
18:19:07 <oddshocks> nirik: Is our internal openstack stuff ready to accept image uploads from Fedimg?
18:19:29 <nirik> oddshocks: we could set that up on the old one... the new one is still in a state of flux.
18:19:37 <oddshocks> cool.
18:19:38 <nirik> oddshocks: what all info do you need? a login/pass?
18:19:52 <oddshocks> nirik: yeah, whatever creds. i can check specifically right now
18:20:16 <nirik> ok. I can get you those... ideally it would upload, but purge old ones or else the list will get very large.
18:20:41 <oddshocks> nirik: yep just looks like auth user and auth pass for openstack clouds
18:20:52 <oddshocks> nirik: right. cool.
18:21:00 <nirik> #info oddshocks and nirik to get fedimg uploading to old infrastructure private cloud.
18:21:04 <oddshocks> I don't know how openstack does image registration but I'll figure it out
18:21:10 <threebean> oh, I almost forgot.  we nearly have rube running in jenkins.
18:21:17 <nirik> excellent. ;)
18:21:18 <threebean> or, it's running.. but still failing on a test or two
18:21:20 <threebean> http://jenkins.cloud.fedoraproject.org/job/fedora-rube/
18:21:34 <nirik> what triggers runs? just timed?
18:21:46 <threebean> so, for people who don't know, rube is an automated test suite for our staging webapps.  it opens up firefox with selenium and tries to click through everything.
18:22:04 <threebean> getting people to contribute more tests would be really helpful -> https://github.com/fedora-infra/rube/
18:22:11 <threebean> nirik: yeah, just timed at the moment.
18:22:28 <nirik> #info more rube tests needed
18:23:05 <nirik> #topic Sysadmin status / discussion
18:23:12 <nirik> ok, on the sysadmin side...
18:23:13 <lmacken> threebean: can you hook me up with admin privs for it?
18:23:19 * threebean nods
18:23:25 <nirik> #info got fasClient fedmsg enabled and running from lockbox01.
18:23:43 <threebean> big win there :)
18:23:50 <nirik> Hopefully this will stop all those fasClient timeouts from when things were running from cron.
18:24:03 <nirik> one nitpick... fedmsg shows me as running the playbook. :)
18:24:19 <nirik> or whatever user restarted the hub on lockbox01 last.
18:24:38 <nirik> #info new retrace machines being provisioned now.
18:24:52 <nirik> #info qa09/virthost-comm03 next to be provisioned.
18:25:34 <nirik> I'm not sure I want to start too much on the sysadmin side this week, as we might go into freeze next tuesday.
18:25:54 <nirik> but the usual porting of stuff from puppet to ansible and rhel6 to rhel7 is still ongoing.
18:26:06 <nirik> #topic nagios/alerts recap
18:26:16 * adimania_ needs to finish a couple of tickets before freeze
18:27:01 <nirik> I also closed out some older tickets...
18:28:05 <nirik> http://tinyurl.com/ms5u5qm
18:28:34 <nirik> hum, that might not be the right report.
18:29:41 <nirik> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=8&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=8&eday=14&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=3&hoststates=7&servicestates=120&limit=25
18:29:46 <zodbot> nirik: http://tinyurl.com/ng2ouvc
18:30:05 <nirik> so, we have gotten a lot of packet loss on vpns, etc...
18:30:11 <nirik> hopefully thats cleared up
18:30:27 <nirik> busgateway01 is still producing a lot of alerts.
18:31:06 <nirik> threebean: can we bump the time on that busgateway monitoring some more?
18:31:36 * threebean nods
18:32:05 <threebean> not sure why its just nauncier that keeps bugging out there.. :/
18:33:00 <nirik> #topic Many many thanks to Toshio
18:33:25 <nirik> As some of you know, abadger1999 is stepping back from Fedora  (hopefully only for a little while)
18:33:39 <nirik> Just thought I would mention it here and say many thanks to him for all his work. ;)
18:33:42 <abadger1999> Thanks nirik :-)
18:34:34 <nirik> If theres anything fedora related you want from him, catch him soon. :)
18:35:14 <nirik> #topic Upcoming Tasks/Items
18:35:15 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:35:27 <nirik> Anything upcoming anyone would like to note or schedule?
18:35:43 <nirik> we may be going into freeze next tuesday. We will see if we have a test compose by then.
18:36:15 * lbazan here
18:36:45 <bochecha> nirik, does staging get frozen as well?
18:36:51 <nirik> also, there is another mass rebuild starting later today... so even though we aren't in freeze, but carefull about any changes that could impact builders
18:37:05 <nirik> bochecha: nope. Staging never freezes. ;)
18:37:43 <bochecha> alright, just wondering if we'd be able to move forward with puppet->ansible migrations during the freeze :)
18:37:47 <nirik> there's a script in ansible repo that lists whats frozen and not
18:37:55 <nirik> yep. we should be able to no problem.
18:38:59 <nirik> #topic Open Floor
18:39:07 <nirik> ok, any items for open floor? questions, comments?
18:39:10 <adimania_> if I can recall, we might not be able to run playbooks on frozen stuff.
18:39:26 <lanica> I've been working on trying to recreate the openid issue as most recently seen on noc01 ... Ticket #4479 https://fedorahosted.org/fedora-infrastructure/ticket/4479
18:39:51 <lanica> I have yet to recreate it in my lab.
18:39:52 <nirik> adimania_: right. When a host is frozen, any changes to it need to be sent to the list first and get at least 2 +1's from releng or sysadmin-main before being applied.
18:40:02 <nirik> lanica: yeah, it's a weird one. ;(
18:40:03 <adionditsak> Do we usually use Puppet or Ansible? I master Puppet, but i am interested in Ansible. Have seen you talk about Ansible, but just logged into a server with puppet agent.
18:40:16 <lanica> I'm not sure how long it takes to trigger or if perhaps it is an issue that involves the front proxy servers
18:40:18 <nirik> adionditsak: both right now. We are moving from puppet to ansible.
18:40:28 <nirik> some hosts are still in puppet
18:40:34 <lanica> If anyone has more data or ideas...please add it to the ticket!
18:40:56 <adionditsak> Ok cool :-)
18:41:22 <lanica> I also do not have access to noc01 to try to collect more information, which adds a degree of difficulty.  If there are other servers that I can be given access to that have had the issue in the past...that would be helpful.
18:41:51 <nirik> lanica: we don't actually have any other hosts using mod_auth_openid. ;( This was the first one.
18:42:10 <lanica> Ok, I somehow missed that
18:42:24 <nirik> lanica: you should be able to login to log01 tho, and look under /var/log/hosts/noc01*/ and see if there's any more info in logs. ;(
18:43:04 <nirik> doubt there will be, but possible I guess. ;)
18:43:10 <lanica> Ok, thanks.  puiterwijk did collect the basic error messages that seemed most relevant and sent them to me and I added it to the ticket.  I'll dig through there some more in case there are some hints.
18:43:29 <nirik> thanks for looking into it
18:43:35 <lanica> NP
18:44:15 <nirik> alright. Anything else? or shall we call it a meeting?
18:45:03 <nirik> Thanks for coming everyone. Do follow up over in #fedora-admin, #fedora-apps and #fedora-noc. ;)
18:45:06 <nirik> #endmeeting