19:00:07 <nirik> #startmeeting Infrastructure (2013-10-31)
19:00:07 <zodbot> Meeting started Thu Oct 31 19:00:07 2013 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:07 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:07 <nirik> #meetingname infrastructure
19:00:07 <nirik> #topic welcome y'all
19:00:07 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
19:00:07 <zodbot> The meeting name has been set to 'infrastructure'
19:00:07 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
19:00:19 * pingou here
19:00:26 * masta waves
19:00:36 * abadger1999 here
19:00:41 <threebean> hi :)
19:00:41 * anshprat here
19:00:44 <smooge> Hello from my phone
19:00:48 * lmacken 
19:00:59 <ianweller> hi
19:01:05 <ianweller> also from phone :I
19:01:12 <ianweller> :o *
19:01:44 <nirik> hello smooge's phone. meet ianweller's phone. :)
19:02:14 <dgilmore> hola
19:02:21 <dgilmore> ¿como estas?
19:02:43 <nirik> :)
19:02:50 <nirik> #topic New folks introductions and Apprentice tasks
19:03:05 <nirik> any new folks want to say hi, or apprentices have questions or comments?
19:03:10 <masta> yes
19:03:18 <masta> I'm new - hello all
19:03:24 <anshprat> hi masta
19:03:40 * croberts waves
19:03:48 * masta waves back
19:03:50 <nirik> hey masta. Welcome
19:03:58 <masta> glad to be here
19:04:08 <nirik> you wanted to help out with the aarch64 stuff with dgilmore ? or did you have other interests as well?
19:04:44 <dgilmore> nirik: im going to get him involved in helping me
19:04:48 <dgilmore> not just aarch64
19:04:54 <masta> will probably just focus on aarch64, and while doing that watch and learn from all of you... pickup tasks where I can
19:04:55 <nirik> excellent.
19:05:13 <nirik> well, welcome and do ask questions as things come up, etc.
19:05:16 <croberts> nirik: i closed that wiki ticket with the timeout i will be working on picking up more and working on them
19:05:42 <nirik> croberts: great. I can dig out the wiki related ones if you want those specifically. ;)
19:05:45 <masta> I'm sure I'll be helping rel-eng alot... that is my goal
19:05:51 <croberts> nirik: yes please :)
19:05:51 <anshprat> askbot - upstream says they have hardcoded version on one of the dependency - and that could be the issue.
19:06:11 <anshprat> so I need to figure out how to fix that without hardcoding version dependency
19:06:11 <nirik> anshprat: hey. :) Back from vacation?
19:06:25 <nirik> #topic Applications status / discussion
19:06:38 <anshprat> nirik: :) yes, but haven't been able to spend much time yet.
19:07:23 <lmacken> bodhi bug #960642 popped back up this week, but I am fairly confident that I squashed it yesterday.
19:07:29 <nirik> anshprat: no worries. I reenabled puppet on it (since it mails every 30min about puppet being off... we can disable it again when you are ready to work on it)
19:07:40 <nirik> #info still working on askbot upgrade in stg
19:07:44 <anshprat> nirik: alright
19:07:54 <nirik> #info bodhi bug 960642 hopefully quashed.
19:08:06 <nirik> Any other application news?
19:08:09 <pingou> fedocal on stg is running a pre-release of 0.3.0
19:08:33 <lmacken> working on the bodhi2 masher as a fedmsg consumer, which will let us take advantage of twisted deferred callback chains, easy threading, etc.
19:08:41 <pingou> and the unit-tests coverage of pkgdb2 bumped to 97% today (with all the precautions we should take on this measurement)
19:08:51 <lmacken> pingou: win!
19:08:53 <nirik> #info fedocal in stg is 0.3.0 prerelease. Please test.
19:09:21 <pingou> lmacken: but that does imply fedmsg becomes a source of information that we rely on, right?
19:09:22 <nirik> lmacken: cool. How far are we from tossing up a cloud instance or something where we can see bodhi2. ;)
19:09:49 * croberts goes afk for a few
19:10:11 <lmacken> pingou: yeah, to trigger a push it would be. we could also potentially avoid the public bus by having the bodhi-client send the message locally on a releng box, etc.
19:10:30 <lmacken> but I think we could do it w/o changing any of the current bodhi masher messages
19:10:41 <nirik> someone at the door here, brb
19:10:42 <pingou> cool
19:11:17 <threebean> lmacken: cool.  if you need to change them though, we can update fedmsg.meta to match it no problem.
19:11:25 <threebean> agility!
19:11:27 <lmacken> nirik: there are no widgets or templates ported, which is probably a little ways off. the json api is solid at the moment, and I'm going to focus on getting headless mashing & API working first
19:11:38 <lmacken> python-fedora compat, etc.
19:11:57 <lmacken> threebean: cool. I'd also want to ensure that the masher message is signed by someone in releng
19:12:01 <smooge> Sounds good
19:12:24 * threebean nods
19:12:46 <threebean> lmacken: yeah, we can make the required cert group readable by an appropriate releng group.
19:12:52 <ianweller> datagrepper 0.2.1 still has most of the new 0.2.x features broken in production, i'll be working on that this weekend hopefully
19:13:02 <lmacken> threebean: perfect.
19:13:02 <ianweller> (at least, openid login is broken)
19:13:18 <ianweller> (this is due to it being two servers and only having tested it in staging on one)
19:13:26 <lmacken> is /raw still sluggish?
19:13:32 <ianweller> yeah i don't know why it is.
19:13:39 <ianweller> it was fine when we upgraded and restarted apache
19:13:40 <lmacken> are we using indexes?
19:13:44 <nirik> cool.
19:13:47 <ianweller> upgraded datagrepper / restarted apache that is
19:13:52 <anshprat> puiterwijk: any news on postja?
19:14:17 <ianweller> so, datagrepper getting really slow / impossible to use is still an open issue
19:14:59 <anshprat> :!
19:15:25 <threebean> lmacken: yes
19:15:31 <smooge> Ianweller if response >20 reboot
19:15:43 <lmacken> threebean: I'm not seeing any in the model?
19:15:46 <ianweller> smooge: basically
19:15:52 <threebean> lmacken: nope.  manually created.
19:16:00 <lmacken> :(
19:16:03 <lmacken> on which cols?
19:16:07 <threebean> timestamp
19:16:14 <threebean> maybe the user and package relations too.. I forget.
19:16:26 <nirik> so whatever became of the idea of checking datagrepper messages stored vs logs or whatever to see if we are dropping anything? I guess thats pending daragrepper being fully working?
19:16:40 <threebean> yeah, I've been running that as of earlier this week.
19:16:49 <threebean> zero dropped messages detected so far.
19:16:55 <lmacken> I'd probably index user.name for sure
19:16:55 <threebean> (I'm just checking koji, though).
19:17:03 <nirik> threebean: nice!
19:17:15 <nirik> could we run a check on other stuff too, just to be paranoid?
19:17:18 <pingou> that's nice
19:17:20 <threebean> https://gist.github.com/ralphbean/7204706
19:17:25 <lmacken> threebean: excellent.
19:17:34 <threebean> everyone run that ^^
19:17:42 <threebean> erm.. please.
19:17:43 <pingou> at the same time ? :)
19:18:31 <threebean> pingou: well, maybe not ;p
19:18:37 <pingou> doh :(
19:18:54 <threebean> I dunno.. it *should* be able to take it, but, as we're saying, datagrepper is feeling sluggish.
19:19:33 <nirik> #topic Applications status / discussion
19:21:31 <smooge> I am not to touch hardware today
19:21:51 <ianweller> threebean: i wonder if the datanommer db is getting deadlocked somewhere
19:22:37 <ianweller> also, surprisingly, that script you posted runs fine on KU EECS fedora machines. apparently the koji client is installed
19:22:41 <abadger1999> ianweller: to test that -- do you have two datanommer frontend machines?  do they both get slow at the same time?
19:22:47 <threebean> ianweller: :D
19:22:51 <abadger1999> ianweller: does restarting one make both fast again?
19:23:00 <pingou> threebean: I got one missed
19:23:04 <ianweller> abadger1999: we have two datagrepper frontend machines. i haven't actually done any real testing, so everything i'm saying is just hearsay
19:23:15 <abadger1999> k
19:23:44 <threebean> pingou: hm. did the script output which thing was missed?
19:23:51 <pingou> * For 2013-10-31 19:20:40.236530 fedmsg did not have https://koji.fedoraproject.org/koji/buildinfo?buildID=475069
19:23:57 <threebean> pingou: and.. was it a race condition?
19:24:05 <pingou> threebean: that's all I have
19:24:14 * threebean check datagrepper for that build
19:24:14 <ianweller> threebean: 475071 koji didn't have
19:24:36 <lmacken> we should sync up the model with the db tweaks by hand, like: __table_args__ = (sa.Index('index_name', 'colname'),)
19:24:44 <threebean> +1
19:24:58 <lmacken> because there may be some low-hanging indexable fruit
19:25:51 <nirik> ok, any other applications news?
19:26:10 <nirik> pingou: how's the jenkins working with the new f19?
19:26:28 <pingou> nirik: so far didn't see/hear any problem
19:26:34 <nirik> cool.
19:26:43 <pingou> more a request for a F20 host :)
19:26:51 <pingou> but I'm not sure we have the IP for it
19:27:06 <nirik> we can get some more I think...
19:27:09 <pingou> so I'm thinking replace F18 by F20 when F20 is beta/release
19:27:11 <nirik> just need to reserve them
19:27:21 <pingou> ok, I'll need to see that with you then :)
19:27:31 <nirik> yep. Ping me out of meeting and we can try and do it.
19:27:50 <pingou> not today, but around half november would be good for me
19:27:50 <nirik> #topic Sysadmin status / discussion
19:27:56 <nirik> pingou: sounds good.
19:28:03 <nirik> lets see... on the sysadmin side...
19:28:35 <smooge> Hardware 100 smooge 0
19:28:45 <nirik> abadger1999: want to share any ansiblefest stuff?
19:28:48 <smooge> :)
19:28:53 * pingou takes the magnet from smooge
19:28:57 <nirik> we have our new bladecenter up and I am trying to get iscsi working on it.
19:29:25 <abadger1999> Lets see... I think we could move the fasClient runs from pull based on each host to push based from lockbox01.
19:29:35 <abadger1999> That might help with our nightly fasClient emails
19:30:17 <nirik> abadger1999: that does mean things have to be ssh reachable from lockbox01, but if they aren't a missed fasClient run is likely the least of their problems.
19:30:33 <abadger1999> nirik: yep.
19:30:57 <abadger1999> nirik: If we have things that are permanently not reachable via ssh from lockbox01, ansible won't be able to work with them at all
19:31:03 <nirik> yep.
19:31:04 <abadger1999> we'll have to configure those hosts differently.
19:31:35 <abadger1999> * f13 pointed me at authprogs https://pypi.python.org/pypi/authprogs as a possible better way to control normal users and scm repos on pkgs.fp.o and fedorahosted.
19:31:38 <nirik> we only have 2 of those that I can think of... the sign-vault machines. On those we only enable sshd when we need to push changes. It's off most of the time.
19:31:49 <abadger1999> It looks promising, puiterwijk was going to look into it.
19:31:53 <abadger1999> <nod>
19:31:57 <nirik> cool
19:32:11 <abadger1999> nirik: do we run fasClient on those two machines regularly or infrequently?
19:32:45 <abadger1999> If regularly, we probably should just leave them pull-based.
19:32:46 <nirik> I'd have to look, I think it runs there... but we could do it as needed, there's few people with access.
19:33:09 * croberts is back
19:33:14 <abadger1999> If infrequently, we can just do fasClient runs at the same time as we explicitly are pushing changes to them.
19:33:19 <abadger1999> <nod>
19:33:21 * nirik nods
19:34:34 <nirik> ok, anything else sysadmin wise?
19:34:36 <abadger1999> For those who don't know, there's the ansible-examples github
19:34:44 <pingou> oh cool
19:34:51 <nirik> yeah.
19:35:02 <abadger1999> https://github.com/ansible/ansible-examples
19:35:19 <abadger1999> It's supposed to have examples that might be good starting points.
19:35:20 <pingou> git grep to the rescure!
19:35:23 <pingou> rescue*
19:35:37 <nirik> yeah, git grep is something I use everyday. ;)
19:35:41 <nirik> very handy
19:36:07 <abadger1999> ansibleworks is going to start up a web service that makes uploading, downloading, rating, etc community provided roles easier in a few months.
19:36:17 <abadger1999> (it'lll be called ansible-galaxy)
19:36:36 <abadger1999> that's about all the big exciting stuff :-0
19:36:38 <nirik> should be a nice source of stuff...
19:36:45 <nirik> abadger1999: whats the licensing on that?
19:36:50 <anshprat> nirik:  http://tech-blog.flipkart.net/2013/10/hostdb/  - will we benefit from something like this rather than just inventory in pvt git?
19:36:55 <nirik> or can people specify?
19:37:43 <nirik> anshprat: we have talked about inventory software in the past, but never found one that really was that nice... I've not looked at hostdb tho.
19:37:46 <anshprat> its a " tool to help manage data center inventory and write applications around  it. - See more at:  http://tech-blog.flipkart.net/2013/10/hostdb/#sthash.qyJQYcfn.dpuf"
19:38:02 <abadger1999> nirik: for the exaples in ansible-galaxy?  I don't know for sure -- I imagined it would be uploader-specifies an OSS approved license.
19:38:11 <abadger1999> like googlecode/sourceforge.
19:38:19 <nirik> abadger1999: ok, just curious
19:38:35 <nirik> anshprat: can take a look and discuss on list?
19:38:39 <abadger1999> Rather than "unless specified, this is CC-BY" like stackexchange, etc.
19:38:41 <anshprat> nirik: sure
19:38:54 <abadger1999> Probably should ping the ansibleworks guys to see if that's the case, though.
19:39:17 <nirik> abadger1999: we do want to make sure, because we would likely add to our repo and it's public...
19:39:36 <abadger1999> Yeah.
19:40:29 <nirik> #topic Upcoming Tasks/Items
19:40:29 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
19:40:37 <nirik> anything anyone wants to schedule or note?
19:40:46 <nirik> I'll note we slipped another week on f20 beta...
19:40:52 <nirik> so, we are frozen another extra week. :(
19:40:53 <pingou> :(
19:40:56 <threebean> :(
19:41:14 <smooge> Jan 1st
19:41:22 <nirik> smooge: of what year? :)
19:41:26 <threebean> smooge: we taking bets?
19:41:34 <anshprat> we have 3 freezes per release I guess ?
19:41:43 <nirik> anshprat: yep. alpha, beta, final.
19:41:55 <nirik> usually they are 2 weeks.
19:42:09 <nirik> but alpha was 3 this time, and beta is 4 now.
19:42:21 <anshprat> so a quarter an year of freezes, atleast.. :!
19:42:40 <nirik> yeah.
19:43:04 <nirik> in a number of ways the freezes are nice tho... forces you to plan things and work on stuff thats not immediately pushing out.
19:43:12 <nirik> just anoying when they get long
19:43:54 <anshprat> yeah..
19:44:03 <nirik> #topic Open Floor
19:44:10 <nirik> anyone have any items for open floor?
19:44:21 <nirik> questions, comments, ideas, book suggestions?
19:44:42 <anshprat> nirik: the feedback from apprentices..
19:44:51 <anshprat> any comments / words on those?
19:45:07 <anshprat> (the monthly mail that you sent..)
19:45:15 <nirik> yeah, some. I suppose I should republish out anything from there that's interesting...
19:45:28 <nirik> but we will be doing a new one tomorrow. ;)
19:45:31 <nirik> or...
19:45:41 <nirik> I could mail apprentices and ask/set replies to the list?
19:46:01 <nirik> then everyone could see and comment? or is that too open for feedback?
19:46:11 <pingou> nirik: propose both
19:46:24 <anshprat> yeah, am fine with that..
19:46:46 <nirik> pingou: yeah, I can...
19:46:47 <croberts> nirik: can i send you an email for a list of the wiki tickets
19:47:00 <nirik> croberts: sure, or I can send you one with them. ;)
19:47:08 <nirik> or we can meet over in #fedora-admin after the meeting...
19:47:59 <croberts> ok sounds good :)
19:48:04 <nirik> #info apprentice feedback mail will optionally go to list next time.
19:48:17 * nirik will close out things in a few if no further comments.
19:48:52 <nirik> thanks for coming everyone!
19:48:55 <nirik> #endmeeting