18:00:06 <nirik> #startmeeting Infrastructure (2014-09-25)
18:00:06 <zodbot> Meeting started Thu Oct 16 18:00:06 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:06 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:07 <nirik> #meetingname infrastructure
18:00:07 <nirik> #topic aloha
18:00:07 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:07 <zodbot> The meeting name has been set to 'infrastructure'
18:00:07 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:00:09 * relrod here
18:00:13 * lanica is here for the infra meeting.
18:00:13 * bwood09 here
18:00:34 * mpduty .
18:01:05 * pingou 
18:01:37 * nirik will wait another minute or so for folks to wander in.
18:01:45 <nirik> smooge is out today having fun somewhere. ;)
18:01:58 * michel_slm here
18:02:55 <nirik> #topic New folks introductions and Apprentice tasks
18:03:04 <nirik> ok, any new folks like to introduce themselves today?
18:03:12 <nirik> or apprentices with questions or comments or ideas? :)
18:03:20 * michel_slm note the meeting date is incorrect
18:03:43 <nirik> oh, sorry about that. cut and pasta. ;)
18:03:51 <michel_slm> :)
18:03:54 <nirik> I can change it after the meeting is over...
18:04:27 <nirik> #topic Freeze reminder
18:04:36 <nirik> just a quick reminder that we are in beta freeze.
18:04:40 * oddshocks here
18:04:53 <nirik> make sure you don't make any changes to frozen hosts without posting a request and getting approval.
18:05:06 <nirik> #topic Applications status / discussion
18:05:13 <nirik> anything new on the application side today?
18:05:42 <pingou> I have some pull-requests pending review
18:05:53 <pingou> and some more in line for SmootherFrOgZ for FAS3 :)
18:05:58 <nirik> #info pingou has some pull requests waiting for review
18:06:24 <nirik> since we are in freeze there will likely be less in the way of updates in production...
18:06:31 <nirik> but of course things can get tested in staging
18:06:43 * abompard sparsely here
18:07:01 <pingou> trashy worked quite a bit on the UI of progit
18:07:26 <pingou> he redesigned the whole front page
18:07:40 <nirik> cool.
18:08:11 <pingou> and we pushed the list/ticket re-design already in the dev instance
18:08:17 <pingou> which got email notification now :)
18:08:37 <nirik> Sometime we need to have a higher bandwith talk about the copr-git and dist-git and progit and see if we can come up with some roadmap on where we want to go with it all.
18:08:48 <nirik> cool.
18:09:55 <nirik> Oh, also on applications news Koschei folks have started in on the process to get it setup in infrastructure...
18:10:12 <nirik> #info Koschei RFR filed and discussion happening now.
18:10:30 <nirik> #info progit ui improvements and ticket/list redesign and email notifications
18:10:49 * pingou still isn't clear what's the difference b/w a git server and dist-git (wrt copr's need)
18:11:11 <nirik> well, it would have to have some lookaside component, or store big blobs in git.
18:11:33 <nirik> but yeah, need to flesh out exactly what the needs/use cases are.
18:11:41 <pingou> +1
18:12:08 <nirik> IMHO the one thing I would like to have for pkgs is pull request ability.
18:12:18 <pingou> spechub :)
18:12:19 <nirik> so people could submit patches/merge them easier.
18:12:21 <nirik> yeah.
18:12:42 <pingou> I still have a couple of things on the list for that, but the basic idea should be duable easily
18:12:57 <pingou> but maybe we want to do that after the port of pkgs to el7/gitolite3
18:13:01 <nirik> right.
18:13:32 <nirik> ok, anything else new application wise?
18:13:35 <pingou> and I'd like to ask threebean for some help on it :) (wrt caching pkgdb info w/ fedmsg)
18:13:37 <vgologuz> nirik, pingou a always thought, that main reason for dist-git to have some storage to srpm, so that 1) user could upload from localhost 2) doesn't rely on remote host with original srpm in case of rebuild
18:14:08 <nirik> vgologuz: yeah, but perhaps for that use it would be better/easier to just store the src.rpm as is?
18:15:00 * threebean here
18:15:01 <vgologuz> maybe, but we still need to invent some tools for it), so I'm also interested in other use cases
18:15:21 <nirik> yeah. althougth something like the lookaside cache could also just work for src.rpms instead of source...
18:15:38 <nirik> but agreed. Want to build it so it's useful to as many folks as we can. :)
18:15:39 <vgologuz> where dist-git or git-annex more usefull
18:15:45 <michel_slm> main srpm of one bbranch and compat or newer alternate srpm in another branch sharing tarballs perhaps
18:16:10 <michel_slm> e.g. python3 in epel and python in fedora
18:16:41 <threebean> neat idea
18:16:50 <nirik> michel_slm: yeah, in those cases it could just (re) use the existing packages lookaside...
18:16:57 <nirik> if the package source was exactly the same
18:17:13 <michel_slm> ah yes
18:18:03 <vgologuz> michel_slm, hmm but we could have single .spec which build both python3 and python version, and don't build python3 on epel
18:18:38 <nirik> perhaps we should open a discussion on list on this? that way we could perhaps collect some use cases from people who want this...
18:18:38 <vgologuz> (but i think it's a bit of offtopic right now )
18:19:08 <michel_slm> vgologuz: I brainfaded and was thinking of lookaside, nevermind that
18:19:21 <nirik> #info will open discussion about copr git and such for more input.
18:19:29 <nirik> threebean: you have any application news for this week?
18:20:06 <threebean> hm, new staging compose box is up and ready to help debug some mash issues offline.
18:20:24 <nirik> that will be very nice when we get it able to mash. ;)
18:20:40 <threebean> oh, and the anitya backend is making progress in dev, but that won't hit production or anything until a good while after the freeze is up.
18:21:00 <pingou> is anitya froze?
18:21:16 <nirik> so on that, the cnucnu stuff is still going until we fully implement anitya? or ?
18:21:25 <threebean> yeah, good questions.
18:21:27 <nirik> pingou: I wouldn't think so
18:21:33 <pingou> nirik: I would agree :)
18:21:38 <threebean> eh, not me.
18:21:49 <threebean> I'd say we keep running the old cnucnu stuff for now since people are expecting it to work.
18:21:58 <pingou> nirik: yes cnucnu keeps running until we have the full replacement in place, imho
18:22:11 <nirik> good. just confirming. thats what I would prefer too.
18:22:16 <pingou> threebean: you think anitya is froze?
18:22:22 <threebean> I dunno about switching over before the freeze is up.
18:22:42 <threebean> so, "anitya" as in the release-monitoring.org stuff.. I say, no.  it's not frozen.
18:22:47 <pingou> ah, you meant switching, yeah +1 on waiting after freeze yes
18:23:03 * threebean nods
18:23:08 <nirik> theres no great hurry, unless it's all ready and raring
18:23:45 <nirik> #topic Sysadmin status / discussion
18:23:49 <nirik> Lets see....
18:24:03 <nirik> pingou migrated some things (including fedocal) to rhel7 just before the freeze.
18:24:17 <nirik> I guess several other things are done in stg and waiting until after freeze to do.
18:24:44 <pingou> waiting in stg: pkgdb, elections, nuancier
18:24:50 <pingou> fedocal was fully migrated
18:24:56 <pingou> went all nice and smooth :)
18:25:05 <nirik> great, just like we like it.
18:25:22 <nirik> Also, rhel6.6 came out right at freeze and I pushed it out...
18:25:50 <nirik> I was really hoping it would make ansible super fast for us, but there's some issue with ssh pipelining that needs to get sorted out.
18:26:08 <nirik> #info rhel 6.6 updates made just at the start of freeze
18:26:34 <nirik> #info fedocal moved to rhel7. pkgdb, elections, nuancier all done in stg and waiting for after freeze.
18:27:26 <nirik> #info we have 151 rhel 6.6 instances, 76 rhel7.0
18:27:51 <nirik> Also, this week we had another lovely ssl doom. SSLv3 is no longer very safe to use. ;(
18:28:10 <bwood09> nirik, do you have any thoughts on what we briefly discussed on Friday? (ticket 847)
18:28:12 <nirik> We adjusted all our sites to not use it except koji and it seems we are ok on koji anyhow.
18:28:29 <nirik> .ticket 847
18:28:30 <zodbot> nirik: #847 (Require housekeeping for Fedora Hosted.org projects?) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/847
18:28:43 <bwood09> oh, neat; didn't know zodbot could do that, lol
18:29:11 <nirik> bwood09: so, did you have a list of projects we should look at? or were we going to generate one?
18:29:22 <bwood09> We're going to have to generate one
18:29:30 <bwood09> I couldn't figure out a way to collect a list :/
18:29:56 <nirik> yeah, will need to figure out a way to find them...
18:30:25 <nirik> for the wiki page, we might find a project with the default one and download and checksum it. Then, anything with the same checksum has that...
18:30:25 <bwood09> Easiest way, I'm thinking, would to just measure folder sizes; I hadn't had a chance to try it yet
18:30:30 * danofsatx is here, was filing tickets against rolekit
18:30:31 <bwood09> that too
18:30:32 <nirik> yeah, not sure for git...
18:30:46 <bwood09> actually, the checksum thing might work a lot better
18:30:49 <bwood09> hadn't thought of that
18:31:05 <nirik> there might be a way to run a 'git log' and see if it says no commits?
18:31:51 <nirik> not sure on mailing lists either. ;(
18:32:26 * bwood09 looks at git docs
18:32:37 <bwood09> We could always check log size...
18:32:47 <bwood09> but that might not work either
18:32:54 <nirik> for lists perhaps just looking to see if there is an archive...
18:33:01 <nirik> I think it doesn't make one until there's at least one post.
18:33:11 <bwood09> that would make sense
18:33:54 <nirik> or if archive is really small...
18:34:27 <pingou> and the oldest post
18:34:42 <bwood09> that too
18:35:12 <nirik> so, yeah, I would say try and gather a list of those and then we can just look it over and see if it looks sane to close/remove all those.
18:35:24 <bwood09> alright
18:35:36 <bwood09> Do we have a way to notify project owners?
18:36:24 <nirik> yeah, whoever is TRAC_ADMIN for trac, owns the git files group for git, and is -owner for the mailing lists.
18:36:31 <bwood09> alright
18:37:21 <bwood09> Welp, I've got a meeting at Pensacola State in about 20 minutes; I've got to run over there. :/
18:37:26 <nirik> #info bwood09 working on ticket 847 to generate lists
18:37:36 <nirik> bwood09: cool. Thanks for the discussion...
18:37:48 <bwood09> Tonight I'll get back on the fedhosted server and take a look at the projects
18:38:04 <bwood09> Hopefully by tomorrow we'll know if something'll work easily
18:38:22 <bwood09> Thanks, nirik
18:38:22 <pingou> bwood09: let us know if you need help
18:38:28 <bwood09> of course :)
18:38:31 <nirik> thanks
18:38:37 * bwood09 is now away
18:38:38 <nirik> #topic nagios recap
18:38:43 <nirik> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=10&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=10&eday=16&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=3&hoststates=7&servicestates=120&limit=25
18:38:51 <zodbot> nirik: http://tinyurl.com/koh8gx5
18:39:26 <nirik> there's a bunch of proxy08 messages in there... I'm not sure why it was freaking out totally. Althought I have a theory now I should test...
18:39:52 <nirik> load on it was very high. I noticed there's a collectd selinux thing now with rhel6.6 hosts. Might have been causing it some issue...
18:40:32 <nirik> anyhow, here's hoping to a quiet week or two with nagios. ;)
18:40:34 <pingou> the second on nuancier makes sense, since it's used only once every release
18:40:49 <pingou> but we should probably increase the limit then
18:40:58 <nirik> I don't think it's hitting the time limit...
18:41:13 <nirik> might have been related to moving the stg one to rhel7?
18:41:19 <nirik> not sure
18:41:29 <pingou> it's datanommer
18:41:43 <pingou> OK: datanommer has not seen a 'nuancier' message in 54 days, 9 hours, 52 minutes, 10 seconds
18:41:46 <pingou> looks OK :)
18:42:10 <threebean> it times out from time to time due to how long in the past the nuancier messages are.
18:42:27 <threebean> it just takes postgres a while to respond, afaik.
18:42:28 <pingou> ah ok
18:42:30 <nirik> perhaps we should make nuancier emit a ping message every so often or something.
18:43:05 <threebean> or remove the check ;)
18:43:15 <pingou> threebean: do we want to try a new pg version at one point?
18:43:18 <nirik> true, I guess thats an option...
18:43:20 <pingou> to test speed ?
18:43:26 <threebean> it's not all that useful.. nuancier would have to be broken for 6 months or something in order for it to notify us.
18:43:38 <nirik> yeah, removing +1 then
18:43:45 <pingou> wfm to
18:43:47 <pingou> +o
18:43:51 <nirik> pingou: it's on rhel7 postgres now... which was much faster I think?
18:43:54 <threebean> long term, we should try the new bjson column types in modern postgres
18:44:42 <pingou> cool
18:45:02 <nirik> #topic Upcoming Tasks/Items
18:45:02 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:45:07 <threebean> https://www.youtube.com/watch?v=6-2tpHLLM1o
18:45:07 <nirik> anything upcoming anyone would like to schedule or note?
18:45:33 <nirik> I have one: I am going to be out next wed, should be around thursday morning and thru the next weeks meeting, and then will be out friday. ;)
18:45:43 <nirik> so, if you need me for something, see me before then.
18:46:47 <threebean> cool.  nothing scheduled or upcoming here.. ;)
18:47:52 <nirik> #topic Open Floor
18:47:57 <nirik> anyone have anything for open floor?
18:48:09 <pingou> FAD plans getting along
18:48:34 <pingou> I have the green ligth to book the rooms, hopefully we'll have the green light to buy the tickets soon :)
18:49:04 <pingou> https://fedoraproject.org/wiki/FAD_MirrorManager2_ansible-migration_2014 got renamed and updated
18:49:21 * pingou eof
18:49:33 <nirik> cool. ;)
18:50:04 * relrod won't be able to make the FAD :(
18:50:10 <relrod> too close to finals week
18:50:17 <mpduty> can one participate remotely?
18:50:18 <pingou> relrod: arf :(
18:50:20 <pingou> mpduty: sure
18:50:51 <pingou> we'll make sure to email/blog the schedule and how to join in
18:50:54 <nirik> absolutely.
18:50:56 <nirik> yeah.
18:51:10 <mpduty> thats nice
18:51:15 <nirik> ok, if nothing else, will close out in a minute or so.
18:51:56 <nirik> Thanks for coming everyone!
18:51:58 <nirik> #endmeeting