18:00:03 <nirik> #startmeeting Infrastructure (2015-01-29) 18:00:03 <zodbot> Meeting started Thu Jan 29 18:00:03 2015 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:03 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:03 <nirik> #meetingname infrastructure 18:00:03 <nirik> #topic aloha 18:00:03 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:00:03 <zodbot> The meeting name has been set to 'infrastructure' 18:00:03 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:00:10 * pingou here 18:00:14 * sborza here 18:00:20 * fale here 18:00:25 * lmacken 18:00:34 * puiterwijk here 18:00:37 * michel_slm here 18:00:38 <dgilmore> hola 18:00:41 <taedori> here 18:01:05 <Corey84> .hello corey84 18:01:09 <zodbot> Corey84: corey84 'Corey84' <sheldon.corey@gmail.com> 18:01:11 <Corey84> may be afk at times 18:01:13 <janeznemanic> hi 18:02:36 <nirik> #topic New folks introductions and Apprentice tasks 18:02:46 <nirik> any new folks like to introduce themselves? 18:02:52 <nirik> or apprentices with questions or comments? 18:03:07 * threebean waves 18:03:19 <Corey84> o/ 18:03:19 <smooge> here 18:03:37 <puiterwijk> zimbr 18:03:42 <Corey84> no need to "raise hands" here new comers 18:04:19 <nirik> :) 18:04:32 <puiterwijk> errr, sorry for the spam :) 18:05:11 <taedori> Nice meeting you guys. I am from C# .Net world. actually transitioning to linux world. I am learning python. and want to contribute in near future. 18:05:23 <nirik> taedori: welcome. :) 18:05:44 <nirik> taedori: do hang out in our #fedora-apps channel and ask questions... and folks there can point you at some places to start. 18:06:17 <taedori> yes sure. 18:07:34 <nirik> #topic Applications status / discussion 18:07:41 <nirik> any applications news to talk about this week? 18:08:32 <threebean> lots of stuff happened today.. ;p 18:08:33 <nirik> There were some minor releases of a few things... 18:08:38 <nirik> and yeah, today has been busy. ;) 18:08:58 <pingou> #info new anitya just hit prod, 0.1.16 18:09:04 <threebean> we got a new fedmsg message type from koji earlier in the week -- one for when rpms are gpg signed by our signing server, sigul. 18:09:11 <threebean> and an FMN release went out with a new rule to ignore that message.. 18:09:26 * mirek-hm is here 18:09:50 <lmacken> threebean: will people need to reset their filters to the defaults to pick it up? 18:09:50 <pingou> #info new koji release sending fedmsg message when a package is signed in sigul (our signing server) 18:10:00 <threebean> lmacken: yup. or add the new rule by hand. 18:10:06 <pingou> #info new fedmsg_meta release to handle this new message 18:10:25 <pingou> #info new FMN release to be able to include/exclude these new messages 18:11:06 <pingou> #info new python-fedmsg-genacls release to be used on RHEL7 with gitolite3 18:11:27 <Corey84> taedori, may have some great pyhton epub copies for you later 18:11:38 <threebean> #info new the-new-hotness release in staging that auto-adds packages to release-monitoring.org when they're added to Fedora 18:11:59 <mirek-hm> copr sign new packages by GPG key, old packages are still unsigned so we did not put gpgcheck=1 to .repo file. 18:12:00 <nirik> Oh, I also created a mirrormanager2 mirrorlist server... it still needs some work, but we have something to try and test out with now. 18:12:14 <nirik> #info copr now signs new package builds. 18:12:18 <nirik> mirek-hm: great stuff! 18:12:33 <taedori> Thanks Corey84 18:12:37 <pingou> #info new mirrormanager2 based mirrorlist server 18:13:08 <Corey84> nirik, if my server is not showing in manager2 list let me know I plan to go full /alt/pub when Southern_Gentlem drops the initial 21 updates next week 18:13:15 <nirik> threebean: is that the thing that sometimes says "new upstream release of foo, but cannot figure out what it's called in Fedora?" 18:13:33 <Corey84> im presently showing private only i believe 18:13:46 <nirik> Corey84: this is just the mirrorlist part... the part that sends users the list of mirrors. ;) 18:13:56 <nirik> but yeah, we can check yours 18:14:09 <threebean> nirik: no, that's the other was around.. the project exists in release-monitoring.org, but there's no mapping to the project's name in Fedora. 18:14:34 <threebean> we could/should write a script that tries to get that mapping as close to 1-to-1 as possible. 18:14:36 <nirik> threebean: ok. I noticed that I think on a 'chrome' package... which of course we don't ship 18:14:43 <threebean> right 18:14:46 <Corey84> kk 18:14:57 <lmacken> the recent koji update also included the runroot plugin, which once configured will let us do things like mash & compose atomic trees 18:15:07 <Corey84> was thinking you meant mirrormanager-client v2 my bad 18:15:12 <nirik> lmacken: yep. Yea! 18:15:23 <pingou> lmacken: cool! 18:15:30 <nirik> Corey84: the rest is coming too, we just are doing the lists first. ;) 18:15:37 * tflink is getting around to adding changes to the ansible playbooks to support local taskotron deployments 18:15:49 <tflink> not sure if that's of much interest to the folks here, though 18:16:06 <Corey84> my bad ill get back in the carriage and stop tugging the on the bridle :P 18:16:10 <nirik> tflink: oh, I noticed the other day that autoqa was still/again up... is that supposed to still be around/ 18:16:12 <nirik> ? 18:16:16 <nirik> autoqa01 that si 18:16:24 <Corey84> tflink, need a hand let me know 18:16:32 <tflink> no, I don't think we still need it 18:16:47 <tflink> making that more definite - no, we don't still need it 18:16:58 <nirik> cool. I can quash it, or you can 18:17:35 <nirik> ok, anything else application wise? 18:18:00 <nirik> It seems out hyperkitty is delayed, but hopefully will be possible by pycon 18:18:07 <nirik> There was a blog post from abompard_ 18:18:34 <Corey84> pycon is when again 18:18:35 <nirik> Not sure where we are on bodhi2... need packaging work? 18:18:40 <tflink> nirik: I'm not sure what the process for that is off the top of my head, if you have the spare cycles, feel free. otherwise, I'll figure it out 18:18:45 <lmacken> Corey84: apr 9-17 18:19:10 <threebean> http://aurelien.bompard.org/post/2015/01/23/News-of-the-Mailman3-front 18:19:15 <nirik> mirek-hm: new cloud is looking promising this time? ;) 18:19:41 <lmacken> nirik: we've been working on some of the email plumbing this week, and I need to forward-port a handful of features that crept into bodhi1 before we get it in staging 18:19:50 <mirek-hm> nirik: no promises. I'm again fighting with rabbitmq 18:19:59 <nirik> lmacken: cool. does the runroot stuff help you out any? 18:20:18 <nirik> mirek-hm: ok. ;( at least we seem to be getting further and iterating faster. ;) 18:20:34 <mirek-hm> nirik: but OS have my FULL attention and top priority for me 18:20:47 <lmacken> nirik: yeah, once it's configured I'll be able to use it to mash pretty much immediately. Assuming in the config file we have it nfs mount /mnt/koji/mash/ 18:20:52 <nirik> yeah, I really really want to decommission the old one 18:21:11 <puiterwijk> mirek-hm: if you need any help, just let me know. I'm more than happy to help wherever I can 18:21:22 <nirik> mirek-hm: me too. 18:21:30 <tflink> is the hoped-for deployment date of bodhi2 still during the f22 cycle? 18:21:52 <lmacken> tflink: That's what we're shooting for 18:22:06 <nirik> tflink: I think we should know more once we have a actual stg instance... 18:22:08 * tflink questions the wisdom of doing that mid-release 18:22:17 <nirik> then we can see what needs to be done and how much issue we are hitting 18:22:42 <nirik> yeah, we want to make sure it's ready. 18:22:55 <tflink> but I've not been involved with bodhi2, so I might just be overly-cautious 18:23:37 <mirek-hm> puiterwijk: nirik: I will ask when I reach dead end. But it is most time consuming. Change something run packstack, wait 20 minutes; see if the change changed something and if changed something then check what else was broken this time... and during that 20 minute you should be strong enough to not look on G+ or FB or you will not return to work after those 20 minutes :) 18:23:51 <nirik> threebean: whats the status/timeline on FMN by default + anytia? still shaking out issues? 18:24:03 <lmacken> as long as the bodhi API remains stable, we'll have to see how many other things explode to really see how easily we can pull the switch 18:24:17 <pingou> nirik: I still have 1 issue to fix on anitya for the backend 18:24:25 <nirik> mirek-hm: yeah :) 18:24:33 <threebean> nirik: moving as quick as we can before freeze. 18:24:45 <Corey84> +1 on tflink reluctance 18:24:46 <threebean> fmn is already 'default' in so far as new packagers are automatically set up. 18:24:50 <nirik> cool cool. Just wanted to know if we needed to punt further. 18:25:15 <threebean> I'm just looking for a round tuit to edit and run the script to add all the existing packagers (to fmn). 18:25:22 <tflink> I'm just worried about the possibility of something going wrong during freeze and causing slips 18:25:30 <Corey84> not that we can't do it or do it gracefully but why needlessly add the stress 18:25:34 <nirik> tflink: a completely valid concern 18:25:53 <Corey84> more worried of slips personally not the freeze 18:25:56 <threebean> pingou: remind me what the last issue on anitya is? 18:26:06 <pingou> threebean: urls with * in them 18:26:14 * threebean nods 18:26:21 <threebean> and we can't take hotness to prod without that? 18:26:35 <pingou> we could but I'd prefer to try to tackle it before 18:26:41 <threebean> sure thing. 18:26:46 <pingou> but it's been running like this for a little time now :) 18:26:59 <nirik> alright, any other application side things? or shall we move along to sysadmin? 18:27:29 <nirik> #topic Sysadmin status / discussion 18:27:40 <nirik> So lots of fun this week... 18:27:59 <nirik> #info short mirrorlist outage on tuesday. We think cause was a bad pkl, but not sure. 18:28:15 <nirik> #info glibc updated for ghost and all servers rebooted on tuesday. 18:28:46 <nirik> #info pingou and bochecha have been working on pkgs01.stg and it's ready for wide testing now! Hurray! 18:29:23 <michel_slm> hurray! 18:29:27 <nirik> #info fedoramagazine had some dns issues, but they should be all cleared up 18:29:47 <nirik> #info proxy03 and proxy04 were reinstalled rhel7/ansible and so far seem to be working well. 18:30:10 <nirik> #info koji01/02 were installed as rhel7/ansible and we switched to them in production. 18:30:28 <nirik> #info all builders and hubs were updated to latest git head koji packages 18:30:42 <Corey84> was that the one we were all checking dnssec tues for nirik ? 18:30:44 <nirik> so, we should have that runroot plugin (but it's not configured yet) 18:31:01 <nirik> Corey84: oh yeah... 18:31:16 <nirik> #info letoto has loaded all fedora folks gpg keys into dns now. 18:31:28 <nirik> ^ thats the dnssec thing. 18:32:12 <nirik> #info down to just 44 hosts in puppet. 18:32:36 <nirik> #info relrod fixed up ntp on hosts 18:32:56 <nirik> we are going to see if the old 'guest can get its time from the host' thing works or if we need to run ntp on guests too 18:33:25 <nirik> I fixed up some idempotentcy issues with ansible playbooks last weeked. 18:33:30 <nirik> there's still more to clean up though. 18:33:53 <threebean> any outstanding projects left for the port to ansible? 18:34:11 <michel_slm> is there a list of Ansible tasks somewhere? 18:34:12 <threebean> we can slowly move the proxies over as we gain confidence that they're not broken. 18:34:21 <nirik> mostly I think we are down to remote sites where reinstalling is a pain 18:34:47 <nirik> oh, I made a db01.stg a while back, but still there's dbs on db02.stg that need migrated. 18:35:10 <nirik> Hosted is still on the agenda. That will be some work. 18:35:26 <nirik> we need to build out trac for rhel7 for it, and then setup playbooks, etc. 18:35:44 <nirik> we need a kojipkgs playbook/setup. 18:35:58 <nirik> (thats just apache/squid converted from puppet, shouldn't be too hard) 18:36:10 <nirik> people still needs doing 18:36:11 <smooge> threebean, I started on people and got stuck.. I think the proxy stuff might help me 18:36:20 <threebean> nice, nice. can test kojipkgs in stg too. 18:36:20 <nirik> torrent 18:36:38 <smooge> #info DNS zones have been cleaned up so they all refer to the same set of proxies. 18:36:47 <nirik> and some releng machines... which need to stay rhel6 as bodhi1 needs to run on them for now. 18:37:07 <smooge> #info DNS templates code has been fixed so that zones don't give "this serial is out of bounds rejecting zone." 18:37:12 <nirik> we are reaching the end. ;) 18:37:37 <threebean> :p 18:37:40 <threebean> FAD made a big difference. 18:37:45 <dgilmore> nirik: torrent needs to go away or we need to redo it 18:37:57 <nirik> http://paste.fedoraproject.org/177673/42255665 is all the hosts left in puppet. 18:38:09 <nirik> dgilmore: I wish we could drop it, but everytime we suggest it, tons of people yell. 18:38:39 <dgilmore> nirik: right. we did talk of using a service provided by ibiblio at some point 18:38:55 <nirik> yeah, and then the guy who was doing that left ibiblio and the service died. ;( 18:39:10 <dgilmore> :( good thing we did not move then 18:39:16 <nirik> yeah, no kidding 18:39:27 <dgilmore> so I guess torrent needs someone to redesign it 18:39:28 <nirik> anyhow, we can investigate. perhaps there's better software now 18:39:31 <nirik> yep 18:39:52 <dgilmore> I have a few complaints about the usage 18:40:00 <nirik> #info torrent setup needs a redesign. 18:40:18 <nirik> anything else on the sysadmin side? 18:40:37 <dgilmore> nirik: one quick thing 18:40:43 <dgilmore> maybe not meeting worthy 18:40:49 <nirik> sure thing, go ahead 18:41:02 <dgilmore> need to work out a way to get pbrobinson onto the builders 18:41:24 <dgilmore> I see two ways, add him to sysadmin-main or add fas etc on to the builders 18:41:26 <nirik> dgilmore: we could easily add fasclient back and add sysadmin-build or whatever 18:41:30 <nirik> yeah 18:41:41 <dgilmore> and let sysadmin-releng login and sudo 18:42:13 <dgilmore> I do not mind too much how we do it but it has to be done 18:42:39 <dgilmore> i guess another option is have a releng-lockbox 18:42:44 <nirik> I'd be fine adding fas back for now... not that I don't think he would be fine in main, but typically we wait until someone has done some big project/work before adding them. 18:42:49 <dgilmore> and allow root from it to ssh to the builders 18:43:09 <smooge> I thought sysadmin-releng had sudo at one point 18:43:11 <nirik> that seems like it might be more complicated... 18:43:24 <Corey84> dgilmore, torrents need to go ? 18:43:34 <dgilmore> smooge: it was removed when skvidal redesigned the builder setup 18:43:36 <smooge> yes. I have a long diatribe on them 18:43:45 <smooge> dgilmore, got it 18:43:52 <nirik> yeah, builders currently have just root. Nothing else. 18:44:00 <smooge> Corey84, yes. I have a long diatribe on them. After meeting 18:44:04 <Corey84> if someone wants to help with the layout side (ui) I can throw some time in on that 18:44:08 <dgilmore> I kinda like not having accounts on it 18:44:17 <Corey84> smooge, kk 18:44:30 <dgilmore> but setting up a box just to allow some folks access seems a bit of overkill 18:44:32 <Corey84> accounts on torrents ? new to me 18:44:43 <smooge> no we moved to build system :) 18:44:45 <dgilmore> Corey84: no accounts on the builders 18:44:55 <Corey84> builder servers? 18:45:03 <dgilmore> yes 18:45:04 <nirik> dgilmore: right. I am leaning toward fas on them again, but we could discuss more out of meeting? 18:45:24 <dgilmore> nirik: sure. just wanted to bring up we need to do something 18:45:33 <dgilmore> details can be sorted 18:45:43 <nirik> ok. yeah, I also need to get together with him and work on ansiblizing aarch64 boxes 18:45:49 <nirik> yeap 18:45:52 <nirik> #topic Upcoming Tasks/Items 18:45:52 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:46:01 <nirik> anyone have any upcoming items they want to schedule or discuss? 18:46:15 <nirik> note that smooge and I will be at our main datacenter the week of feb 9th 18:46:23 <nirik> branching is also that week. 18:46:28 <pingou> FOSDEM this week-end 18:46:29 <threebean> all those EU events are coming up soon, right? 18:46:32 <puiterwijk> Staging upgrade to Ipsilon can finally take place now that we have a EPEL7 build, so will be doing that this week or early next week 18:46:34 <pingou> DevConf the week-end after 18:46:41 <Corey84> +1 dgilmore a bit overkill it seems 18:46:41 <nirik> threebean: yeah 18:46:53 <threebean> pingou, puiterwijk: have fun :) 18:47:01 <pingou> we were discussing if we could move pkgs01.stg to prod on Monday 18:47:02 <puiterwijk> CfgMgmtCamp next monday and tuesday, and infra.next on wednesday 18:47:11 <pingou> giving us 2 days to fix things before travel to DevConf 18:47:13 <nirik> puiterwijk: excellent. was going to ask about that... let me know if there's anything I can do to help with it. 18:47:15 <pingou> threebean: will do :) 18:47:23 <dgilmore> nirik: might try to get with you and make sure the aarch64 boxes are rebuilt then 18:47:39 <puiterwijk> nirik: will do, though most of it is just going to be porting templates and configuration 18:47:45 <Corey84> Cydrobolt, and I were on the mail list about some upcoming hackathons and makerspace sponsorships ( more an ambass. thing but figured i'd mention ntl) 18:47:46 <dgilmore> nirik: pbrobinson and I will be working together in Brno that week 18:47:47 <nirik> dgilmore: yeah, they need to be... one of the things we are going to do in phx2 is move them tho to a common rack 18:48:20 <puiterwijk> (Ipsilon might be "FedOAuth 3", but it's a major change, so none of the config or ui templates look alike) 18:48:27 <nirik> cool. 18:48:33 <dgilmore> need to get a plan to redo all teh secondary arch setup and ansibilise it all 18:48:33 <pingou> puiterwijk: nice :) 18:48:51 <Corey84> +1 ansibilise secondary 18:48:57 <nirik> threebean: you game to talk about github2fedmsg for a few? 18:49:01 <threebean> sure :) 18:49:09 <nirik> #topic learn an app - github2fedmsg 18:49:10 * pingou has a meeting in 10minutes 18:49:19 <nirik> yeah, sorry, not much time... 18:49:27 <Corey84> same here ....use github /bitbucket religously 18:49:27 <threebean> so, github2fedmsg is a web service that bridges github activity onto the fedmsg bus 18:49:45 <threebean> it has a little self-service dashboard which you can access here 18:49:47 <threebean> https://apps.fedoraproject.org/github2fedmsg 18:50:08 <threebean> you login with FAS, then login with GitHub (it keeps track of the association between your accounts there) 18:50:14 <Corey84> so you setup as an appication (like tfa on GH token) ? 18:50:36 <Corey84> s/appication/application 18:50:42 <threebean> Corey84: right. yeah, and it's already setup with an application key in our infra. for users, they don't have to worry about that part. 18:50:54 <Corey84> sweet 18:51:10 <threebean> it uses that application key and the oauth token it gets when you login to install webhooks on the github repos you choose. 18:51:14 <Corey84> nice had to walk some kids through that last night not fun 18:51:22 <threebean> it can be a mess ;) 18:51:47 <threebean> so then with those installed webhooks, github will POST back to the webapp in our infrastructure about events as they happen. 18:51:49 <Corey84> whoever set all that up so nicely deserves a brew or two :) 18:52:11 <threebean> the github2fedmsg app receives those POSTs, verifies their signature is really from github, then re-signs them with a fedmsg key and broadcasts them on our bus. 18:52:14 <Corey84> so just sub to the respective whooks 18:52:16 <Corey84> ? 18:52:25 * threebean nods 18:52:29 <threebean> yup 18:52:42 <nirik> this can be any github repo anyone cares about? or you have to be an admin for that github project to add the hooks? 18:52:55 <threebean> nirik: hm, you have to have admin on that github project, yes. 18:52:58 * Corey84 feels a warp drive speed increase in workflow coming when fully used 18:53:32 <threebean> nirik: so you can add your "own" repos as well as repos of any organization you have sufficient rights in. 18:53:36 <Corey84> much like sponsorship for maintainers tho someone can add you or whitelist you tho 18:54:09 * Corey84 will be initiating later today when he has 5 mins to spare 18:54:36 <threebean> the source code for it all is here -> https://github.com/fedora-infra/github2fedmsg 18:55:09 <threebean> this is maybe the most interesting block -> https://github.com/fedora-infra/github2fedmsg/blob/develop/github2fedmsg/views/webhooks.py#L69-L136 18:55:16 <threebean> any questions or anything? 18:55:38 <nirik> agpl ? I guess we never should hotfix it, but make new releases. ;) 18:55:52 <threebean> ah, it's agpl with that extra clause iirc. 18:56:03 <threebean> the one that gives us 30 days to get our srpms in line. 18:56:10 <nirik> oh right 18:56:33 <nirik> cool. Thanks threebean! 18:56:37 <threebean> np! 18:56:38 <nirik> #topic Open Floor 18:56:49 <nirik> just a few minutes left for open floor. Anyone have anything? 18:56:55 <tflink> who all is going to be at devconf next week? 18:57:08 <nirik> puiterwijk and pingou I think... 18:57:28 <puiterwijk> stickster as well, I think 18:57:40 <nirik> and dgilmore ? 18:57:45 <puiterwijk> oh, and I heard jwb is there 18:57:49 <puiterwijk> I think Dennis as well yeah 18:58:03 <tflink> k, I want to run some of our taskotron plans past someone to make sure they aren't any crazier than I think they are and that'd be easier in person 18:58:40 <nirik> tflink: BTW, I pinged on that fw ticket. Hopefully we will get something soon. That same ticket also has the ipmi for qa hosts for beaker in it. 18:58:43 <michel_slm> anyone here going to FOSS Asia the month after that? 18:59:01 <puiterwijk> tflink: I'd be glad to, just let me know when you arrive/have time 18:59:10 <tflink> also, the secarch folks are asking for blockerbugs instances. I'll know more about it after devconf but figured that I'd mention it here 18:59:34 <nirik> oh? seperate ones? 18:59:37 * dgilmore will be at fosdem and devconf 19:00:04 <tflink> nirik: at this point, it'd have to be. adding multiple arch support to the codebase isn't going to happen before f22 starts up 19:00:27 <smooge> michel_slm, I don't think so. 19:00:31 <nirik> tflink: ok 19:00:33 <puiterwijk> for those interrested, I'm giving a talk at saturday on centralized authn 19:00:34 <tflink> puiterwijk: I'll be in brno from feb 5-18 19:00:54 <puiterwijk> tflink: I from feb 5 to 9, so we should be able to find a time 19:01:27 <nirik> cool. well, if nothing else will close out the meeting in a minute or so 19:02:22 <nirik> thanks for coming everyone. 19:02:24 <nirik> #endmeeting