18:00:00 <nirik> #startmeeting Infrastructure (2015-06-18) 18:00:00 <zodbot> Meeting started Thu Jun 18 18:00:00 2015 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:01 <nirik> #meetingname infrastructure 18:00:01 <nirik> #topic aloha 18:00:01 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson 18:00:01 <nirik> #topic New folks introductions / Apprentice feedback 18:00:01 <zodbot> The meeting name has been set to 'infrastructure' 18:00:01 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:11 * threebean 18:00:12 <dgilmore> hola 18:00:13 <puiterwijk> hello 18:00:16 <nirik> any new people around or apprentices with questions or comments? 18:00:19 <cydrobolt> hello! 18:00:23 * sborza waves hello 18:00:25 * oddshocks pops in 18:00:26 * relrod here 18:00:54 * Corey84 is apprentice but not any questions coming to moind 18:01:07 <nirik> #topic GSoC student update - kushal 18:01:18 <nirik> any GSoC students want to give a update on their project? 18:01:20 <kushal> Who all are here? 18:01:26 * sonalkr132 is here 18:01:26 <kushal> nirik, thanks :) 18:01:37 <nirik> kushal: no problem. :) 18:01:53 <Corey84> been active with the tox stuff mostly on github tho (not student tho) 18:01:57 <AnuradhaW> Hi, I'm here. 18:02:10 <kushal> sonalkr132, AnuradhaW your updates please. 18:02:29 <sonalkr132> @rahulrrixe and @sshagarwal was here a moment ago 18:02:35 <prth> i have completed the basic implementation of dynamically created multi cropping overlays. 18:02:39 <AnuradhaW> I have worked on getting a better layout for Q/A page of askfedora. 18:02:43 <sshagarwal> I am still here 18:02:43 <prth> next, I'll be polish the ui 18:02:53 <rahulrrixe> Kushal: Written a blog for this week update https://medium.com/@rahulrrixe/becoming-git-pro-by-getting-into-under-the-hood-417054b3f4aa 18:03:15 <AnuradhaW> And the mentors have pointed out some issues with the detailed coding of the page and I'm currently fixing those. 18:03:22 <kushal> Okay 18:03:46 <sonalkr132> As of I was working on setting up git server for GG 18:04:08 <sonalkr132> Only last part of puzzle is left, ie integration of sparkleshare 18:04:18 <sonalkr132> #link https://chasingcrazydreams.wordpress.com/2015/06/18/awesomeness-of-rack/ 18:04:26 <prth> s/polish/polishing 18:04:27 <rahulrrixe> I have completed all the backend code for the git integeration and working on UI. I think first milestone will be complete by today as I have to fix some minor bugs in pull-request and merging code. 18:04:45 * danofsatx is back 18:04:53 <kushal> We will talk to your mentors for the midterm next week. 18:05:06 <Corey84> rahulrrixe, nioce blog 18:05:14 <kushal> I will also count the presence here. 18:05:22 <sshagarwal> We needed a protocol to demonstrate the Skinkglue way of adding protocols to the client using addon approach. So, I am in the process of writing the protocol to display folder emails (email files placed locally in a folder). 18:05:33 <kushal> nirik, you can continue I guess. 18:05:38 <rahulrrixe> Corey84: Thanks :) 18:05:51 <nirik> kushal: thanks. 18:06:07 <nirik> on to announcements/info 18:06:11 <nirik> #topic announcements and information 18:06:11 <nirik> #info Got the runroot plugin working in our production koji - threebean / nirik 18:06:11 <nirik> #info arm03-packager and arm03-qa instances announced and reinstalled with f22 - nirik 18:06:11 <nirik> #info production koji store mounted ro on koji01.stg - nirik 18:06:11 <nirik> #info grew storage for secondary arch kojis - nirik 18:06:12 <sshagarwal> kushal: hi --^ 18:06:12 <nirik> #info fixed fed-cloud09 playbook to be idempotent for check/diff at least - nirik 18:06:13 <nirik> #info initial people01 setup in ansible/rhel7. Still needs work - nirik 18:06:15 <nirik> #info mote pushed to production at https://meetbot.fedoraproject.org - still working out some kinks - cydrobolt / threebean 18:06:18 <nirik> #info next trip to PHX2 will be July 27->July 30th. Hardware to be installed needs to be there by the 27th. 18:06:23 <nirik> any further discussion on those or things to add? 18:06:47 * roshi is here 18:07:00 <nirik> on to a discussion item added by mdomsch... 18:07:03 <nirik> #topic - mdomsch - Retire MM 1.4.4 from Fedora and EPEL repos 18:07:05 * roshi wonders why 1800 UTC always sneaks up on him... 18:07:15 <nirik> from the gobby doc: 18:07:18 <nirik> Not sure I can attend, but I see no reason to keep mirrormanager 1.4.4 in the Fedora 23 18:07:18 <nirik> repositories. We have a couple choices: 18:07:18 <nirik> 1. transition owner to someone and let them put mirrormanager 2.x into the repos 18:07:18 <nirik> 2. orphan and delete MM 1.4.x from FC23, let the releases in FC21-22 remain as-is 18:07:19 <nirik> Happy to do whichever the team desires, and if someone with pkgdb admin rights wants 18:07:20 <nirik> to take care of it too, don't wait on me. Thanks! -mdomsch 18:07:21 <nirik> (P.S. it's in EPEL repos too, follow the same logic). 18:07:34 <nirik> IMHO, I'd like to see us take over the package and update it to mirrormanager2. 18:07:52 <puiterwijk> yeah, agreed 18:07:54 <nirik> pingou / adrian: any thoughts on that idea? 18:08:17 <threebean> I think pingou is away.. 18:08:25 <nirik> yeah... 18:08:30 <Corey84> agree with taking it in 18:09:18 <threebean> I dunno if it would be easier or harder to do it with a separate package and provides/obsoletes? 18:09:29 <threebean> or, if that makes a difference. 18:09:31 <nirik> yeah, not sure. Would need a re-review... 18:10:34 <threebean> eh, maybe table this for next week so pingou and adrian can comment. 18:10:38 <nirik> ok, I'll try and move this forward someway. yeah. 18:10:45 <puiterwijk> it doesn't matter either way. Except that with a new package, we'd be more clear that it's a rewrite 18:10:49 <Corey84> obsoletes would kinda make senses (to me at least) 18:11:07 <nirik> puiterwijk: well, yeah, but 1.x vs 2.x might say that too 18:11:24 <Corey84> not everyone checks that tho sadly 18:11:30 <puiterwijk> nirik: fair enough. But anyone else runniung it and just doing "yum update" might except it to Just Work(R) 18:11:40 <Corey84> that too^ 18:12:02 <Corey84> would need some hefty docs i'd think to go that route 18:12:03 <nirik> puiterwijk: sure, we could also just update rawhide 18:12:10 <nirik> and epel7 (which doesn't yet exist) 18:12:25 <puiterwijk> fair enough 18:12:34 <nirik> then f23+ will get it. 18:12:44 <puiterwijk> people won't expect to be able to just update from F22->F23 without any work, I guess :) 18:12:55 <nirik> but then we will need to keep epel6 on the old one... 18:13:04 <nirik> but I doub't there's too many users. 18:14:27 <kushal> nirik, I have an open question for the open floor. 18:14:31 <nirik> ok, anything more to discuss, or shall we move on to letting threebean teach us about fedmsg 18:14:43 <smooge> well if it is side packages mirrormanager1x mirrormanager2x , could both be in EPEL? 18:14:45 <nirik> kushal: ok, can wait for that, or you want to do it now? 18:14:48 <Corey84> why would that requires epel6 still what am i missing 18:14:58 <kushal> nirik, I can wait :) 18:14:59 <nirik> smooge: they could, then we would need to maintain them both f 18:15:21 <nirik> Corey84: epel6 has mirrormanager 1 in it. Upgrading it to mirrormanager2 is not allowed in the epel guidelines. 18:15:49 <Corey84> ah im not following the epel as closely apparently 18:16:06 <nirik> #topic Learn about: fedmsg - threebean 18:16:13 <nirik> threebean: the floor is yours. :) 18:16:16 <threebean> cool :) 18:16:36 <threebean> so, if there's questions about stuff as I go, just ask. 18:16:48 <threebean> i'm not sure what corners people are more interested in, so.. I'll just overview. 18:16:59 <threebean> http://fedmsg.com/ 18:17:20 <threebean> so, fedmsg is our in-house message bus that connects most of the 40-some services we run. 18:17:32 <threebean> it's built on top of zeromq for the underlying message passing. 18:18:05 <threebean> it's a pub/sub style system.. so it can't do things like request-reply or other more special-purpose patterns. 18:18:22 <Corey84> would it need to tho? 18:18:30 <threebean> Corey84: not yet it hasn't. 18:19:03 <threebean> one of the design guidelines going in was to make it as 'resilient' as possible, so that if some or any of the pieces go down, the others can function as gracefully as possible. 18:19:35 <threebean> this means that, if you stand up one of our fedmsg-enabled webapps on your laptop, it can run and publish fedmsg's without having to setup any other services. 18:19:55 <threebean> but.. this has no effect, because nothing else is configured to listen to that lonely service running on the laptop. 18:20:36 <threebean> we don't have a central broker, like other messaging systems do. but we do have a variety of optional services that can play portions of the role played by a central broker. 18:20:36 <Corey84> so each services while connected can be standalone? 18:20:38 <rahulrrixe> threebean: Is there any available software architecture diagram of fedmsg? I am interested in learning its underlying components. 18:20:43 <threebean> Corey84: right, right. 18:21:16 <threebean> rahulrrixe: I think this might be the best that we have -> http://www.fedmsg.com/en/latest/topology/ 18:21:19 <roshi> what services aren't currently on the bus and is there a roadmap to get them on the bus? 18:21:24 <Corey84> sounds alot like how tox is architected non central and standalone (or even popcorntime) 18:21:46 <threebean> rahulrrixe: but there are many more services now that aren't in that diagram.. it would just get too big. 18:22:07 <Corey84> threebean, this is the same fedmsg in git2fedmsg yes? 18:22:09 <threebean> roshi: there's this doc for the status of different apps -> http://www.fedmsg.com/en/latest/status/ 18:22:14 <Corey84> github* 18:22:25 <threebean> Corey84: yes. :) 18:22:42 <Corey84> g2k thats what side i use most not the webapp 18:22:46 <threebean> roshi: but it needs a little love. for instance, we need to add 'zanata' to it. 18:23:08 <rahulrrixe> threebean: It suffiecient for now. BTW thanks :) 18:23:12 <roshi> is querying datanommer the best means to gather historical data from the bus? 18:23:17 <Corey84> zanata is the reminder stuff ? 18:23:28 <threebean> Corey84: it's a translation platform at fedora.zanata.org 18:23:39 <Corey84> ah 18:23:52 <threebean> the *big* missing piece from the bus right now is bugzilla content. I can only assure you that we've been working on it, it's slow going, and unfortunately there's no ETA yet. 18:24:00 <threebean> it would be pretty valuable information to have in the stream. 18:24:11 <threebean> the best way to query for history stuff, yes, is (technically) datagrepper 18:24:18 <threebean> https://apps.fedoraproject.org/datagrepper/ 18:24:38 <threebean> if you have an app that lives inside fedora infrastructure, you can circumvent that webapp and query the datanommer postgres db directly 18:24:43 <Corey84> what are the biggest snags on bz implementation then 18:24:49 <threebean> which would be slightly faster than having to setup and teardown HTTP requests to datagrepper 18:25:34 <threebean> Corey84: just stuff on the RH bz side. they need to publish messages to many other communities besides fedora (like jboss) and they're working on how best to do that in a way that makes everyone happy. 18:25:37 <roshi> is there a batch archive of the database we can populate something local with? So we don't have to send queries over the network for testing? 18:25:39 <Corey84> brb, coffee refill 18:25:42 <threebean> roshi: yes 18:25:47 <roshi> awesome 18:25:55 <threebean> http://infrastructure.fedoraproject.org/infra/db-dumps/ 18:26:01 <threebean> roshi: that snapshot is updated nightly 18:26:06 <roshi> awesome 18:26:31 <threebean> so, we have our Fedora Infrastructure deployment and there are others at debian and data.gouv.fr. 18:26:53 <roshi> wow, 5 gigs of data in it already... 18:26:59 <threebean> roshi: that's compressed, too ;) 18:27:04 <roshi> haha 18:27:19 <threebean> you should be able to get your local box subscribed to both the Fedora infra and the debian buses by uncommenting a line in your local /etc/fedmsg.d/endpoints.py file. 18:27:32 <threebean> release-monitoring.org and pagure.io, although they are run by us, have "their own" buses. 18:27:42 <threebean> but we loop their feeds back into the Fedora bus to make things convenient. 18:27:44 * Corey84 back 18:27:55 <threebean> also -- centos infra started discussing adopting fedmsg in the last few months. 18:28:02 <threebean> In the news, there was a talk on fedmsg data at the Spark Summit just recently: 18:28:04 <threebean> http://chapeau.freevariable.com/2015/06/summit-fedmsg.html 18:28:10 * nirik notes #fedora-fedmsg has a (mostly) live stream of our fedmsgs (with some of the noiser ones filtered). 18:28:21 * threebean nods 18:28:38 <threebean> roshi: if you're looking at trawling through all that data, it could be handy to have the list of topics and example messages from the docs which can be found here: 18:28:40 <threebean> http://fedora-fedmsg.readthedocs.org/en/latest/topics.html 18:29:07 <roshi> sure 18:29:10 <threebean> for kicks, we have a visualization of some of the earlier activity from the bus (2012) 18:29:13 <threebean> http://threebean.org/so-i-turned-the-fedmsg-data-into-a-git-log-and.webm 18:29:15 * Corey84 adds to channels 18:29:39 <threebean> you should be able to generate your own modern version of that with the script in this repo 18:29:41 <threebean> https://github.com/ralphbean/fedmsg2gource 18:30:04 <nirik> might be cool to do one for this week with the mass rebuild going 18:30:12 <nirik> if someone wants a fun project 18:30:16 <threebean> agreed :) 18:30:42 <Corey84> no time here but would be interested 18:31:05 <threebean> so, if you're interested in how fedmsg is deployed in our infra, the majority of the config is held here: 18:31:07 <threebean> http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/fedmsg 18:31:47 <threebean> so, in a system like this, services have to know about each other. in a broker-centered setup, the broker manages introducing everybody to everybody... 18:32:01 <threebean> .. so, as we've grown over time, our configs have gotten a little hairy 18:32:09 <threebean> http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/fedmsg/base/templates 18:32:20 <threebean> in particular, the "endpoints" files are a lot to wade through. 18:32:28 <Corey84> the intent is to kick the broker then? 18:32:44 <threebean> Corey84: right. no broker to crash and then bring down the whole infrastructure with it. 18:33:04 <threebean> one of the things I've been working on recently, is trying to automatically generate our fedmsg configs from our ansible inventory definition 18:33:15 <Corey84> makes sense BUT how would they stay synced manual sync on connecting? 18:33:33 <Corey84> assuming such a crash 18:33:46 <threebean> Corey84: oh, I guess I don't understand the question. can you rephrase? 18:34:43 <lmacken> the hubs have the ability to query datanommer to see if they missed anything. I think that is what you are asking? 18:34:54 <Corey84> assuming a crash (one that in broker sys would crap out everything in the new system what sync / re-sync mechanism is there) 18:35:07 <Corey84> indeed thanks lmacken 18:35:11 <threebean> ah, yeah. that's it. :) 18:35:59 <Corey84> so the hub would essentially call home to the voiceamil machine to check for missed alerts? 18:36:10 * threebean nods 18:36:14 <Corey84> ah cool 18:36:36 <threebean> yeah - in our case, the badges.fedoraproject.org awarder daemon checks for its last message when it restarts and then asks datanommer for all the messages between then and now. 18:36:46 <threebean> so it can award badges to people it missed while it was down. 18:36:57 <Corey84> seems it would allow for offline updating too and fetching /pushing new stuff in that fashion 18:37:23 * threebean nods 18:37:56 <threebean> as for new things being built on top of fedmsg, the #fedora-apps team is working on at least two 18:38:32 <Corey84> so bootstrapping in essentially for new stuff 18:38:35 <threebean> there's a new statscache daemon we've been working on that should allow quicker access to statistics that you'd have to compute yourself from big datagrepper qureies 18:38:37 <threebean> http://threebean.org/blog/statscache/ 18:39:06 <threebean> and there's the new fedora-hubs project that people are collaborating on over in #fedora-hubs 18:39:38 <threebean> it has a smart-cache mechanism driven that will be driven by fedmsg 18:39:39 <lmacken> there's also tool called fedmsg-notify that lets you get fedmsg desktop notifications ;) http://lewk.org/blog/fedmsg-notify 18:39:39 <threebean> https://pagure.io/fedora-hubs/blob/develop/f/README.rst#_131 18:39:48 <threebean> http://fedoraproject.org/wiki/Fedora_Hubs 18:39:57 <threebean> oh yeah - the desktop notifications are amazing :p 18:40:06 <threebean> lmacken++ 18:40:19 <usernkey> Hi guys, This is my first meeting. I introduced myself a couple of weeks ago in the mailing list and I would like to know if I can get the apprentice access so I can start exploring the infrastructure. 18:41:15 <nirik> usernkey: welcome. Sure we can get you added... 18:41:23 <jcvicelli> Me 2 18:41:52 <nirik> see me in #fedora-admin after the meeting. 18:42:00 * nirik goes back to listening to threebean on fedmsg 18:42:05 <usernkey> thx 18:42:16 <threebean> I don't really have anything else prepped. 18:42:20 <threebean> if anyone has other questions? 18:42:25 <Corey84> +1 fedmsg-notify sweet option I use it 18:42:29 <threebean> dreams? plans? 18:42:53 <nirik> now that we have a pretty sold fedmsg we should try and go back and replace things we have now that are cron jobs. 18:42:58 <nirik> see if we can reach 0 cron job 18:43:10 <nirik> (or very low anyhow) 18:43:17 <threebean> would be cool, yeah. 18:43:41 <nirik> one thing we talked about ages and ages ago... was allowing users to submit messages... 18:43:44 <threebean> as we think of them, filing tickets is the best way. 18:43:53 <nirik> is that still on the roadmap? or no longer? 18:44:11 <threebean> eh, no longer. but only because the use-case isn't clear, at least to me. 18:44:46 <threebean> if we have a solid use case, I think we could do it with a HTTP -> fedmsg bridge service, kind of like how github2fedmsg works. 18:44:57 <nirik> well, the case I guess is trying to get users more involved. So, if we have a fedmsg from some user that they run foo all the time or update foo in testing we can offer them foo related stuff... 18:45:09 <nirik> but yeah, needs a lot more thought. 18:45:32 <nirik> and probibly talking to the workstation folks (at least I think it's more a workstation thing than a server or cloud) 18:45:47 <threebean> yeah - the way we kind of do that now is that they 1) update foo in testing and then 2) provide karma feedback in bodhi and *then* they get a fedmsg message published and a badge and hooray! 18:46:20 <nirik> yeah. 18:46:30 <nirik> perhaps that was a poor example. :) 18:46:33 <threebean> nirik: who should I ping in workstation land? any suggestions? :p 18:47:16 <nirik> I guess it would be more: 1) user uses foo a lot (launches, windows? dunno), 2) we have a updates-testing one, 3) we offer to let them test it if they want, 4) they do and provide karma 18:47:22 <randomuser> oh, if you're taking http->fedmsg questions, is zanata messaging getting any traction? 18:47:50 <nirik> threebean: not sure. I guess I can try and come up with a more articulate proposal and run it by you and then them? 18:47:55 <threebean> randomuser: yeah! we have 1/2 of it solved. 18:48:01 <randomuser> nice 18:48:05 <threebean> nirik: sounds good :) 18:48:38 <threebean> randomuser: at our request, they added a webhooks interface (like github has) and it works on a local test with https://github.com/fedora-infra/zanata2fedmsg 18:48:58 <threebean> but.. they forgot to cryptographically sign their messages. so, we can't stand it up because it would open our bus to abuse. 18:49:04 <randomuser> heh 18:49:07 <threebean> so, they're adding that feature.. and then we can play. 18:49:23 <randomuser> awesome. big news for translators, nice work! 18:49:31 <threebean> woot woot 18:49:49 <Corey84> eta on the stand up? 18:49:50 <puiterwijk> threebean: can you tell me with whom you're in contact? I've also got in contact with some of the Zanata devs recently 18:50:08 * threebean finds the bz ticket 18:51:20 * threebean waits and waits on bz search 18:51:36 <threebean> https://bugzilla.redhat.com/show_bug.cgi?id=1213630 18:52:10 <threebean> we're just waiting on that.. then we could deploy, I dunno, a month afterwards. just need to tweak our bridge software for their changes, setup a node, etc.. 18:52:26 <nirik> cool 18:52:46 <puiterwijk> threebean: cool, thanks for the link 18:53:00 <nirik> ok, anything else? shall we go to open floor? 18:53:16 <nirik> thanks much threebean! 18:53:32 <nirik> #topic Open Floor 18:53:38 <nirik> anyone have any items for open floor? 18:53:42 <kdas_> nirik, I am looking at https://fedoraproject.org/wiki/Changes/Two_Week_Atomic 18:54:00 <lmacken> threebean++ 18:54:01 * nirik nods. 18:54:22 <kdas_> nirik, to have a full test system to test the atomic/cloud images, we will need a baremetal box to run the tool. 18:54:33 * nirik sighs. 18:54:46 <kdas_> nirik, just wanted to inform that, and get your thoughts about how to skip that :) 18:54:54 <nirik> well, thats a major pain. 18:54:57 <Corey84> much appreciated threebean 18:55:04 <Corey84> very informative 18:55:05 <nirik> we don't have random idle hardware sitting around doing nothing. ;) 18:55:13 <nirik> and asking for such out of the budget cycle is not fun 18:55:14 <kdas_> nirik, basically I will have to run qemu-kvm to boot and run some images. 18:55:32 <kdas_> nirik, I know, that is why I am asking how to go ahead with this. 18:55:41 <nirik> kdas_: can it be done as part of a koji plugin? 18:56:06 <kdas_> nirik, can be, but I really do not want take down koji accidentally 18:56:34 <nirik> sure. I guess open a discussion about it on the releng list? 18:56:35 <kdas_> * want to :) 18:56:39 <kdas_> nirik, Yup. 18:56:44 <kdas_> nirik, thanks for the input. 18:56:46 <nirik> and perhaps folks there can suggest the best way forward 18:56:59 <Corey84> opening in releng sounds good 18:57:02 <nirik> it might be possible to get a machine, but it will be painfull I am sure. 18:57:06 <kdas_> nirik, also do you if there is any nested virt which is good enough, 18:57:16 <kdas_> * do you know 18:57:24 <nirik> last I tried nested virt it was completely unusable. ;( 18:57:33 <Corey84> ^^ same 18:57:40 <nirik> but it was a while back. ;) 18:57:43 <Corey84> but that was > 6 months ago 18:57:48 <Corey84> may be better now 18:58:26 <nirik> ok, anything else for open floor? 18:58:31 <nirik> thanks for bringing it up kdas_ 18:59:00 <puiterwijk> kdas_: I've heard it might work reasonable nowadays 18:59:07 * randomuser is wrapping up an RFR ticket right now 18:59:08 <puiterwijk> at least in some situations 18:59:29 <dgilmore> nirik: kdas_: we should be able to use nested virt to do automated testing 18:59:32 <puiterwijk> kdas_: I can forward you to the person that told me that after the meeting, he might have ideas 18:59:47 <nirik> well, we can give it a go... 19:00:06 <dgilmore> its apparenly improving rapidly 19:00:27 <dgilmore> kdas_: we can talk about it in person next week 19:01:16 <nirik> ok, thanks for coming everyone. 19:01:18 <nirik> #endmeeting