16:01:35 #startmeeting env and stacks (2013-12-17) 16:01:35 Meeting started Tue Dec 17 16:01:35 2013 UTC. The chair is mmaslano. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:01:35 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:01:45 #meetingname env and stacks 16:01:45 The meeting name has been set to 'env_and_stacks' 16:01:59 #chair abadger1999 pkovar tjanez samkottler bkabrda handsome_pirate hhorak juhp 16:01:59 Current chairs: abadger1999 bkabrda handsome_pirate hhorak juhp mmaslano pkovar samkottler tjanez 16:02:30 * samkottler waves & will have to leave in about 15 minutes for a doctor's appointment 16:03:01 * pkovar1 is here 16:03:05 who else is here 16:03:30 o/ 16:04:05 I'm here 16:05:28 * tjanez is being late 16:06:17 I'm here too 16:06:46 #topic init process 16:06:55 great, let's start 16:07:04 we don't have much time for something like PRD 16:07:17 Hi, sorry for being late.. 16:07:38 so I created something inspired by Cloud PRD https://fedoraproject.org/wiki/User:Mmaslano/Draft:Env_and_Stack_PRD 16:07:53 * pingou around (and late, sorry) 16:08:26 the real tasks are still little fuzzy, but I plan to work on them more if you agree with it 16:08:53 mattf: great you are here. You can tell us what you need 16:09:07 mattf is from Big data SIG 16:09:17 willb, would you start? 16:10:25 mmaslano, willb has been feeling most of the pain dealing w/ non-primary (c/c++,python,perl) language stacks in fedora 16:10:37 I think a lot of our headaches in Big Data relate to language packaging ecosystems vs Fedora 16:10:46 in particular, I have been working on Scala lately 16:12:29 in short, the main issues are widespread Ivy for dependency management (mizdebsk has been working on this) and very brittle versioning (there is the expectation that you'll be able to have particular versions of the runtime, compiler, and libraries to run any nontrivial application) 16:14:22 like a lot of language stacks (e.g. rubygems, maven, eggs), there is the assumption that any developer would just be using the language stack instead of downstream dependency management, so I've had to work around a lot of those sorts of issues 16:15:15 all those places where we have to work around things the language community does are hurdles to eventually automating / reducing maintenance package work 16:16:28 we were already speaking about automation of packaging (not ready) and hosting such packages in separated repositories 16:16:29 we also ran into this around node.js, which has its package/dep management tool available in fedora, but very little of the language ecosystem / base libraries available 16:17:04 what would be good solution for you? 16:17:18 SCL, or latest versions from upstream? 16:17:22 for us the application is where we want to focus our time, but we find ourselves in the toolchain management and not getting to the application 16:18:07 willb, mattf, I'm glad to hear your perspective w.r.t. Stacks. Problems like you describe are very aligned with the scope of our WG 16:18:30 the one dep i've been harping on lately is jetty. the latest from upstream doesn't support the java version that our app upstream wants to use. so having the latest in fedora is a problem. 16:18:59 jetty seems to touch a staggering number of projects in our SIG, too 16:19:54 mmaslano, something that would have helped (and probably will in the future) is a way to provide language ecosystems in fedora in a way that aligns w/ how the language itself is used 16:20:27 all the modern languages have their own dep & ver management tooling 16:20:37 and most of it doesn't align well w/ fedora 16:21:34 mmaslano, as for SCL, they could very likely help. however, they represent an add-on to fedora 16:22:04 almost like rpm fusion, or all the extra repos from early fedora version 16:22:09 so what was that about jetty? 16:22:15 in the scala case, Fedora would allow us to provide scala 2.9 and 2.10 packages in the same release, but to really work with scala, we'd need both of those resolvable via Ivy, and not just "scalac29" and "scalac210" binaries 16:22:30 mattf: yes, on last fesco meeting was decided, that additional repositories are acceptable, but content has to be reviewed anyway 16:23:53 sochotni, rsquared is the best person to talk about the jetty situation. the exec summary is the version of jetty in fedora does not work w/ java 6 and the hadoop upstream isn't ready to abandon java 6. so we end up maintaining a patch to hadoop for jetty 9 that will live purely in fedora for quite a while to come 16:25:02 sochotni, i believe there was some chatter about compat packages for jetty libs too. that might be a workaround for f21/22 or so, but ultimately there's a mismatch in expectations between fedora and languages like java / scala / js that should be resolved 16:25:26 (i should note, there's been a lot of good work on figuring it out for java) 16:25:29 mattf: we don't jave JDK 6 in Fedora 18+ 16:26:08 sochotni, i believe the compat lib discussion hinged on fedora policy and the compat library working w/ java 7 16:26:26 mattf: compat lib of what package? 16:26:27 of course that workaround might stop working if the only jetty code doesn't work w/ say java 8 or java 9 16:26:58 sochotni, we should chat w/ rsquared on the specifics 16:27:03 So, the question is, is WG willing to work towards integrating upstream language packaging systems 16:27:31 This could complement the existing RPM-based packaging 16:28:17 tjanez: good point 16:28:30 It would certainly benefit the needs of the Big Data SIG 16:28:43 this is probably too specific task 16:28:44 The Big Data SIG use case could also be put in the PRD 16:28:55 might be a good test case but that's probably it 16:29:03 i'd argue that steps in that direction will benefit the needs of groups that want to bring applications written in "new" languages to fedora 16:29:52 (where "new" == actually really old languages) 16:29:55 mmaslano, do you feel upstream packaging is in our scope or not? 16:30:13 tjanez: that's not something you will solve in a WG 16:30:14 tjanez: I thought we agreed it could be done automatically 16:30:22 you need someone who will *actually* work on the code 16:30:24 tjanez: at least for some languages 16:30:41 again with my specific case: right now, if you want to use Scala for real work on Fedora, your option is basically to install a bunch of binaries maintained by upstreams. It would be great if we were able to better support "new" language ecosystems wholly in Fedora. 16:30:43 all you can agree is "yes we want to do that" 16:31:08 #info Big Data specific case: right now, if you want to use Scala for real work on Fedora, your option is basically to install a bunch of binaries maintained by upstreams. It would be great if we were able to better support "new" language ecosystems wholly in Fedora. 16:31:26 nice 16:31:31 sochotni: well, I hope our WG will start working on things we put in the PRD 16:31:31 and by working I mean coding 16:31:51 still someone has to write automation for "new" language and maintain it 16:32:07 also I don't think packages without review can get into Fedora 16:32:12 legal issues.. 16:32:47 mmaslano: reviews could be done the first time and every-time something big changes in the package, but simple rebase could be done outomatically 16:33:07 sochotni: First I would like to see if we at least agree that complementing RPM packaging with upstream packaging is a viable way for the future 16:33:13 that's an interesting point. there's always going to be a human component to packaging. however, we could probably write down that small list and try to automate the rest. 16:33:25 tjanez: I agree 16:33:44 mmaslano: well, we would start with one language 16:33:52 so the packaging tool should be able to say "this rebase is suspiciously serious, we need a manual review" 16:34:13 mmaslano: but the common thing with all languages would be the concept of co-existance with RPM packages 16:34:28 now I'm confused 16:34:39 mattf: do you want to package upstream into rpm or not? 16:34:49 what I *could* imagine is some wrapper around pip/rubygems or alternative packaging systems 16:35:11 mmaslano: I think the tool should produce rpms in the end 16:35:18 mmaslano: +1 good question 16:35:25 but policy of updates...my head starts to hurt just thinking about it 16:35:37 mmaslano, i want to package upstream into something that fedora will happily include directly in its ecosystem 16:35:59 mattf: I'd like to see something like that too 16:36:20 in other words, the form (rpm, deb, npm, other) isn't as important to me 16:36:41 sochotni: even now many people do big updates that introduce new issues according to guidelines and only if someone notices it it gets resolved.. 16:36:48 #proposal add automatic packaging of upstreams into WG goals 16:36:59 hhorak, too true! 16:37:47 mattf, well from the consumer (aka big data user) point-of-view, the form is not important, from the distribution point-of-view it is 16:38:14 hhorak, there's little infrastructure that i see for CI around fedora. you're a lib that has 12+ users, well when you update you should maybe rebuild your deps to find breaks. 16:38:24 hhorak, fedora has a very reactive model atm 16:38:26 what I can't imagine is including a really new package without a review.. I mean at least initial review would have to be done manually, but with something like fedora-review it shouldn't be too hard 16:38:36 tjanez, that's fair 16:40:03 #proposal add automatic packaging of upstreams into WG goals. Initial review of packages will be neded. 16:40:03 mmaslano, I'm ok with that, I would just like to clarify a bit more what the final form of that packaging would look like 16:40:03 would it be Fedora-approved subset of upstream packaging repositories 16:40:03 and the users would use tools like pip 16:40:04 or would it be an rpm repository automatically generated with *2rpm tools 16:40:05 any votes? 16:40:07 and the users use dnf (distro tools) 16:40:18 mmaslano: +1 16:40:29 tjanez: I would stay with rpm, we alredy know how it works 16:40:33 (most of the time) 16:41:01 tjanez, to some extent, the existing processes are important for consumers as well as for the distro. For any reasonable list of advantages we can list for using Fedora, most of them come from being well-integrated with the Fedora ecosystem and from having packages that follow Fedora processes. 16:42:25 tjanez: I don't think we should provide any bits outside rpms, it would require to solve all the issues that rpm solves today again and again for every language stack 16:42:36 hhorak, +1 16:43:07 Ok, you convinced me 16:43:22 I just wanted to put it out so we discuss it 16:43:23 tjanez: what I can imagine is creating something that would work like pip, but would actually work with rpms in the background.. (not familiar to pip, so maybe it is not easy) 16:44:01 IMHO RPM/rubygems integration and xmvn are good examples of language-specific packaging systems working well in Fedora 16:44:15 hhorak: why? we have dnf/yum already. it's not obvious to me, why create another tool for installation 16:44:18 hhorak: +1 16:44:49 mmaslano: it would be just a conveniece wrapper for dnf/yum 16:45:10 * pingou doesn't see the point of wrapping wrapper 16:45:18 tjanez: I wouldn't say just in case of dnf 16:46:37 it seems to me Fedora users need some way how automatically generate packages from upstream. Implementation details can be solved later 16:47:02 well, the use case I can see is to attract developers with different backgrounds 16:47:03 they maybe very familiar with pip/rubygems/... 16:47:19 ok, fair enough 16:48:33 tjanez: yeah, but which tool would you pick? pip/rubygems? you would probably have to patch all these tools 16:49:12 mmaslano, I would change your proposal to something like: automatic packaging of upstream language repositories into rpm-based repositories. Initial review of distributed packages would still be needed 16:49:17 mmaslano: I understand that as users just don't want to use dnf, because the style of work is way different from their language-specific tools.. that's why they would like to use what they use in other distributions (language specific ways, pip for example).. but command name doesn't matter, it's more about style of work, so if dnf can be as easy to use as these language-specific tools, then we can stay with pure dnf.. Maybe I didn't ca 16:50:14 hhorak: we are missing end of sentence, not all irc clients can display so long messages ;-) 16:50:18 tjanez: ok +1 16:50:24 mmaslano: yes, you would have to patch all of them. Or rather, replace them with commands that have the same CLI 16:51:02 mmaslano: but yea, this is an implementation detail that can be added or not later 16:51:08 tjanez: at this moment it seems to me as far away future, because we don't even have the automatic packaging :) 16:51:10 mmaslano: sorry ;) my point was to try to understand why people don't like dnf/rpm/yum and prefer language specific tools 16:51:32 the advantage of distribution tools for language specific purposes is not that they can install $random_language_pack but that they work in isolated environments (like with python-virtualenv) 16:51:50 atleast that is how i see it 16:52:00 mmaslano: we do have a number of *spec tools :) 16:52:01 Subfusc: I guess in near future everything will be in container. So that's solved 16:52:40 #proposal Add to tasks/goals of our WG: Automatic packaging of upstream language repositories into rpm-based repositories. Initial review of distributed packages would still be needed 16:53:46 My proposal would actually just clarify the "The automatic packaging" task in mmaslano's PRD draft 16:53:49 tjanez: still +1 16:54:46 tjanez: +1 16:55:37 It would be great if we also add a short summary of what mattf and willb said earlier 16:55:39 Subfusc: thanks for that point. After quick reading it seems like virtualenv is kind of a python version of software collections, or do I understood it wrong? 16:56:17 Maybe as a "problem case" the WG is solving with that particular task 16:56:19 tjanez: do you want to sum it up? 16:56:40 Yes, I can and then put it in the wiki 16:57:11 tjanez, if you want some more detail, I wrote up some of the Scala-specific issues in August (http://chapeau.freevariable.com/2013/08/making-fedora-a-better-place-for-scala.html) and have been posting updates to the wiki here: https://fedoraproject.org/wiki/SIGs/bigdata/packaging/Scala 16:57:23 Maybe better if I do it after the meeting 16:57:42 willb, thanks, I will look at it 16:57:49 thanks 16:58:29 #action tjanez will sum up Big Data SIG needs 16:58:40 hhorak: I'm not that familiar with software collections, but it works in complete isolation with every virtualenv having its own libraries and it lives on $HOME so that non-admins can administer it. Software collections does not seem to completely envelop this concept 16:59:20 the install in home folder is the primary feature that makes virtualenv easy to use for developers 16:59:33 mmaslano: should "problem cases" be a new section or just a sub-bullet under automated packaging task 16:59:41 * mmaslano will need to leave in few minutes 16:59:56 tjanez: the whole task section is open for editing 17:00:04 Subfusc: right, user-specific installation is missing.. 17:00:06 I guess the rest is okay 17:00:22 tjanez: I copied topic which were discussed in last few weeks 17:00:36 Ok, are there any takers for writing up other tasks in the PRD? 17:00:59 if the task list gets longer in the future, tasks could be separated into two groups as discussed two weaks back, to primary and secondary (less priority) tasks 17:01:18 hhorak: its also easy to install and edit libraries without affecting anything else (which a system wide installation could not emulate) 17:01:20 We have to speed things up a bit 17:01:38 Jan 14th is coming very quickly 17:01:40 tjanez: I'll try to specify some. But it should be short definition up to 6 sentences 17:01:55 tjanez: yeah, that's the reason why I've started with prd without discussion 17:01:58 When will we have our next meeting? 17:02:06 tjanez: you mean what we discussed so far or some new tasks? 17:02:24 hhorak: the things from our discussions 17:02:30 tjanez: imho January 7 17:02:54 not much time, let's add as many tasks as possible now 17:03:01 and we can review it by email 17:03:04 I think we have to look through the IRC logs and extract the things already said 17:03:41 I tried to do it on my blog, but only from last 3 meetings 17:04:11 Ok, I'll look at your blog post 17:04:43 for the record http://mmaslano.livejournal.com/ 17:04:51 I'll certainly help, since I'll have more free time around the holidays 17:05:11 #action everyone will look at https://fedoraproject.org/wiki/User:Mmaslano/Draft:Env_and_Stack_PRD and think/work on tasks. 17:05:32 #info deadline for PRD is January 17th 17:05:44 mmaslano: ^date -d '2014-01-07' +"%V" says 02 so it should be at 13:00UTC 17:05:45 or 14th? 17:05:47 I hope others will join to create a better PrD 17:06:02 tjanez: hopefully 17:07:02 mmaslano: well it's actually Jan 13 (see: https://fedorahosted.org/fesco/ticket/1196) 17:07:10 #undo 17:07:10 Removing item from minutes: 17:07:16 #info deadline for PRD is January 14th 17:07:43 #info next meeting will be on 13:00UTC January 7th 17:07:50 tjanez: thanks 17:08:01 if you don't have any other topics, I would like to close the meeting 17:08:09 mmaslano, thanks for having us 17:08:24 mmaslano, you still put in the wrong date? 17:08:44 or has it been changed from Jan 13 to Jan 14? 17:09:15 thanks all 17:09:42 aaa 17:09:49 #undo 17:09:49 Removing item from minutes: 17:09:55 thank you all and enjoy Chrismas! 17:10:02 #undo 17:10:02 Removing item from minutes: 17:10:10 #info deadline for PRD is January 13th 17:10:17 #info next meeting will be on 13:00UTC January 7th 17:10:20 mmaslano: sorry for the trouble 17:10:35 I hope meeting log will be nice :) 17:10:42 let's go home 17:10:48 ciao 17:10:49 Thank you for a good meeting 17:10:52 #endmeeting