13:01:52 <hhorak> #startmeeting Env and Stacks (2014-09-16)
13:01:52 <zodbot> Meeting started Tue Sep 16 13:01:52 2014 UTC.  The chair is hhorak. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:01:52 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
13:02:17 <hhorak> #chair pkovar tjanez samkottler bkabrda hhorak juhp mmaslano vpavlin sicampbell
13:02:17 <zodbot> Current chairs: bkabrda hhorak juhp mmaslano pkovar samkottler sicampbell tjanez vpavlin
13:02:36 <juhp_> hi
13:02:53 <hhorak> #topic init process
13:03:03 <hhorak> Hi!
13:03:18 <bkabrda> hey!
13:03:29 <mmaslano> hi
13:04:30 <hhorak> Regarding the ML discussion so far, I expect we'll have heated discussion, so let's start with it.
13:04:55 <hhorak> unless somebody wants to add something to the topics list..
13:05:06 <bkabrda> hhorak: i have one minor thing
13:05:30 <hhorak> bkabrda: what's that?
13:05:43 <bkabrda> just before this meeting, I was ask by developers of Koschei, if we, as a group, could promote Koschei and recommend it as a good tool for all Fedora packagers
13:06:50 <hhorak> bkabrda: technically, I see it as a good idea, but do we have releng's POV if it does not break koji? (I mean so many scratch builds...)
13:07:21 <juhp_> CI?
13:07:24 <bkabrda> hhorak: it does not, it's already been running for some time. it only runs scratch builds if overall Koji load is below a certain level
13:07:45 <mmaslano> juhp_: sort of CI, yes
13:08:13 <bkabrda> juhp_: it runs scratch builds of packages when their dependencies are updated, basically
13:08:23 <hhorak> #idea developers of Koschei would like to see Koschei promoted and recommend as a good tool for all Fedora packagers
13:08:45 <juhp_> thanks
13:09:06 <hhorak> #topic running Koschei for all Fedora packages (promote and recommend it)
13:09:08 <ncoghlan> sorry I was late folks - flaky internet :(
13:09:16 <bkabrda> #link http://koschei.cloud.fedoraproject.org/
13:09:23 <bkabrda> #link https://github.com/msimacek/koschei
13:09:52 <hhorak> ncoghlan: hi, you're not late, we haven't started with repositories yet :)
13:10:25 <bkabrda> hhorak: I don't think we should run it for all. we should just promote it so that packagers themselves suggest important packages to run there
13:10:53 <bkabrda> the problem is that if too many packages are there, they'll be rebuilt in very long intervals not to overload koji
13:11:04 <juhp_> currently it is just run for http://koschei.cloud.fedoraproject.org/groups ?
13:11:06 <bkabrda> so we should really stick to "the important"
13:11:14 <mmaslano> bkabrda: so do we need more machines?
13:11:16 <bkabrda> juhp_: afaik, yes
13:11:45 <bkabrda> mmaslano: not right now, but if we were to add many more packages, then likely yes
13:12:05 <juhp_> let's add glibc ;oP  just kidding
13:12:15 <hhorak> #info Koschei only runs scratch builds if overall Koji load is below a certain level, but too many packages would mean they'll be rebuilt in very long intervals not to overload koji
13:12:22 <bkabrda> I guess it should work in a way "we recommend it, packagers start using it, we find out how much HW we need)
13:12:52 <juhp_> how do people opt in currently?
13:13:00 <ncoghlan> having fought those kinds of battles a fair bit, good metrics can be powerful for making the case for more machines
13:13:01 <bkabrda> also, I think it's Koji that would need the new HW, not koschei server itself :)
13:13:24 <ncoghlan> bkabrda: right
13:13:59 <bkabrda> juhp_: right now, I think you need to write to msimacek
13:14:07 <bkabrda> (at redhat.com)
13:14:34 <bkabrda> but AFAIK there are plans to create a dynamic web UI, where packagers would be able to add/remove packages themselves
13:14:37 * kushal is here too
13:14:49 <juhp_> bkabrda, okay - or pull request? :)  I see
13:14:57 <hhorak> bkabrda: that sounds fine to me. Also manually acking packages seems fine to me, so we don't get into troubles in the beginning..
13:15:06 <juhp_> sure
13:15:15 <bkabrda> juhp_: yeah, I guess. I'm not *that* familiar with koschei
13:15:44 <bkabrda> hhorak: yes, for the time being. I agree
13:15:52 <hhorak> #info there are plans to create a dynamic web UI, where packagers would be able to add/remove packages themselves
13:16:15 <juhp_> maybe I should mention it to the Workstation WG - sounds like it could also be useful for gnome etc - though they have their own CI too upstream I think
13:16:44 <bkabrda> so my proposal is that people from our group should find out more about koschei; if everyone likes it, we can vote on some sort of formal recommendation next time
13:16:56 <bkabrda> sounds good?
13:17:22 <juhp_> sure +1  sounds like a useful tool
13:17:25 <hhorak> bkabrda: yeah, making an AI for everybody..
13:17:44 <hhorak> #action everbody should find out more about koschei; if everyone likes it, we can vote on some sort of formal recommendation next time
13:17:57 <bkabrda> thanks :)
13:18:24 <juhp_> I don't think I/we can use it for Haskell packages though because ghc is too strict about version deps
13:19:04 <bkabrda> juhp_: yeah, it really depends on the respective SIGs/packagers
13:19:06 <juhp_> (as we have to rebuild anyway)
13:19:14 <juhp_> sure
13:19:28 <hhorak> juhp_: if we identify packages not suitable, we should track them..
13:19:35 <juhp_> right
13:19:45 <hhorak> anyway, ready for moving to the next topic?
13:20:04 <bkabrda> hhorak: yep
13:20:56 <hhorak> #topic Language specific mirrors for Fedora Playground compliant packages
13:20:56 <hhorak> #info the idea was a bit discussed on ML already: https://lists.fedoraproject.org/pipermail/env-and-stacks/2014-September/000507.html
13:21:32 <ncoghlan> OK, I'll start with a bit of context
13:21:46 <hhorak> ncoghlan: sure, go on
13:22:46 <ncoghlan> for several of Red Hat's internal tools, we currently deploy fully via RPM
13:22:46 <ncoghlan> for all the components of the web services
13:22:48 <ncoghlan> with more deployment options (like OpenShift) becoming available, we're reevaluating whether that makes sense
13:22:57 <ncoghlan> and for the projects where we run the one-and-only instance in the world, it really doesn't
13:23:21 <ncoghlan> we'd like to move those to a more software-as-a-service type deployment model
13:23:51 <ncoghlan> and in those cases, because it's the dev team responsibile for updates, the RPM format doesn't add any benefit at the application layer
13:24:06 <ncoghlan> what *is* still useful though is Fedora's review processes
13:24:19 <ncoghlan> when we go direct to upstream, we bypass all that
13:25:17 <hhorak> #info fully deployment in RPM for everything does not make sense in projects run one-and-only instance in the world
13:25:20 <ncoghlan> which would mean repeating the work
13:25:21 <ncoghlan> it would also mean when we review new packages, Fedora wouldn't benefit
13:25:23 <ncoghlan> hence the idea of being able to have a mirror where we could do the initial Fedora Playground checks
13:25:52 <juhp_> so you are after curated repos?
13:25:56 <ncoghlan> *without* the next step of setting up a COPR for the package
13:26:00 <ncoghlan> yeah
13:26:09 <ncoghlan> because we're going to have to do that work anyway
13:26:19 <ncoghlan> and I think it makes more sense to offer to do it upstream
13:26:26 <ncoghlan> rather than inside the firewall
13:27:08 <ncoghlan> we could seed the repos by flagging the packages that have already been accepted into Fedora
13:27:58 <ncoghlan> I'm confident we can make it work for at least Python, since devpi is well set up to handle a use case like this
13:28:28 <ncoghlan> #link http://doc.devpi.net/latest/
13:29:17 <ncoghlan> I'm less sure of the mechanics when it comes to other languages
13:29:21 <bkabrda> I have to admit that I've been playing with this idea for few weeks myself and I think there may be other benefits to it... and I agree that for Python, devpi is the way to go with this
13:29:38 <mmaslano> I would be fine with trying it at least for languages, which make sense
13:29:51 <mmaslano> if ncoghlan and bkabrda are saying it might work for Python, then let's try it
13:30:00 <ncoghlan> the other benefit I liked is a more incremental review path for new packages
13:30:15 <mmaslano> I'm not sure about all languages, they have their specifics and not so linux based upstreams
13:30:16 <ncoghlan> language mirror -> Fedora Playground -> Fedora
13:30:39 <juhp_> Haskell has something similar to this too called Stackage (a subset of the main Hackage package source repo)
13:30:53 <juhp_> bkabrda, right - I think licensing is just a small component
13:32:03 <juhp_> consistent dependencies of package sets etc
13:32:17 <bkabrda> so other benefit could be, that we would offer prebuilt binaries (the new Python packaging format called wheel) of upstream packages that have C extensions (and always fail installation from PyPI, since one needs gcc and *-devel packages)
13:32:34 <ncoghlan> making our devs happier about not having to learn RPM is a bonus on my end, but not one I'd expect to matter so much to the WG :)
13:33:24 <juhp_> is there an rpm packaging tool for python packages?
13:33:29 <ncoghlan> bkabrda: yes, that's on my list of benefits too
13:33:30 <ncoghlan> pretty sure it's a Python specific one, though
13:33:34 <hhorak> #info the idea is to be able to have a mirror of upstream repositories where we could do the initial Fedora Playground checks
13:33:34 <hhorak> #info for python devpi seems to be the way to go with this
13:33:34 <hhorak> #info we could seed the repos by flagging the packages that have already been accepted into Fedora, the incremental review model could look like language mirror -> Fedora Playground -> Fedora
13:33:35 <hhorak> #info other languages may have their specifics and not so linux based upstreams
13:33:56 <ncoghlan> juhp_: bkabrda's pyp2rpm
13:34:03 <juhp_> cool
13:34:13 <bkabrda> ncoghlan: yes, but I'm assuming that every language/ecosystem will just have different set of advantages
13:34:22 <ncoghlan> so in theory the devpi -> COPR could be automated as well
13:34:33 <juhp_> ncoghlan, sure
13:34:44 <bkabrda> ncoghlan: in theory and with some hints for pyp2rpm, yes :)
13:34:44 <ncoghlan> bkabrda: yeah
13:35:36 <ncoghlan> bkabrda: working on it! One day we'll get to the end of the metadata 2.0 saga and actually have a usable extension system :)
13:36:16 <hhorak> there is always problem with bundling in upstream repos, what would happen in this regard in the Fedora's repo? would we care or we would deliberately not care?
13:36:41 <ncoghlan> for the mirror & Playground, I'd say we wouldn't worry about it
13:36:54 <bkabrda> hhorak: deliberately not care, I think. we don't care in COPR/Playground, either
13:37:08 <ncoghlan> fixing those kinds of issues is where the policy compliance step in getting into Fedora proper comes in
13:38:14 <ncoghlan> this is aimed at teams like mine where we're happy to handle the maintenance implications
13:38:25 <hhorak> #info un-bundling and similar maintaining work would be solved in the step 'playground -> Fedora'
13:38:42 <mmaslano> ncoghlan: can you solve problems with depedencies outside of python modules? Let's say your module depends on some system library in exact version
13:38:44 <ncoghlan> rather than those where sysadmins are trying to deploy apps without the aid of a dedicated developer team
13:38:57 <mmaslano> ncoghlan: many languages don't specify such data in their metadata
13:39:10 <ncoghlan> mmaslano: that's where the wheel binary format comes in
13:39:30 <mmaslano> ncoghlan: magically solved, I understand :)
13:39:46 <ncoghlan> mmaslano: and some neat tricks in devpi that let us do the binaries on a Fedora/EPEL basis
13:40:09 <ncoghlan> while having a shared mirror for the source packages
13:40:36 <ncoghlan> longer term, we want to solve the external deps problem, but that's rather complicated :)
13:41:00 <bkabrda> ncoghlan: I'd even say generally not solvable, but we can talk about that some other time
13:42:26 <mmaslano> ncoghlan: I thought external deps, thanks
13:42:44 <mmaslano> I guess remember a list of known external deps
13:42:45 <hhorak> so how would the source differ from upstream repo? would it actually differ or it would stay deliberately untouched like real mirror?
13:42:52 <bkabrda> anyway, what I still haven't figured out (not even for myself) is whether we would want to keep older versions of packages... or just provide newest ones/just provide selected versions/...
13:42:56 <ncoghlan> bkabrda: there's a reason I'm starting with the distro specific repo approach instead :)
13:43:30 <ncoghlan> hhorak, bkabrda: I think those are open questions we don't need to decide here
13:43:45 <bkabrda> hhorak: that's another fair question that I haven't really found out answer to (yet :))
13:44:38 <ncoghlan> for tonight, I'd be looking to answer "What's the next step?"
13:44:40 <ncoghlan> a Python/devpi pilot as an F22 proposal?
13:45:13 <bkabrda> ncoghlan: I think we need to have these things figure out to create a proposal out of it
13:45:15 <ncoghlan> we at least have PEP 440 approved now, so if we wait for pip 1.6, we'll be able to post patched packages without version conflicts with upstream
13:45:59 <bkabrda> ncoghlan: IMO we should first create the devpi instance and start experimenting with different ways of using it; then when we figure out what/how we want to do, we can create a Fedora feature
13:46:51 <ncoghlan> bkabrda: OK, so maybe the next step is to create a draft for discussion at the next WG meeting?
13:47:28 <bkabrda> ncoghlan: yep. and I guess I can request an instance in Fedora cloud to start playing around with devpi
13:47:43 <ncoghlan> bkabrda: sounds good
13:48:20 <ncoghlan> one question mark in my mind is how well it would cope with a 15k+ package white list
13:49:06 <bkabrda> ncoghlan: I guess we'll find out (and improve where needed) :)
13:49:31 <hhorak> #info dependencies on some system libraries are complicated, could be partly solved by new format wheel and some tricks in devpi
13:49:32 <hhorak> #info we should first create the devpi instance and start experimenting with different ways of using it; then when we figure out what/how we want to do, we can create a Fedora feature
13:49:39 <ncoghlan> as I haven't looked into the details of how that is implemented
13:49:40 <ncoghlan> still, "try it and see" is likely the easiest way to answer that :)
13:49:57 <bkabrda> ncoghlan: same here :)
13:50:08 <hhorak> #action bkabrda will request an instance in Fedora cloud to start playing around with devpi
13:50:28 <mmaslano> if you have some plan, we could look at other languages
13:50:54 <ncoghlan> my main problem in those cases is I don't know the mirroring tools at all
13:52:01 <ncoghlan> I've started bugging some of the other devs in the Brisbane office to join the WG
13:52:02 <ncoghlan> or, rather, the mailing list
13:52:08 <ncoghlan> since we have Perl, Ruby and Java apps to consider as well
13:52:29 <mmaslano> ncoghlan: around me are sitting people with knowledge of all these languages
13:52:42 <mmaslano> ncoghlan: you can stop thinking about Java, msrb is looking at maven :)
13:52:49 <mmaslano> and they have special needs
13:53:07 <mmaslano> but other languages group can be inspired by your approach, or might not.
13:53:10 <ncoghlan> mmaslano: it's also in the other direction, though - these are the folks that will be *using* it to deploy internal apps
13:53:53 <bkabrda> I guess more opinions from people with different perspectives won't hurt :)
13:53:54 <mmaslano> ncoghlan: ok, they can talk :)
13:54:22 <ncoghlan> I know what will work for the Python apps
13:54:22 <ncoghlan> but don't have the same level of knowledge for our other apps
13:54:27 <ncoghlan> anyway, we have somewhere to start
13:54:57 <ncoghlan> if it expands beyond Python over time, cool :)
13:55:07 <hhorak> back to draft for discussion for some of the next WG meetings -- do we have a volunteer who will prepare something after some initial testing?
13:55:47 <ncoghlan> I'm happy to do a write-up after we experiment a bit with bkabrda's devpi instance
13:56:23 <hhorak> ncoghlan: bkabrda: great, thanks!
13:56:55 <hhorak> #action ncoghlan will do a write-up after we experiment a bit with bkabrda's devpi instance
13:57:02 <ncoghlan> well, I can start the write-up straightaway
13:57:03 <ncoghlan> and fill in more details as we try different things
13:57:13 <hhorak> ncoghlan: even better :)
13:59:35 <hhorak> So, I'm still thinking how this could work in practice.. I see why developers would like to work with our new shiny repos rather than with RPMs.
13:59:35 <hhorak> But I'm still uncertain why they would not use upstream repo directly? Or they would not notice?
14:00:16 <ncoghlan> hhorak: in our case, it's because we want the additional level of review that Fedora provides
14:00:28 <bkabrda> hhorak: speaking for Python, one advantage would be that they would get precompiled C-extension packages
14:00:42 <ncoghlan> and yeah, the ability to post wheel files
14:00:46 <bkabrda> so they wouldn't need gcc and "whatever-devel" packages on their machines
14:01:04 <ncoghlan> PyPI doesn't allow those for Linux yet, because we need to figure out the external dependencies problem
14:01:25 <ncoghlan> and that's a nightmare in the general case for Linux
14:01:33 <ncoghlan> but eminently solvable in the distro specific case
14:01:41 <juhp_> binaries?  wouldn't they violate copr policy?
14:01:59 <bkabrda> juhp_: for copr, we would rebuild the upstream source again
14:02:03 <juhp_> okay
14:02:16 <bkabrda> at least that's my idea of how we would do that
14:02:24 <ncoghlan> bkabrda: yeah, agreed
14:02:43 <ncoghlan> devpi actually allows us to keep the source and wheel files in separate indices
14:03:13 <juhp_> okay cool
14:03:31 <ncoghlan> with the per-version wheel indices inheriting from a shared source index
14:04:28 <juhp_> wheels are distro specific?
14:04:34 <hhorak> #info reasons why developers would not use upstream repo directly and rather use Fedora's playground is because they might want the additional level of review that Fedora provides + they would get precompiled C-extension packages
14:05:00 <ncoghlan> juhp_: binary dependencies on Linux are distro specific
14:05:11 <bkabrda> hhorak: I'm assuming that we can come up with more reasons, these are just some that occured right now :)
14:05:39 <ncoghlan> juhp_: and since the wheels can contain built binaries linked to distro binaries...
14:05:48 <juhp_> ok
14:06:37 <hhorak> bkabrda: I hope there are
14:07:48 <juhp_> anyway having a devpi instance for Fedora/EPEL sounds useful
14:08:05 <ncoghlan> hhorak: from my point of view, one reason is "we're going to set up a private PyPI mirror for Red Hat's internal use anyway, and hosting that in Fedora rather than inside the firewall means it might be useful to others"
14:08:06 <ncoghlan> the amount of work is basically the same for us either way
14:08:55 <hhorak> ncoghlan: right, sounds more than fine to me.
14:10:38 <hhorak> So, enough interest to discuss another topic or should we move it to the next meeting? (SCLs, building above them and their position in Fedora/EPEL)
14:10:55 <juhp_> ncoghlan, the problem may be more about getting consensus on the policy for the "mirror"
14:11:07 <hhorak> (I should ask first if we're done with the current topic already :) )
14:11:45 <ncoghlan> it just ticked over to Wednesday morning here - did we want to move on to SCLs for a bit?
14:12:03 <ncoghlan> juhp_: aye, agreed
14:12:07 <juhp_> +1 for moving to SCL
14:12:21 <bkabrda> let's move. we'll continue next week either way :)
14:14:45 <hhorak> #topic SCLs, building above them and their position in Fedora/EPEL
14:15:14 <mmaslano> as far as I know some projects are not packaging their projects as SCL and they are using SCL
14:15:33 <mmaslano> one reason is that they have to run their project on Fedora (no SCL) and on CentOS
14:16:10 <mmaslano> SCL packaging doesn't have to be mandatory if you use smart enough wrapper script for setting up environment
14:17:00 <ncoghlan> right
14:17:52 <ncoghlan> it also works if even a container would be too heavy, but you want to decouple your language runtime upgrade cycles from the OS one
14:17:53 <mmaslano> my recommendation would be not do SCL mandatory, it's up to users
14:18:25 <juhp_> mmaslano, I tend to agree
14:18:34 <ncoghlan> that is, still use an SCL on Fedora, but it might be *older* than the system runtime
14:19:28 <ncoghlan> I view the 4 approaches (parallel install, normal package depending on SCL, SCL depending on SCL, full container) as handling different use case
14:19:34 <mmaslano> I know mainly about OpenShift use-case. They have lot of ifdefs for various systems..
14:19:37 <hhorak> mmaslano: I totally agree, but it also kind of suggests supporting those wrappers, which already was discussed a bit, but not solved yet..
14:20:04 <mmaslano> I'm not sure how they are doing with containers, but they've planned to use wrapper inside the container to switch the version of language
14:20:14 <mmaslano> I guess that didn't work well for Python
14:21:13 <ncoghlan> hhorak: it may also be a case where if
14:21:13 <ncoghlan> "normal packaging depending on SCL" gets unmaintainable
14:21:13 <ncoghlan> it's time to consider "SCL depending on SCL" or "container image"
14:22:10 <mmaslano> yes
14:22:35 <hhorak> yeah, I'm kind of confused when thinking about a python app build for SCL that would not be SCL -- basically where the files would be stored?
14:22:48 <ncoghlan> I'm not sure - virtualenv does let you switch the Python runtime in such a way most Python software will adjust automatically
14:23:10 <ncoghlan> hhorak: consider a single file Python script installed into /usr/bin
14:23:32 <ncoghlan> hhorak: with the support files installed into the SCL
14:23:35 <mmaslano> hm we discussed it many times and Python sounds hard, because you can't get rid off it from minimal install
14:23:40 <mmaslano> of image
14:23:54 <ncoghlan> that's a "normal package
14:24:02 <ncoghlan> but you need the SCL for the Python runtime
14:24:34 <ncoghlan> it's a reasonable thing to want to do, but AFAIK the tools to get the shebang line and spec file right don't exist yet
14:24:59 <mmaslano> ncoghlan: I could ask OpenShift guys how they've solved it if you want
14:25:14 <mmaslano> well I ask anyway :) I'm curious how they are doing with SCL and images
14:25:32 <ncoghlan> note I'm also only considering the case where you *always* use the SCL, even on Fedora
14:25:49 <ncoghlan> yeah, I'd be interested to know
14:26:19 <ncoghlan> at the moment, we just run our command line clients in the system language runtimes
14:26:33 * juhp_ sometimes thinks we should only have SCLs ;)
14:26:37 <mmaslano> #action mmaslano will figure out how OpenShift is using SCL in their images
14:27:28 <ncoghlan> juhp_: there are some spectacular rants online about Linux distros "ruining" dynamic languages by shipping old runtimes :)
14:27:40 <juhp_> :)
14:27:48 <mmaslano> ncoghlan: that's old :)
14:28:06 * mmaslano needs to go to another meeting, bye for now
14:28:15 <hhorak> #info SCL packaging doesn't have to be mandatory if you use smart enough wrapper script for setting up environment
14:28:15 <hhorak> #info tools to get the shebang line and spec file right for non-SCL python packages depending on SCL might not exist yet
14:28:29 <ncoghlan> and I will head to bed - thanks folks, looking forward to working more with you!
14:28:52 <bkabrda> ncoghlan: thanks for coming and see you!
14:29:27 <juhp_> ncoghlan, good night
14:29:46 <hhorak> I guess we can close the meeting, thanks everybody for joining!
14:29:46 <hhorak> Anybody interested in doing a chairman the next week?
14:31:35 <hhorak> Since nobody responds, the volunteer would probably be me :) Never mind, I can do it..
14:31:35 <hhorak> Will close the meeting in 1 minute...
14:32:05 <juhp_> hhorak, sorry I rather wait for another week - next week not so good for me
14:32:18 <hhorak> juhp_: Sure, no problem.
14:32:36 <juhp_> hhorak, thanks
14:32:45 <hhorak> #action hhorak will be chairman for the next meeting
14:32:45 <hhorak> #endmeeting