big_data_sig
LOGS
15:03:08 <tstclair> #startmeeting
15:03:08 <zodbot> Meeting started Thu Nov 21 15:03:08 2013 UTC.  The chair is tstclair. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:08 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:03:21 <tstclair> #meetingname BIG_DATA_SIG
15:03:21 <zodbot> The meeting name has been set to 'big_data_sig'
15:03:39 <tstclair> #meetingtopic Collect_Agenda_Items
15:03:42 <mattf> morning
15:04:35 <tstclair> Morning folks, right now I figure we can collect agenda items.  e.g. - For me I have reviews which have been held up for a while.
15:04:59 <mattf> btw, congrats on getting tachyon in!
15:05:11 <tstclair> mattf, thank you ;-)
15:06:08 <tstclair> pmackinn, willb, mattf, rsquared - other agenda items?
15:06:08 <rsquared> No agenda items from me atm
15:06:11 <mattf> agenda - if i read #fedora-bigdata correctly, it sounds like willb is still deep in sbt/scala, could you share some of the experience / existing issues w/ everyone?
15:06:15 <pmackinn> nothing here
15:06:31 <willb> absolutely
15:07:22 <tstclair> #topic status
15:07:27 <mattf> agenda - there were also a few new folks that showed up looking for ways to jump in, but i don't seem them in the channel atm, so maybe table a topic of figuring out short tasks for initial involvement
15:08:04 <tstclair> We could update the SIG page to add "On Boarding"
15:08:08 <mattf> status - i've been poking a few reviews since it seems like we're developing a backlog
15:08:14 <tstclair> I could take that as an action item
15:10:35 <tstclair> status - I've been circling back to update mesos review package, however there has been a fundamental change in the startup routine for spawn'd processes which causes the whole testsuite to hang under mock.  At this point I'm inclined to disable the tests entirely.  (New issue w/latest rebase)
15:12:33 <tstclair> rsquared, pmackinn  any new status updates to share with the whole SIG?
15:12:44 <mattf> status - i emailed rbergeron about big data sig efforts making their way into f20 announcements, waiting to hear back
15:13:05 <tstclair> mattf, It would be nice if it were a headliner.
15:13:18 <mattf> agreed
15:13:33 <willb> I'll get to my status when we come around to my topic
15:13:44 <pmackinn> status - time-api in updates pending; working on datanucleus-core but new xmvn 1.3 has thrown a wrench into that effort
15:14:06 <tstclair> pmackinn, what's the issue?
15:14:54 <tstclair> willb, fire away whenever.
15:15:08 <pmackinn> xmvn 1.3 doesn't like seeing system-scoped deps now, breaks install phase
15:15:34 <rsquared> status - There's some movement upstream to pull in some of the hadoop patches, and I've been working to support that effort.  hbase is in spec creation phase.  There are a number of jamon packages in review, but it looks like those are going to be picked up.  Other hbase deps will need to be reviewed and approved before hbase and get in
15:16:31 <tstclair> rsquared, is there an hbase page with a link to the deps?  I may be able to take a couple of reviews
15:16:57 <mattf> rsquared, have you reached out to stevel in email or twitter?
15:17:18 * rsquared doesn't twitter
15:17:32 <willb> OK, I will start with a high-level overview (apologies to those of you who've already heard this)
15:17:46 <rsquared> tstclair: https://fedoraproject.org/wiki/SIGs/bigdata/packaging/Hbase#Missing_Java_Dependencies
15:18:23 <willb> We need sbt in Fedora to support most of the Scala ecosystem.  From a Big Data SIG perspective, Spark and Shark are very interesting projects that we could package given sbt support, but there are a lot of other cool Scala projects that would be great in Fedora for a general audience.
15:18:43 <rsquared> mattf: You mean stevel in the hadoop team?
15:19:08 <mattf> rsquared, yeah, looks like he's on a tear updating poms
15:19:45 <willb> From 30,000 feet, the big issues with sbt from a Fedora perspective are that it builds itself, needs to use Ivy to find its own dependencies (many of which also need sbt to build), and is designed to be totally self-contained (i.e. you download a launcher and it pulls down deps, including the Scala libraries and compiler from remote Ivy repositories).
15:20:36 <willb> The scala package itself was also broken for some time in Fedora, but I was able to get it working again and one of the maintainers built it with my spec and patches.
15:20:50 <rsquared> mattf: He's committed a few of the patches I submitted and I've intereacted over jira and some in e-mail via the common-dev mailing list when needed.
15:20:53 <willb> We don't have great Ivy support in Fedora at the moment, but I understand that Mikolaj is working to improve that.  And we've seen Fedora do amazing things with similar language-specific ecosystems (maven, rubygems, python eggs, etc).  For now, I've written a script that generates a local Ivy repository from Fedora's maven metadata and am using that to build sbt.
15:21:20 <rsquared> mattf: There's no real plan for hadoop releases so not sure how to push for inclusion of more difficult changes into a roadmap
15:22:16 <willb> Earlier this year, I did a lot of work to get sbt to build itself against a local Ivy repository, so I could use this script.  (I also got a patch in to sbt that will make it easier to integrate with Fedora in future.)  In the last couple of weeks, I've been ironing out problems with that work:  moving dependency versions from F18->F19 and from sbt 0.12 to 0.13 (and thus scala 2.10 instead of scala 2.09), squashing random
15:22:16 <willb> dependencies on my environment (like jars that sbt had cached in my home dir), fixing issues that crop up in rpm builds but not in local builds, etc.
15:22:29 <mattf> rsquared, there's a plan, just hard to ferret out. you should consider reaching out to him.
15:24:42 <rsquared> mattf: It didn't seem that way when discussing java 7.  The impression I got was no one had any roadmap past the 2.2.0 release so they couldn't properly plan for java 7/jetty 9 inclusion
15:25:28 <rsquared> mattf: If there was a plan, no one with knowledge of it spoke up in the thread
15:26:28 <willb> So I've been building up a spec for sbt from my experience building it locally.  It's been somewhat slow going because the workflow is "fire off a build and then add any transitively-carried Ivy dependencies that we didn't already have."  Based on the volume of errors with each build, though, I suspect I'm very close to getting a bootstrap build of sbt under rpmbuild.
15:27:50 <willb> I should also note that the sbt upstream developer has been extremely helpful, even with non-sbt-specific issues like Ivy details
15:28:29 <mattf> sounds like a tedious effort. anything we should do w/ the java sig (or others?) to make it less tedious?
15:29:14 <willb> yes, first-class Ivy support in Fedora (a la xmvn) is basically a must to make this supportable in the long term
15:29:49 <willb> right now, I'm at ~100 dependencies in the locally-generated Ivy repository
15:33:01 <tstclair> Are there any finite self contained items, 1st class Ivy support is a bit much.
15:33:21 <tstclair> I'm sure the java-sig guru's would be better suited for that one.
15:33:56 <willb> tstclair, it is already on Mikolaj's plate IIUC
15:34:44 <willb> but I think this case (along with other JVM build tools that rely on Ivy to lesser extents) is an important motivation
15:35:26 <tstclair> rsquared, I'll try to pickup a review today.
15:36:12 <tstclair> are there any other items folks wanted to discuss?
15:37:01 <mattf> nothing here
15:37:12 <rsquared> tstclair: The jamon packages depend on each other iirc.  Likely only 1 review can get done at a time until that review is built in rawhide.
15:37:28 <tstclair> rsquared, ack
15:38:28 <tstclair> ok if there are no more items, we can call it.  Thanks everyone!
15:38:38 <mattf> thanks tim!
15:38:41 <willb> thanks tstclair
15:38:52 <tstclair> #endmeeting