15:00:42 <hagarth> #startmeeting
15:00:42 <zodbot> Meeting started Wed Jan 29 15:00:42 2014 UTC.  The chair is hagarth. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:42 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:42 <glustermeetbot> Meeting started Wed Jan 29 15:11:00 2014 UTC.  The chair is hagarth. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:43 <glustermeetbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:00:58 * kkeithley is here
15:00:59 <hagarth> who do we have here now?
15:01:02 * purpleidea is here
15:01:08 * ndevos listens in
15:01:16 * raghu` is here
15:01:24 * jdarcy is present
15:01:41 <hagarth> great, we seem to have all our usual suspects.
15:01:45 <hagarth> let us get going.
15:01:53 <hagarth> #topic AI from last week
15:02:30 <hagarth> me and kkeithley had a chat on release-3.4 and release-3.3 maintenance, so that was completed.
15:02:55 <hagarth> ndevos helped with new key for enabling compression xlator - thanks ndevos!
15:03:05 <kkeithley> but I still don't get a "submit patch" button in gerrit for 3.4 and 3.3 patch sets
15:03:35 <hagarth> kkeithley: yes, we need to sort that out. Did try quite a few things, need to check what I am missing in my gerrit arsenal ;)
15:03:59 <hagarth> we setup a page for 3.5 testing - #link https://docs.google.com/spreadsheet/ccc?key=0ApI11Dqpup82dGZCYi1PT3pHdTNFYlZXMEdBOGdweUE#gid=0
15:04:34 <hagarth> lalatenduM and me are still looking into the bug triage process - will have it sorted this week.
15:05:14 <hagarth> #topic 3.5.0
15:05:25 <hagarth> we got 3.5.0beta2 out this week
15:05:43 * lalatenduM here
15:05:51 <hagarth> some tests are being run and test owners are covered in the link I pasted above
15:06:20 <hagarth> we plan to have some results by end of this week
15:06:33 <purpleidea> fwiw, beta1 built great with puppet, however beta2 had a small issue... not sure why exactly
15:07:11 <hagarth> there are still a few vacant slots as far as testing components are concerned, please help us by picking up such components and providing some test coverage
15:07:20 <hagarth> purpleidea: what kind of problem did we run into?
15:08:03 <purpleidea> hagarth: i got an error from glusterd complaining it didn't like the volfile, but only on one of the hosts... they're all identical though. i saved some logs
15:08:34 <hagarth> purpleidea: it might be good to analyze that if you can fpaste it later
15:08:50 <lalatenduM> hagarth, I was looking at pk's patch http://review.gluster.org/6854, we might want it for 3.5
15:09:06 <purpleidea> hagarth: indeed. i'm going to try and reproduce it first
15:09:14 <hagarth> lalatenduM: that has already been merged in release-3.5.
15:09:23 <lalatenduM> hagarth, awesome :)
15:09:28 <hagarth> known issues with 3.5 at the moment:
15:09:47 <hagarth> 1. compression translator does not work well with write-behind and afr - we need to fix this.
15:10:32 <hagarth> 2. aux-gfid-mount crashes on most activities performed on it .. there does seem to be some problem with lookup. Need to address this.
15:11:16 <hagarth> 3. a few patches to be incorporated for both geo-rep and quota, hopefully beta3 will contain all of them
15:11:31 <lalatenduM> hagarth, nice :)
15:12:06 <hagarth> so the plan is to release beta3 this week, run more tests and harden that for a potential release candidate
15:12:36 <hagarth> again if you notice any problems with 3.5, please add it to the 3.5 blocker bug or send out a note on gluster-devel
15:12:58 <hagarth> lalatenduM, msvbhat_, purpleidea: can we update the spreadsheet with the status of our testing by Friday?
15:13:25 <purpleidea> sure
15:13:41 <hagarth> purpleidea: cool, thanks!
15:13:42 <lalatenduM> hagarth, seems difficult for me, will try for sure
15:13:48 <hagarth> #action test owners to update 3.5 test page with results or send an email on gluster-devel.
15:14:00 <hagarth> lalatenduM: just noted an action item lest we forget
15:14:08 <lalatenduM> hagarth, sure
15:14:33 <hagarth> I think we need to arrive at what features in 3.5 should be termed as beta soon.
15:15:30 <hagarth> wonder what would be our criterion for calling a feature beta:
15:15:37 <hagarth> 1. inadequate test coverage ?
15:15:47 <hagarth> 2. not fully functional as expected ?
15:16:02 <hagarth> 3. one of 1. or 2.
15:16:10 <hagarth> 4. both 1. & 2. ?
15:16:10 <lalatenduM> hagarth, I will vote for 2
15:16:35 <purpleidea> imho, if it's not functioning as expected, it's not even beta yet
15:16:57 * ndevos agrees with purpleidea
15:17:00 <lalatenduM> hagarth, agree with purpleidea
15:17:27 <hagarth> ok, how should we qualify new features that don't get any test coverage?
15:17:54 <kkeithley> tech preview?
15:18:04 * purpleidea agrees with kkeithley
15:18:13 <hagarth> kkeithley: +1
15:18:19 <lalatenduM> kkeithley, +1
15:18:29 <kkeithley> or untested!
15:18:50 <lalatenduM> I think untested is better
15:18:54 <kkeithley> untested; you've been warned
15:19:03 <hagarth> I am very hesitant about pushing out a feature as fully supported in a release if we don't know its state from pre-release testing.
15:19:25 <kkeithley> yeah, one of the Red Hat mantras is "we ship it, we support it"
15:19:40 <hagarth> we have had our share of issues with quota, replace-brick and rdma in 3.4.x, we should not hit the same scenario with 3.5 as well
15:19:41 <lalatenduM> IMO "untested" gives a clear picture
15:20:12 <hagarth> lalatenduM, kkeithley: agree, should we add a warning in CLI for such features?
15:20:27 <ira> 2. with minor issues, only.
15:20:43 <hagarth> so that folks who don't RTFM understand the state of a feature?
15:21:00 <lalatenduM> hagarth, not sure, can it be easily done?
15:21:03 <purpleidea> hagarth: seems you're being overly kind to people that don't rtfm
15:21:13 * kkeithley knows that won't stop people from expecting things to work
15:21:21 <purpleidea> it's distributed storage, not a mail client :P
15:21:24 * ira agrees with kkeithley
15:21:39 <hagarth> lalatenduM: it would be trivial to have that
15:21:39 <lalatenduM> purpleidea, yup :)
15:21:55 <hagarth> should we just release note such features?
15:22:08 <ira> untested things?
15:22:14 <hagarth> ira: yes
15:22:14 <lalatenduM> hagarth, I think release note is fine
15:22:26 <ira> hagarth: They should be release noted as broken.
15:22:32 <purpleidea> hagarth: would it make sense to rename a command in cli from "command" to "alpha_command" instead?
15:22:43 <purpleidea> or require a --this-is-alpha flag?
15:22:51 <jdarcy> The other option would be to disable such features in the RPM defaults.  I don't like it, but thought I'd throw it out there.
15:22:51 <lalatenduM> ira, I think "untested" would be more clear
15:23:00 <hagarth> purpleidea: we can add an attribute in CLI to identify such features.
15:23:14 <ira> lalatenduM: I'd rather people know the likely truth ;)
15:23:44 <hagarth> ira: how about likely broken? ;)
15:23:54 <lalatenduM> ira, the truth is no body tested it :) , actually I was talking abt features which did not have test coverage
15:24:14 <ira> hagarth: I'll accept that.
15:25:02 <lalatenduM> hagarth, ira , are you guys talking abt "inadequate test coverage" or " not fully functional as expected"?
15:25:22 <lalatenduM> i mean for "likely broken"
15:25:25 <ira> lalatenduM: I'm talking about inadequate test coverage.
15:25:37 <lalatenduM> ira, ok
15:25:40 <ira> lalatenduM: If we aren't confident it is working...
15:25:44 <hagarth> lalatenduM: inadequate test coverage here too.
15:25:52 <lalatenduM> ok
15:26:09 <hagarth> I think it is safe to err on the side of caution - we can always remove that tag in a subsequent minor release
15:26:20 <hagarth> if we get more testing cover subsequently
15:26:29 <kkeithley> fwiw, the picture will get better soon. lpabon (who's not here :-( ) is setting up more testing, including unit tests (with junit, instead of something a little newer :-( )
15:27:18 <hagarth> kkeithley: yeah, it is based on cmockery, CI tests ftw!
15:27:46 <ira> Sounds good.  :)
15:28:04 <kkeithley> I'll take whatever we can get.
15:28:18 <hagarth> I plan to increase the scope of our nightly regression tests in jenkins also in the days to come - so we should get better with respect to quality (hopefully soon)!
15:28:23 <kkeithley> junit and cppunit are a bit dated though.
15:28:48 <purpleidea> hagarth: on the subj. of testing, has anyone tried the vagrant stuff i built?
15:29:01 <purpleidea> (just curious)
15:29:02 <kkeithley> wish I could remember what an acquaintance of mine said his shop was using that was newer/better than {j,cpp}unit
15:29:04 <hagarth> purpleidea: I thought kshlm did try that
15:29:22 <purpleidea> yup
15:29:48 <lalatenduM> purpleidea, I would , not getting time to do that as of now
15:29:59 <hagarth> purpleidea: he did find it to be quite useful
15:30:32 <hagarth> I think we need to circulate the 3.5 feature tagging guidelines on the mailing lists.
15:30:50 <hagarth> any volunteers for this AI?
15:31:44 <hagarth> #action hagarth to send out guidelines on 3.5 feature tagging on ML
15:31:54 <hagarth> any more questions on 3.5?
15:32:21 <hagarth> guess not, moving on
15:32:23 <hagarth> #topic 3.4
15:32:46 <hagarth> kkeithley: do you want to provide an update on 3.4.3?
15:32:52 <kkeithley> not much to report here. I'm going to open a tracker BZ
15:33:09 <kkeithley> and seed the planning page with some more backports
15:33:41 <hagarth> kkeithley: we would need to pull in http://review.gluster.org/6854
15:33:55 <kkeithley> yes, just as soon as.... ;-)
15:34:04 <hagarth> kkeithley: I know that ;)
15:34:28 <hagarth> #action hagarth to sort out gerrit for release-3.4 and release-3.3
15:34:33 <kkeithley> anyway, and run a qa1 or alpha1 release
15:34:46 <hagarth> kkeithley: yeah, sounds good.
15:34:48 <kkeithley> run a qa1 or alpha1 release soon
15:35:06 <hagarth> anything else on 3.4?
15:35:32 <hagarth> guess not, let us move on.
15:35:36 <hagarth> #topic 3.6
15:35:58 <jdarcy> Need more feature requests.
15:35:58 <hagarth> we have a few more features appearing on the planning page for 3.6 - #link http://www.gluster.org/community/documentation/index.php/Planning36
15:36:16 <hagarth> jdarcy: yeah, I plan to add a few more this week.
15:36:44 <hagarth> all, please try to add more requests sooner than later
15:37:04 <lalatenduM> hagarth, what about bit rot , is it also for 3.6?
15:37:17 <purpleidea> is "tiering" likely for 3.6 plans?
15:37:26 <hagarth> lalatenduM: that is what Shishir intends to do.
15:37:41 <jdarcy> purpleidea: Tiering is a subset of data classification.
15:37:43 <hagarth> purpleidea: yes, data classification does include "tiering".
15:37:45 <lalatenduM> hagarth, awesome :)
15:38:01 <purpleidea> cool
15:38:20 <hagarth> jdarcy, kkeithley: I am wondering if we should attempt to get better multi-tenancy support in 3.6?
15:38:21 <kkeithley> where does Shishir's bitrot stand in relation to lvm- or btrfs-based bitrot detection?
15:38:26 <jdarcy> FWIW, we're starting to get beaten up about that.  Ceph/HDFS/Swift are all making moves in that direction.
15:38:43 <hagarth> jdarcy: yeah
15:38:50 <jdarcy> hagarth: What parts?
15:39:17 <hagarth> kkeithley: Shishir intends to make the bitrot computation and verification pluggable. so we should be able to leverage multiple schemes.
15:39:19 <kkeithley> I'm being pulled in about seven different directions, all away from multi-tenancy
15:39:47 <hagarth> jdarcy: I was thinking about the directory labeling scheme that we had discussed some time back.
15:39:59 <jdarcy> "Better SSL" covers a few multi-tenant pieces.  Biggest remaining part is the changes to brick directory structure and daemon management.
15:40:17 <jdarcy> That stuff's going to take a while, so it would be good to get some of the infrastructure pieces into feature-request form.
15:41:05 <jdarcy> Also, same story as kkeithley.  Too many other demands.
15:41:19 <hagarth> jdarcy: agree, given our target to improve adoption in OpenStack and other public clouds, we  need to target some parts of it in 3.6
15:42:28 <hagarth> yeah, let us see how we can get that done. I think we can also attempt to have some of this covered in GSoC which should start in the 3.6 window
15:42:31 <kkeithley> I'm inclined to think that we should make sure 4.0 scalability includes those. And then maybe it doesn't make sense to try to cram it into 3.x
15:42:47 <kkeithley> 4.0 scalability work
15:43:30 <hagarth> kkeithley: that is a good thought too - we probably should do a discussion next week on how to slot features into 3.6 and 4.0
15:44:03 <hagarth> or identify the rationale for doing something in 3.6.
15:44:10 <kkeithley> sure
15:44:29 <hagarth> since johnmark is not around, let us tag him with an AI.
15:44:34 <jdarcy> One could probably make the same "leave it until 4.0" argument for the glusterd scaling work.
15:44:54 <hagarth> #action johnmark to reach out to usual suspects for adding feature pages in 3.6 planning page
15:45:00 <hagarth> :)
15:45:14 <hagarth> jdarcy: agree, that seems like a natural fit for 4.0
15:45:35 <kkeithley> I wasn't aware that we were doing glusterd scaling work in 3.6; I thought that was the point of 4.0.
15:45:50 <kkeithley> but I'm probably out of the loop on that
15:46:09 <hagarth> kkeithley: it is on the Planning36 page atm, but we can debate more next week.
15:46:28 <hagarth> anything else to be discussed on 3.6?
15:46:46 <purpleidea> hagarth: one small thing
15:47:06 <hagarth> purpleidea: go ahead
15:47:17 <purpleidea> if someone can cc me on the interfaces for some of the new features, long before they're beta, i can maybe integrate them with puppet in advance...
15:47:26 <jdarcy> http://www.gluster.org/roadmaps/ is seriously in need of an update
15:48:17 <hagarth> purpleidea: sure, would it be useful to add you as a reviewer in gerrit when a new patchset appears with an interface addition/change?
15:48:36 <hagarth> jdarcy: am glad that you brought it up - another AI for johnmark.
15:48:54 <purpleidea> hagarth: up to you... i probably don't need to look at c code, but knowing the command line interfaces so that i know how to "build" and use the features would be beneficial...
15:49:07 <hagarth> #action johnmark to help with updating http://www.gluster.org/roadmaps/
15:49:20 <purpleidea> i just found out by accident last week about how to do ssl in gluster... if i know in advance of how it will work, i can have it ready to test and automate long before a beta.
15:49:40 <hagarth> purpleidea: what would you prefer to look at - feature pages or cli code (which usually happens to be in C)?
15:50:20 <hagarth> purpleidea: feature pages do get stale over time - code will be more current.
15:50:26 <purpleidea> hagarth: feature pages atm don't include enough details. C is okay, but example command line usage is even better
15:50:44 <hagarth> purpleidea: sure, let us try to evolve something from now on around this.
15:51:06 <jdarcy> "Bro pages"
15:51:09 <purpleidea> none of this is essential, and i know you guys are busy, but i'm just trying to see if i can do a better job and get ahead of some of the features before they're released
15:51:27 <lalatenduM> hagarth, purpleidea I think daily builds will help in this area
15:51:43 <hagarth> purpleidea: really appreciate that - we would want to make it convenient for you too
15:51:48 <purpleidea> lalatenduM: would be great. esp. if rpms exist
15:52:14 <hagarth> purpleidea: yes, we have nightly builds at #link http://download.gluster.org/pub/gluster/glusterfs/nightly/
15:52:29 <jdarcy> http://bropages.org/
15:52:57 * kkeithley is scared to look
15:53:00 <purpleidea> hagarth: not sure if anyone has access to hook me up with rhel7 images, but that's another thing to test early if it's available
15:53:31 <hagarth> jdarcy: good idea, none of our man pages are current too
15:53:39 <hagarth> purpleidea: is centos7 available?
15:54:06 <lalatenduM> hagarth, I think you mean CentOS 7 beta
15:54:08 <purpleidea> hagarth: don't think so. no rush though
15:54:16 <hagarth> purpleidea: http://seven.centos.org/
15:54:23 <ndevos> purpleidea: teh beta is pubilc: http://ftp.redhat.com/redhat/rhel/beta/7/x86_64/iso/
15:54:33 <hagarth> lalatenduM: yes, I meant the beta
15:54:38 <purpleidea> ndevos: oh :P shows what i know!
15:54:58 <hagarth> awesome, let us move on to our next topic
15:55:13 <hagarth> #topic bug triage process
15:55:34 <hagarth> I found some good guidelines at #link http://www.mediawiki.org/wiki/Bug_management/How_to_triage
15:56:00 * lalatenduM will take a look
15:56:02 <hagarth> lalatenduM: the section at the bottom has more details - I assume we can base our process on this and create a wiki page in gluster.org.
15:56:21 <lalatenduM> hagarth, cool, will go through this and let you my feedback
15:56:33 <hagarth> lalatenduM: thanks
15:57:05 <hagarth> another aspect I have often noticed is lack of guidelines on what information needs to be provided along with a bug report
15:57:22 <lalatenduM> hagarth, yup , I agree
15:57:23 <hagarth> which causes iterations for a bug to be addressed
15:57:41 <lalatenduM> hagarth, without info the bug is less useful
15:57:49 <hagarth> for EL builds, should we ask for sosreport to be included by default?
15:58:18 <hagarth> or would sosreport be too heavyweight?
15:58:25 <lalatenduM> hagarth, if sosreports sometimes more than 20MB we cant attach it to bugzilla
15:58:35 <lalatenduM> hagarth, yeah, i was gonna say that
15:58:56 <hagarth> lalatenduM: yeah, should we first attempt with a .tgz of the glusterfs log directory?
15:59:09 <lalatenduM> hagarth, agree
15:59:49 <hagarth> if anybody has brilliant ideas on how we can support a 100+ node cluster effectively, it would be a great discussion on the mailing list :)
16:00:01 * purpleidea *cough*
16:00:14 <purpleidea> hagarth: ^
16:00:33 <hagarth> lalatenduM: we can probably note that in the list of pre-reqs for a bug report.
16:00:36 <kkeithley> now where to get 100+ nodes
16:00:38 <jdarcy> We could use Chef.
16:00:41 <purpleidea> i brought up the management aspect a while back, but i think people were busy... i needed some help with some of the algorithms...
16:00:42 * jdarcy ducks.
16:00:46 <purpleidea> jdarcy: ;)
16:00:53 <hagarth> jdarcy: haha
16:01:03 <purpleidea> jdarcy: i'm the first to say that i don't like puppet... but i don't know of anything better
16:01:29 <hagarth> purpleidea: yeah, revive that again on gluster-devel now that we have more context?
16:01:30 <purpleidea> kkeithley: vm's on someone's cluster
16:02:03 <hagarth> johnmark might be kind enough to sponsor such a cluster ;)
16:02:17 <jdarcy> Sadly, the problems with 100+ nodes go even deeper than the management level.  We'll run into problems with membership, daemon management, RPC, repair/rebalance behavior, etc.
16:02:30 <purpleidea> hagarth: http://www.gluster.org/pipermail/gluster-users/2013-December/038281.html
16:02:37 <hagarth> purpleidea: noted
16:03:00 <hagarth> jdarcy: agree, but even with something like 32 nodes we would run into supportability problems
16:03:10 <purpleidea> hagarth: happy to work on this, but would strongly benefit from a few hours with a strong gluster hacker (which i'm not)
16:03:22 <hagarth> before some of the algorithmic problems start cropping up.
16:03:39 <hagarth> purpleidea: let us continue this discussion on gluster-devel
16:03:48 <purpleidea> (cc me)
16:03:54 <hagarth> purpleidea: will do
16:04:10 <hagarth> #topic open discussion
16:04:18 <hagarth> any quick topics for open discussion?
16:04:28 <kkeithley> while I have everyone's attention. Please note that jenkins takes itself off line if space gets tight in /var/lib/jenkins. And since build.gluster.org has only a single fs (not counting /d) people who put 1GB files in / (anyone want to fess up to /bd_file)?
16:04:38 <purpleidea> jdarcy to write chef module for glusterfs?
16:04:42 <hagarth> ndevos: will you be attending FOSDEM?
16:04:51 <ndevos> hagarth: yes
16:04:58 <kkeithley> people who put 1GB files in / are setting up jenkins for a fall
16:05:04 <jdarcy> I'll be at FAST in Santa Clara in a couple of weeks, again for Summit in April.
16:05:24 <hagarth> ndevos: cool, have fun! johnmark should be there too.
16:05:36 <purpleidea> i'll be at scale in a month if anyone will be around
16:05:46 <ndevos> hagarth: yeah, he should, maybe jclift too, I'm not su sure about others
16:05:57 <jdarcy> Like ships passing in the night...
16:06:22 <kkeithley> I'd like to go to scale, but it conflicts with our meetings in BLR.
16:06:26 <hagarth> jdarcy: we could probably do a walkthrough of your presentation after FAST to understand performance better.
16:06:38 <kkeithley> But if there's no money for me to go to BLR then there's probably no money for me to go to scale
16:06:40 <hagarth> rather the characterization.
16:07:08 <hagarth> ok, that seems to be it for now.
16:07:09 <jdarcy> I'd love to do a community run-through of that material some time.  Maybe three parts, since it's three hours.
16:07:18 <hagarth> jdarcy: awesome, maybe a community hang out?
16:07:23 <jdarcy> Sure.
16:07:38 <hagarth> jdarcy: great, will look forward to that.
16:07:54 <hagarth> thanks everybody for being here today. See you all next week.
16:07:59 * jdarcy waves.
16:08:01 <hagarth> #endmeeting