15:00:16 <kshlm> #startmeeting Community Meeting 2017-02-01
15:00:16 <zodbot> Meeting started Wed Feb  1 15:00:16 2017 UTC.  The chair is kshlm. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:16 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:16 <zodbot> The meeting name has been set to 'community_meeting_2017-02-01'
15:00:26 <kshlm> #topic Roll call
15:00:34 * vbellur partially available
15:01:06 <kshlm> Hey vbellur!
15:01:37 * kkeithley is here
15:01:52 <kshlm> Hey kkeithley
15:02:01 <kshlm> Is is just us here?
15:02:13 * ivan_rossi lurking
15:02:35 <kshlm> Hey ivan_rossi !
15:03:07 <kshlm> Let's wait another 3 minutes before continuing.
15:03:26 <kshlm> In the meantime, I'll check if we have any topics to discuss.
15:04:01 <kshlm> We have one.
15:04:33 <jdarcy> Just one.
15:04:42 <kshlm> Can the person who added it, add their name to it.
15:04:46 <kshlm> jdarcy, Just one.
15:05:08 <kshlm> On policy for reverting patches.
15:05:38 <jdarcy> That was mine.
15:06:03 <jdarcy> I've prepared my intro for that, to save typing time.
15:06:25 <jdarcy> We recently had an incident where a patch to fix one bug passed regression, but was causing regression failures for others. This ended up blocking *most* patch submissions, for days, right before a release. This behavior, driven by downstream, was disrespectful to the upstream community. Proposal 1: maintainers should be able to revert such patches, or mark affected tests as bad, without delay. Among other things, this mean
15:06:28 <kshlm> jdarcy, Cool
15:06:53 <kshlm> jdarcy, Let me just start the topic.
15:07:01 <JoeJulian> here
15:07:09 <kshlm> #topic Policy on reverting patches that are found to cause spurious test failures
15:07:12 * shyam is here now...
15:07:14 <kshlm> Hey JoeJulian!
15:07:22 <kshlm> jdarcy, You have the floor.
15:07:29 <jdarcy> Should I re-paste?
15:07:33 <kshlm> Nope.
15:07:43 <jdarcy> That was all I had to say.  Any thoughts?
15:07:57 <jdarcy> I expect some controversy on proposal 2.
15:08:13 <kshlm> Proposal 2?
15:08:25 <jdarcy> Proposal 2: committers who persist in enabling such behavior should cease to be committers
15:09:07 <JoeJulian> The paste truncated at "... Among other things, this mean"
15:09:17 <jdarcy> Argh.  Let me re-paste line by line.
15:09:24 <jdarcy> We recently had an incident where a patch to fix one bug passed regression, but was causing regression failures for others.
15:09:31 <jdarcy> This ended up blocking *most* patch submissions, for days, right before a release.
15:09:37 <jdarcy> This behavior, driven by downstream, was disrespectful to the upstream community.
15:09:43 <jdarcy> Among other things, this means that the patch to revert or mark should not have to pass regressions itself.
15:09:52 <jdarcy> Proposal 1: maintainers should be able to revert such patches, or mark affected tests as bad, without delay.
15:09:58 <jdarcy> (oops, reversed those two lines)
15:10:03 <jdarcy> Proposal 2: committers who persist in enabling such behavior should cease to be committers
15:10:44 <vbellur> jdarcy: +1 to both from me
15:10:56 <kshlm> I remember this happening before, and we did revert patches.
15:11:01 <jdarcy> Thank you.
15:11:11 <kkeithley> appropos of nothing, did the offendind committer know they had caused the problem? Shoiuld I presume from the context that they did?  (No, I'm not asking who it was.)
15:11:41 <jdarcy> Yes, there was a fix in the works, but it had some internal complexity and risks so it wasn't going to be ready quickly.
15:11:45 <kshlm> +1 for making reverts easier.
15:12:34 <jdarcy> This stuff happens.  That's OK.  I just want to make it easier to contain the damage.
15:12:53 <shyam> +1 to proposal 1 and 2 (although I hope we can track 2 better)
15:13:18 <kshlm> Tracking 2 will be hard with the way we currently do testing.
15:13:22 <jdarcy> Proposal 1 does require some infra change.  If we agree on that, I'll take the AI to follow up with Nigel.
15:13:28 <kkeithley> I'm okaya with quick reverts
15:14:14 <jdarcy> Proposal 2 is part of a larger discussion (which is kind of already in progress) about how we determine who can and can not commit changes.
15:15:02 <jdarcy> Right now, it's very informal and the set keeps growing.  As we grow, I'm not sure that scales.
15:15:51 <kshlm> We need to have some automation there.
15:15:56 <kkeithley> I had the impression there was an explicit understanding that a committer only commits things he or she is an owner of
15:16:25 <kkeithley> perhaps we need to more explicitness in the ownership part
15:16:55 <jdarcy> kkeithley: Yes, and I think that's mostly honored.
15:17:18 <jdarcy> kkeithley: However, since we test every component on every patch, even "commit what you own" allows situations like this.
15:18:22 <jdarcy> If I own the fubar translator, and the test for fubar starts throwing errors at random, that affects everyone.
15:19:43 <kshlm> This means, we have to rethink our testing strategy.
15:20:17 <kshlm> We've been talking about improvements, and this needs to be considered.
15:20:47 <jdarcy> kshlm: Do you mean that a patch for X should only test X, rely on periodic tests to catch cross-component problems?  Something like that?
15:21:10 <kshlm> jdarcy, That's a possible solution.
15:22:14 <jdarcy> That would keep such a "bad patch" from blocking others.  That's good.
15:22:33 <JoeJulian> I wonder if I could leverage gfapi to build a unit-test framework for per-translator testing...
15:22:52 <jdarcy> OTOH, it might still mean a recurring failure in the burn-in tests.
15:23:42 <jdarcy> JoeJulian: I don't know if it'd be GFAPI, but +1 anyway.
15:24:06 <kshlm> Well, someone would need to own that and ensure it doesn't stay broken for long.
15:24:29 <JoeJulian> I was thinking gfapi because I could write it in python and *I* know how to do *that*.
15:24:35 <shyam> JoeJulian: That is possible and has been in thoughts (of a few people) for a very long time, but no movement since, sort of gfapi sandwiches xlator between itself and POSIX so +1 from me as well
15:25:15 <JoeJulian> (and it would finally be something I could contribute!)
15:25:31 <jdarcy> That's a good thing too.  :)
15:26:14 <jdarcy> So, proposal 2 is probably intractable, but should I pursue infra changes for proposal 1?
15:26:36 <JoeJulian> I have to run. Odd commute today. Next time I can stay longer. Thanks.
15:26:41 <kshlm> jdarcy, Yes for the infra changes.
15:26:50 <kshlm> JoeJulian, Nice having you here.
15:27:09 <jdarcy> Thanks Joe!
15:28:54 <kshlm> jdarcy, Do you wish to discuss this further?
15:29:06 <kshlm> We have one action item from the topic.
15:29:07 <jdarcy> No, I think we're done.  For now.  :evilgrin:
15:29:27 <kshlm> #action jdarcy will work with nigelb to make the infra for reverts easier.
15:29:35 <kshlm> jdarcy, Cool.
15:30:36 <kshlm> #topic 3.10 TODO nag in the community meeting
15:30:49 <kshlm> shyam, You can nag now.
15:30:49 <shyam> :) ok here I go
15:31:11 <shyam> Just reaching out in the meeting as well (as I have sent mails on devel for the same)
15:31:52 <shyam> We need release-notes, for the new features, so can contributors please post them, or I write what best I know :)
15:32:26 <shyam> Also, for a couple of days now review requests (so close to beta1) are not happening, can we get some attention there?
15:32:46 <shyam> Further during 3.9 when we called out for testing, feedback almost never arrived
15:33:02 <shyam> So the biggest nag is, how to gain more eyeballs on the release, and what do folks think?
15:33:34 <kshlm> That's question isn't it. How do we get more people to be active upstream?
15:33:43 <shyam> Yepo!
15:33:54 <kshlm> If people did everything upstream this wouldn't be a problem.
15:34:09 <kshlm> I don't have any good answer for this though.
15:34:17 <kkeithley> I have no 3.10  review requests under "My Reviews".   People might think about adding reviewers to their change i
15:34:28 <shyam> Also, an upstream release is important right, we need folks attention and time, to make that happen
15:34:39 <jdarcy> +100
15:34:48 <shyam> kkeithley: These are reviews I sent by mail, so well you are not one of the implicated ;)
15:35:17 <kkeithley> shyam: indeed. I was just commenting about the general case
15:35:38 <jdarcy> The draconian solution would be to say that if you lag on reviews you'll lag on having your own patches reviewed.
15:35:51 <shyam> What is the problem? Attention or Time? I am leaning towards attention, if so should I nag more on the lists?
15:36:03 <kkeithley> review swaps is de rigeur in other circles
15:36:08 <shyam> kkeithley: understood, I typed away a little too soon!
15:36:41 <shyam> Well to be honest, review swaps do not work here, as mostly work is done within a component (xlator) and gets reviewed by that circle, so nothing to swap
15:37:07 <kkeithley> and the flip side is if you _don't_ review other peoples chances, they _won't_ review yours.
15:37:46 <kkeithley> right, they don't, but we should encourage a culture where they do IMO
15:39:01 * kkeithley thinks we could try
15:39:15 <shyam> Hmmm... I would encourage a culture that when someone says "here is what this release needs please focus on this" people understand it and do the best they can, or say why they cannot
15:39:29 <shyam> IOW, some constant release time messaging and education that these call outs are important
15:39:42 <shyam> If we are to do releases every 3 months, then we need the above I would think
15:39:53 <vbellur> shyam: right
15:40:53 <shyam> Right, I have some thoughts, like gerrit dashboards etc. but ultimately people need to understand *that* is what is most important at the moment for the project
15:41:14 <jdarcy> I'd be in favor of a "review bazaar" to set up two- or three-way exchanges of reviews.  Often the problem is not knowing which reviews are tradeable.
15:41:51 <kshlm> The problem is that what we discuss here just stays here.
15:42:04 <shyam> Ok, during release time, the bazaar is the (mythical) release dashboard, else we cannot meet timelines (and that sucks)
15:42:06 <kshlm> People don't seem to notice the mailing lists as much.
15:42:21 <shyam> Wow! kshlm what?
15:42:32 <jdarcy> Do most of our developers even follow the mailing lists?
15:42:50 <jdarcy> AFAICT it's a small set who do, the rest just rely on hearsay.
15:43:00 <kshlm> I've seen developers with inbox counts in the thousands.
15:43:15 <shyam> ok, are you saying all these mails that I sent on 3.10 are... useless? What other medium can I use then? (I really thought people follow the mails)
15:43:18 <kkeithley> It's entirely off-topic to be sure, but people need to appreciate that when their downstream rebases on $this upstream release, that they're already working on their downstream.
15:43:42 <jdarcy> kkeithley: Very true.
15:43:56 <kshlm> Even if they do follow,  they probably just lose the mails among the rest.
15:44:05 <kkeithley> which _is_ their job.
15:44:25 <jdarcy> People know that bug XXYYZZ needs to be fixed for release ABC, but not so much that feature MNOP also needs to be finished for the same release.
15:44:41 <shyam> Hmmm... kshlm we then need a medium or a set of oracles to make this happen
15:45:03 <kkeithley> the floggings will continue until morale improves
15:45:04 <vbellur> I have a fairly straightforward solution to this
15:45:13 <vbellur> let me know if you think this would work:
15:45:45 <vbellur> 1. we have established community guidelines and best practices. we expect all to adhere to these guidelines.
15:46:11 <vbellur> 2. If there are repeated violations of these guidelines from maintainers, they cease to be maintainers.
15:46:50 <vbellur> 3. This means that activity on components owned by such ex-maintainers would slow down
15:47:32 <kshlm> vbellur, The issue here is not just about maintainers. Its all the developers. Everyone has to do their parts during the releases.
15:47:45 <kshlm> But most people just don't seem to know.
15:47:57 <kshlm> How do we go about improving this?
15:48:00 <jdarcy> Would it be correct to say that one of a maintainer's responsibilities under (1) is to make sure people on their teams know and act on current upstream-release priorities?
15:48:15 <kshlm> I'd agree to that.
15:48:17 <vbellur> kshlm: yes, but maintainers are expected to lead their circle of influence in an upstream friendly way
15:48:29 <shyam> maintainers can be the oracles for the message on upstream releases (or +1 to what jdarcy jsut said)
15:48:54 <vbellur> kshlm: maintainers are around to ensure that the developer community adheres to the larger guidelines
15:49:20 <jdarcy> It might work, but I'm a little concerned that it makes maintainers responsible for others' behavior.  Can we address individual behavior more directly?
15:50:35 <vbellur> jdarcy:  maintainers' can possibly highlight the right behavior without being responsible for others' behavior?
15:50:53 <jdarcy> Perhaps if it was clearer that when I ask a maintainer to review something it's OK (and often good) for them to delegate that review to another member of their team...?
15:51:01 <shyam> My thinking is that we should as a last resort to anything approach the maintainers, like "this needs reviews" we cannot reach all developers, so maybe nto such a bad thing?
15:51:26 <shyam> not such a bad thing making the maintainer responsible for the culture that we are discussing
15:51:39 <jdarcy> The important thing seems to be clarity that *a review is needed* and who owns responsibility for getting it done.
15:52:25 <vbellur> jdarcy: right
15:52:33 <shyam> *Clarity* that "review needs to be done" or a "task needs to be done" can be built (I would say easily)
15:52:44 <vbellur> personally I am very frustrated by maintainers' inattention to failing regression tests
15:52:49 <jdarcy> If the maintainer retains that responsibility personally, great.  If they delegate, and that's known to the person who needs the review, also great.  But don't drop it on the floor.
15:53:24 <vbellur> the biggest turn off for me on gerrit is when I see patches with a "red cross" and several rechecks on centos / netbsd
15:53:39 <jdarcy> IMX deadlines work.  Say that a review will be done, or a recurring regression fixed, by date X or there will be consequences.
15:54:07 <shyam> Well, let me add to that list, no real takers for focus areas, no backlogs entered in github till now, all these are maintainer driven I think! So let's focus there is my opinion as well.
15:54:09 <vbellur> jdarcy: +1
15:54:23 <vbellur> shyam: yes, those too
15:54:44 <shyam> jdarcy +1 or start some scoreboard that after X days docks reputation
15:54:45 <jdarcy> vbellur: I agree.  It's hard to know what's really in a reviewable state when perfectly good and important patches get the "X" and low-quality or low-priority patches don't.
15:56:09 <kshlm> We have 5 minutes left.
15:56:18 <vbellur> so how do we proceed here? what is the first deadline that we want to impose in these 5 minutes? :)
15:56:19 <jdarcy> So, to turn this into something actionable, what are we actually asking of maintainers?
15:56:43 <shyam> ok before we go to the end, any immidieate suggestion to gain eyeballs? Reduce mail traffic in maintainers list (from the CI infra maybe) to gain eyeballs on the other stuff?
15:56:46 <vbellur> let us use an executive order since that seems to be the way things get done these days ;)
15:56:58 <kkeithley> how droll
15:57:15 <vbellur> shyam: we could do that
15:57:35 <vbellur> rtalur sent an email about frequently failing tests
15:57:54 <vbellur> shall we start by imposing some deadlines to fix those tests?
15:58:03 <shyam> Ok, (1) Release 3.10 related mails and actions gain importance (of course people can call out if the release owners are being insane about expectations)
15:58:08 <kshlm> Immediate action, get somebody in Bangalore to go knocking on everyones door.
15:58:17 <shyam> kshlm: yes, that!
15:58:19 <jdarcy> How about if we finalize the "top N" failing regression tests, identify responsible parties, and set a deadline to fix by the next meeting?
15:58:34 <kshlm> (That's where I assume most of the current problems arise from).
15:58:35 <vbellur> jdarcy: +1 to that
15:58:50 <shyam> Folks! :) the reviews and release-notes please, we do need to keep a timeline there
15:59:10 <shyam> (or what's left of it)
15:59:18 <vbellur> shyam: as maintainer of release-3.10, go ahead and propose your deadlines
15:59:20 <jdarcy> Consequences: absent a good reason, those who fail to meet the deadline can no longer commit or have own patches reviewed.
15:59:42 <shyam> My deadlines ATM are pretty simple, tomorrow
15:59:49 <shyam> As we are on a day-to-day slip
16:00:01 <vbellur> shyam: go ahead and send out that note
16:00:08 <kkeithley> if you want someone to review your change, make sure to add them as a reviewer in gerrit
16:00:17 <vbellur> I think we have 2 sets of actions here:
16:00:24 <vbellur> 1. deadlines related to 3.10
16:00:25 <shyam> will do
16:00:27 <kshlm> As the release-maintainer from BLR, I nominate rastar_away to do the legwork.
16:00:42 <vbellur> 2. addressing regression tests
16:01:21 <kshlm> vbellur, That's right.
16:01:36 <kshlm> We've decided on 3.10.
16:01:36 <vbellur> let us work on these aspects and review in the maintainers' meeting next week
16:02:16 <shyam> Hope we get a healthy attendance in that meeting
16:02:42 <kshlm> shyam, Do you have a actionable todo out of this?
16:03:11 <kshlm> I'll end the meeting after this.
16:03:20 <kkeithley> btw, the other day amye said that 3.10 had already slipped one week (from the original 14 Feb date).  Is that accurate?
16:03:29 <shyam> I think so, at least a few more nags and consequences are going out and we will get some action going.
16:03:32 <shyam> yes
16:03:43 <kshlm> #action shyam will send out a note on 3.10 deadlines and get someone in BLR to nag in person.
16:03:52 <kshlm> Thanks shyam.
16:04:04 <kshlm> Okay, before ending the meeting just a few announcements.
16:04:05 <shyam> yes kkeithley, we have not started testing or building beta1 RPMs yet (updated the md page with the same)
16:04:24 <kshlm> 3.7.20 has been released and I hope that it's the last of the 3.7 releases.
16:04:36 <kkeithley> okay, thanks, just wanted to confirm what she said
16:04:53 <kshlm> And we have a new GD2 release out. You can actually mount (provided you have specific brick paths) using the mount command.
16:05:19 <kshlm> A few of us will be around at FOSDEM at the Gluster stand and the SDS devroom.
16:05:27 <kshlm> Oh before I forget.
16:05:37 <kshlm> Volunteers for hosting the next meeting?
16:05:59 <kshlm> The next meeting will be on the 15th.
16:06:23 <kshlm> I'm it I guess.
16:06:36 <kshlm> Okay. See you all in the next meeting.
16:06:54 <vbellur> kshlm: thanks!
16:06:54 <kshlm> #endmeeting