Results of Community Research

Tue Jan 19 12:38:07 UTC 2010

Hi Max,

Thanks for doing all this work ;-) I have a few questions and queries,
but that shouldn't undermine the fact that this is really useful data
gathering.

On 19/01/10 02:13, Max Kanat-Alexander wrote:
> 	* Freezes kill the community--every time we had a code freeze, the
> number of contributors would dramatically drop and then recover slowly.

Is this, at least partially, a statistical artifact of the fact that you
are measuring continuity as "contributes every month"? After all, a
freeze means fewer patches, so there may be more people who consider
themselves part of the community (and may have been attaching patches to
bugs to wait for when the tree reopens) but who did not actually check
in or provide a patch for the trunk because it was frozen.

Would it be worth instead doing some sort of three-month rolling average
(if you've contributed in month X, or X+1, or X-1, you get a tick for
month X)? Or perhaps look at attached patches in Bugzilla as well as
checkin data?

Did you find it was common for someone to leave the community and
return? If so, it might be an indication that your data isn't smoothed
enough in this way.

> 	* Despite this, the size of the community generally increased until the
> start of 2005, and then leveled off through 2005, and then started to
> slowly decrease after that. No matter how I reorganized the data, or how
> I analyzed it, 2005 was the critical year.

Is the raw data available to look at? What are the absolute numbers
here? Was the peak community size 4, or 40? Is the addition or removal
of a single contributor a significant factor in the numbers?

> 	1) The "review" and "review-requests" mailing lists were split at the
> start of 2005. Before, all reviewers had at least seen every single
> review request. Now they had to subscribe to review-requests and
> actually *request* all that spam, which nobody was going to do. (I
> didn't even know about the review-requests list by the time I became a
> reviewer, which was after this point.) I think that this particularly
> hurt people's ability to get review "from the wind", since only old-time
> and very dedicated reviewers would have been on the list where those
> requests went.

I didn't even realise there was a review-requests mailing list. Or, if I
ever knew it, I've forgotten.

> 	I'd like to note here that nobody gave "the review process is too hard"
> as a reason for leaving. Some people here and there have banged the drum
> for a lowering of quality as an incentive to attract and retain
> community members, and there was absolutely no evidence in the survey
> that that would improve the situation at all.

As someone who might be interpreted as banging this drum, I think your
summary of the view (or, at least, my view :-) is not quite correct.

Firstly, I am not arguing for "lower quality". As you know, quality is
not a single number - "this patch has a quality of 45". And I have no
issues with someone being asked to change a patch at a code level or to
conform to style guidelines. But on different occasions, I have argued for:

- The acceptance of a patch which provides partial, useful,
functionality in pursuit of a larger goal, rather than waiting for the
submitter to implement the entire feature as envisaged by the team (but
not by the submitter)

- The acceptance of a patch which provides useful functionality, even if
the code would eventually be obsoleted by a rewrite of the feature which
is wanted but un-resourced and un-scheduled

- Within limits, the acceptance of a patch which solves a problem for an
existing user or set of users, even if the Bugzilla team don't
particularly like the way it's done, if the team has no plans to solve
that problem themselves 'the better way'. (The problem here is that
often the problem ends up lingering unsolved for years, to nobody's
benefit. This is what is generally known as "stop energy".)

Taking any patch is a trade-off between the plus of the problem solved
and the contributor encouraged, against the minus that the patch may not
be absolutely 100% as it would be if the reviewer had written it. We
need to consider whether we are weighing the second of those two factors
lightly enough.

As I understand it, your research was looking at people who became
community members and then stopped. My contention is that people who
have their patches rejected for reasons related to one of the above
never _become_ community members - and therefore would not have been
reached by your survey. (You talk more about initial acceptance below,
and that's great.) Therefore, I think your conclusion that the above
sorts of thing are not a factor may not actually be supportable by your
gathered evidence.

> 	Both of the former documentors who responded mentioned the lack of
> kudos from the community as one reason that they weren't motivated to
> stay in the community. 

I'm a "former documenter", in that I used to work on the Bugzilla
documentation and now no longer do. My reasons for stopping were:

- The documentation build system was byzantine in its complexity, and
  every time I got a new machine (or even when I didn't) it needed
  setting up.

- The XML markup gave the ability to produce text, HTML and PDF, but it
  made it very difficult to do things with graphics or layout in order
  to make the documentation more readable. I had to remember an XML
  vocabulary not used anywhere else in my online life.

- Documentation was treated like code, with patches having to be
  prepared and reviewed, rather than like a wiki. Getting fixes in,
  particularly rewrites and reorganizations, was just too much effort.

I've got a fairly thick skin, so the fact that a big documentation
reorganization was met with process-based obstructions rather than a
"wow, thanks for putting in all that work" didn't affect me too much :-)

> personal involvements. But nothing could be further from the truth--the
> personal interactions that people have with community members, the
> amount they feel appreciated, and the amount they feel assaulted, are
> actually the MOST IMPORTANT ASPECTS OF RETAINING COMMUNITY MEMBERS.

Have you read "The Art of Community" by Jono Bacon, recently published?
It's fantastic on this sort of thing - well worth a read. I read it
recently, cover to cover.

Also, if you are around at OSCON time, he runs the Community Leadership
Summit:
http://www.facebook.com/event.php?eid=186487039968
Anyone in the Bay Area at the time should consider attending.

> 	1) Extremely rapid response to review requests. No matter how much
> review a contributor has to go through, they usually will go through it
> as long as there aren't LONG DELAYS between the request and the response.

Definitely. On the main Mozilla side, I've been trying for years to get
a system in place which tracks this across the project and nags the
appropriate people. The trick, in that case, is defining "appropriate
people".

> 	Some people are hopeless programmers, and they're never going to get
> better no matter how much you help them or how many reviews you do. For
> these people, you don't have to imagine nice things to say about them--I
> don't expect anybody to ever make up stuff that isn't true, just to be
> nice. If somebody has demonstrated conclusively that they're never going
> to get any better, or that they're just going to cause trouble one way
> or another no matter what we do, the best thing you can do is simply
> personally choose to ignore them, which is a personal decision that you
> can make for yourself.

I don't actually agree with that. I think the best thing you can do is
tell them why their contributions aren't being accepted, and say that
unfortunately you will not be able to make time in future to review
their contributions. This is possibly (although by no means certainly)
more likely to result in hurt feelings than just ignoring them, but it
is also much more likely that their life will improve as they go and do
something they are actually good at. You certainly don't have to be rude
or brusque when telling them, but I think you are doing them greater
good by telling them.

Also, bad coders might make great support people, or documenters (if
your documentation system isn't the functional and process equivalent of
writing code ;-).

> 	Look, I understand that sometimes coding and working on a collaborative
> project with people who have different viewpoints can get REALLY
> frustrating! I've been an offender in this area just as much as anybody
> has been! But it's really not OK for us to insult other developers
> personally just because we're frustrated with them personally.

I agree with all of what you write. Have you noticed any of this? This
isn't acceptable in a more broader Mozilla context, as well as in a
Bugzilla one, and I take it seriously when I find it.

> 	Well, here's my recommendations for how we handle the review problem:
> 
> 	1. I personally have started to prioritize review requests from
> non-core contributors above anything else, including security work,
> releases, ANYTHING. New contributors are my #1 priority, and I've
> already seen this have a big effect on the number of contributions we're
> getting from outside developers.

Fantastic! :-)

> 	2. I think we should say that if you're a reviewer, it's your
> responsibility to receive the review-requests emails, and we should
> merge review-requests back into review at . 

Sounds absolutely fine to me.

Gerv
_______________________________________________
dev-apps-bugzilla mailing list
dev-apps-bugzilla at lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-apps-bugzilla