Release timing

[Nick Barnes]

At 2005-01-13 10:14:41+0000, Vlad Dascalu writes:
> Gervase Markham wrote:
> 
> > - Has it been around six months since our last release? No :-)
> 
> We never said that releases should be 6 months apart. We only talked 
> about freezes.

I thought we *did* say that releases should be 6 months apart.
Archives, anyone?

There are two quite separate aspects to this.  

1. The customer view

The first aspect is the view of Bugzilla 'customers'.  They like a
steady stream of releases; "every six months" is good for them.  This
relates to evolutionary development models: if you release often, you
get good feedback, can adjust your model of customer requirements, and
end up with a better product (i.e. one which better fits
requirements).  Unsurprisingly, customers like this, and greatly
prefer it to big-bang models (one release after five years, then fold
the project) or to when-its-ready models (which, more-or-less,
Bugzilla has been following until now: release 2.x will be when
features X, Y, and Z are complete).  Predictable release dates are
also good for customer relations because they allow customers to plan
their upgrade path.  This should not be underestimated for large sites
which have a lot invested in templates, customizations, integrations,
plugins, administrative procedures, training of admins and users, and
so on.

Depending on the frequency of regular releases, some customers may
choose to skip a release.  Few will actually *object* to more frequent
releases, as long as they remain regular, as they like being given the
choice, to balance the cost of an upgrade against the benefit of new
features or bug fixes.  However, there is a frequency, a point of
diminishing returns, at which too few customers actually take up each
new release (you need some customers to take up the new releases in
order to get good feedback, which is the whole point of the
evolutionary model).  The actual best frequency will depend on the
nature of your software, the number of your customers, and the cost of
making new releases.  For some projects, this frequency may be
monthly, weekly, or even more often (how often is the BBC News website
"re-released"?).  I strongly suspect that Bugzilla customers would be
very happy with a six-month release cycle.  They are demonstrably
unhappy with a 2-to-3-year cycle, and the consensus here is that a
3-month cycle is too short.

The notion of a "freeze" is totally meaningless to customers (although
problems in change management processes, which freezes are intended to
solve, do have effects on customers, in reducing the responsiveness of
development).

2. The developer view

The second aspect is entirely internal to the Bugzilla development
process.  With the current process, the trunk is frozen for the
"stabilization period" between rc1 (at the branch point) and .0, while
we fix bugs on the branch to make it acceptable as .0.  The length of
this freeze depends on three things.

1. how bad the branch point is, 
2. how quickly we fix it, and 
3. how good it needs to be to become .0.

How bad the branch point is depends on:

1.1. how bad the previous branch point was, and

1.2. how long ago that was.  If the previous branch point was fairly
     recent, we haven't had much chance to mess up the trunk.
     Furthermore, this is not a linear process (see 1.3).

1.3. the net defect introduction rate, i.e. how quickly we mess it up.
     This depends on development practices and also on defect
     discovery (i.e. testing, for Bugzilla), which depends in turn on
     the active users of the trunk.  For instance, there's a plausible
     belief that it depends quite sensitively on how far b.m.o is from
     the trunk (because b.m.o finds bugs and motivates us to fix
     them).  This also depends on how long ago the previous branch
     was, as bug fixes (negative defect introductions) are usually
     applied both to a branch and to the trunk, when the branch is
     recent.  So there's a non-linear process here.

How quickly we fix the branch (the net defect removal rate) depends
on:

2.1. the developers available;

2.2. the motivation of those developers to make it to .0;

2.3. the defect discovery rate (for Bugzilla, this translates as
     testing).

How good the branch needs to be to become .0 depends on:

3.1. how well tested it is, and

3.2. a bunch of other factors, which this message is already too long
     to contain.

Now obviously a long freeze is a bad thing.  Consider it a just
punishment for introducing so many defects into the trunk!  But the
purpose of the above analysis is to show that a long freeze is a
*consequence* of a long release cycle.  It's a non-linear effect (see
1.3 above): if you double your cycle time then you may triple or
quadruple your freeze time.  If you multiply your cycle time by, say,
five (from six months to 2.5 years), then who knows what happens to
the freeze time.

Given a short release cycle, freezes will shorten.  Given a six-month
cycle, I would be surprised if we get freezes of more than about six
weeks.  It might be much less.

How does one move forwards from this observation?  I recommend
planning a six-month release cycle, with two-month freezes.  Announce
a plan like this:

2005-06-01 branch for 2.20, release 2.20rc1, freeze trunk.

2005-07-31 release 2.20.0, thaw trunk.

2005-12-01 branch for 2.22, release 2.22rc1, freeze trunk.

2006-01-31 release 2.22.0, thaw trunk.

Do not set any dates more than two releases ahead.  Sketch a feature
plan for 2.20 and 2.22 based on this timescale, but say that if a
feature is not done at freeze time then it gets yanked.  It's good to
be firm about this, and be prepared to *yank a new feature out of the
trunk* at freeze time, *or even during the freeze*, if it's too buggy
to fix in time.  And if you get a short freeze (i.e. if .0 is ready
*before* the planned release date) then bring the release forward and
you get bonus thaw time.

I promised myself I'd get some paid work done today.

Nick B