Branch release problems

[David Miller]

Tonight we released a 2.18.2 release with a non-functioning query page.

I had to post an announcement telling people not to download it and wait
for an update.  That's not a fun thing to do.  This has also happened
before (branch releases going out with major regressions in them) with
versions 2.16.9 and 2.14.5.  This is really unacceptable and it messes
badly with the public perception of our credibility.

We really need some kind of better QA process on our branch releases.  I
know it's hard because all of the developers run the cvs tip or the
developer releases for their own installations, so nobody's familiar
with the branches, but that's what we have landfill for, and we need to
make sure things get tested before they go out.

At a minimum we need to put together some kind of smoke test document
with lists of features to test for each branch, then we need to make
sure that testing *everything* on that list is made part of the release
process, before a tarball is rolled, no matter how few patches actually
got checked in, or what they touched.

I'd welcome any other ideas anyone has, too.

We've also got almost zero actual QA since MattyT got busy with real
life.  It'd probably be a cool thing to get some people to actually do
QA type things for us again.  Bugzilla is growing by leaps and bounds,
and we definitely have a big enough product to badly need a whole QA
team, let alone the one-man-show that MattyT used to run.

-- 
Dave Miller                                   http://www.justdave.net/
System Administrator, Mozilla Foundation       http://www.mozilla.org/
Project Leader, Bugzilla Bug Tracking System  http://www.bugzilla.org/