Branch release problems

[Jason Remillard]

Hi,

David Miller wrote:
> Tonight we released a 2.18.2 release with a non-functioning query page.
> 
> I had to post an announcement telling people not to download it and wait
> for an update.  That's not a fun thing to do.  This has also happened
> before (branch releases going out with major regressions in them) with
> versions 2.16.9 and 2.14.5.  This is really unacceptable and it messes
> badly with the public perception of our credibility.
> 
> We really need some kind of better QA process on our branch releases.  I
> know it's hard because all of the developers run the cvs tip or the
> developer releases for their own installations, so nobody's familiar
> with the branches, but that's what we have landfill for, and we need to
> make sure things get tested before they go out.
> 
> At a minimum we need to put together some kind of smoke test document
> with lists of features to test for each branch, then we need to make
> sure that testing *everything* on that list is made part of the release
> process, before a tarball is rolled, no matter how few patches actually
> got checked in, or what they touched.
> 
> I'd welcome any other ideas anyone has, too.
> 

I wrote a set or perl tests scripts using the LWP modules, HTML::Form, 
HTML::LinkExtractor, HTML::Lint that are used to test codestriker out. 
It is pretty easy to test a web base application with perl. You write a 
proxy .pm for each page in bugzilla. The proxy uses LWP, and friends to 
manipulate the actual service. Your .t files then use the proxies to do 
the tests. We used to have a smoke test document. The smoke test is all 
automated now + a bunch of other tests that we were not doing before. 
Compared to the complexity of bugzilla, these tests would really not be 
that big of a deal to write. It is really easy to at least make sure all 
of the links are good, and the generated html is well formed.

The code is here.

http://cvs.sourceforge.net/viewcvs.py/codestriker/codestriker/test/

Once the infrastructure is in place, then you can insist that every new 
features get checked in with test cases. I know that there are plans to 
test the pm files inside of bugzilla directly, but you really can't 
escape the fact that it needs to work property at the 
   http/html level. I don't think doing both is unreasonable.

Not everything will be easy to automate, so a test document is also a 
good idea.

Thanks
Jason.