backporting minor UI changes to keep QA scripts working

[Gervase Markham]

David Miller wrote:
> 2) It's a slippery slope to go down.  Bugzilla is continually evolving,
> and this problem isn't going to go away.  The same problem will happen
> again with 2.24 because of UI changes.  Features are going to change,
> too.  Every version is going to break scripts that we used with the
> previous version...  How many branches do we continue to backport UI
> changes to?
> 
> Anyone have thoughts on the matter, in either direction?

It seems to me that the ideal thing is for the test scripts to freeze
along with the codebase. So we don't make any changes to them once the
code and UI are frozen, so they continue to match that code and that UI.

This means that, if you write a spiffy new test script to test something
that's never been tested before, you'll probably just write it for the
trunk. That's fine - Bugzilla's continually moving forward. And you can
use the time you would have spent backporting it to write another cool
test script.

That's the ideal. Now, I know that we are just getting going with test
scripts, and that we have to support 3 versions of Bugzilla and it would
be great to have test scripts to cover them all. And so, there's
pressure to change the stable versions so our initial test scripts can
have wide coverage. But frankly, why run newly-written test scripts
against 2.20? If you find bugs, are we going to fix them? Probably not,
because it's a stable branch.

What is the point of test scripts?
1) To prevent regressions during development before freeze
2) To prevent regressions during security fixing after freeze

1) doesn't apply to 2.20. OK, they might be useful for 2), but we've
done manual testing of security fixes for ages now, and it won't hurt us
to have to keep going a bit longer.

Gerv