Test coverage is 18.4%

[David Marshall]

On 2/15/10 7:19 AM, "Joel Peshkin" <bugreport at peshkin.net> wrote:

> 
> The 18.4% number seems to refer to the t/ tests.   I think, if we are
> serious about this, we need to start by creating a mechanism in the t/
> tests to have it use a test database.  So, you do the following...

I don't agree that a low coverage number from what has always been static
tests means that interest in testing is not serious.
> 
> perl Makefile.PL --db_server=localhost --db=bztest --db_user=foo
> --db_passwd=bar
> make test
> 
> and the tests create their own databases, preloading them with data, and
> run.
> 

I'd personally rather such tests existed in some separate directories from
the static tests.  But I agree that it would be useful to have dynamic
testing against a database.

I've wrestled with this at Yahoo!.  We do a lot of testing with a
LWP::UserAgent and a test database.  It wouldn't really be convenient to
start each test with the same database starting position.  I think sometimes
about creating a minimal database that I can initialize to a know state, run
the tests, and then look at how it has changed, but that may be overkill.
As it is now, for something like creating a new bug, we fill in the form,
submit it to get the new bug number, then go look around the database to
check that the attributes of the new bug are correct.


> Naturally, this also means that we need to figure out how to do the
> equivalent of the selenium tests in a way that won't break all of the
> tests every time a trivial change is made in a template.
> 

If the a change to a template is trivial, shouldn't the corresponding change
to the Selenium tests be equally non-trivial?  If modifying the tests to
reflect the change is non-trivial, how can the change itself be trivial?