Pre-Release Testing Guide - Help Needed

Tue Mar 29 15:54:49 UTC 2005

Does anyone in this list have any testing plans specific to Bugzilla
apart from those that are in the t/ directory?  I'm writing a
Pre-Release Testing Guide from scratch and implementing new automated
tests that really interact with the web page and DB in a smoketest
environment (not production but a full simulation of it).  Anything
anyone has would be helpful at this point.  I'm actually at the point of
writing the architecture spec. for the automated software doing the
testing, however, a large part of doing that work will go straight into
the pre-release testing guide.  Here's a quick sample of what's been
developed so far... (some tests are specific to AMD needs, however, for
the planned release version, they will not appear in the guide).

Availability

Name

Description

Pre/Main

Localconfig Vars

Examines $db_user, $db_name, $db_pass, and $::installation variables to
make sure they exist and are set to point back to the smoketest
environment. 

Pre/Main

Params Vars

Examines certain key variables in the data/params directory to make sure
they are set to point back to the smoketest environment. They are
cookiepath, insidergroup, installation, and urlbase.

Pre/Main

MySQL Connectivity

Connects to MySQL to determine whether or not the smoketest user,
password, and database have been installed. It does this by connecting
to the database, then collecting a count of the records in the 'bugs'
table. If that count is greater than zero, it's assumed that the
environment was properly loaded. Otherwise, it's assumed that the
environment was left unprepared for testing.

Pre/Main

No Stale Smoketest User

This test executes a query against the profiles table in the smoketest
database to determine if the specific smoketest user has been installed
in the working environment. If it is, this is considered a failure.

Pre/Main

Web Connect

Can the testing routines connect to the listed web address plus
testagent.cgi tacked on to the end. If it gets "OK" back, then success,
otherwise, failure.

Main

createaccount.cgi
  - Create Account submit button

Look for the "Create Account" submit button

Main

createaccount.cgi
  - Internal use footer

Look for the text "intended for internal use only" in the page.

Main

createaccount.cgi
  - Top

This checks for the header tag <link rel="Top"
href="http://something.amd.com/bugzilla/smoketest/test/">.

Main

createaccount.cgi
  - Create a new Bugzilla account

Looks for the text "Create a new Bugzilla account".

Main

createaccount.cgi
  - Intentionally not logged in

Make sure it appears that the user is not logged in by looking for the
text "Log out". If it exists, assume the system thinks the user is
logged in and fail. Otherwise, success.

None

Validate appropriate reaction to each CGI

Validate appropriate reaction to each CGI. Make sure the CGI responds
properly for a user that is logged in and one that is not. Make sure it
responds properly for admin vs. non-admin users.

None

Self-Create A User

Self-Create A User

None

Self-Update user's password

Self-Update user's password

None

Create a test bug

Create a test bug

None

Update the test bug's comments

Update the test bug's comments

None

Add a CC to the test bug.

None

Remove the CC from the test bug.

A significant amount of development needs to be done in order to make
the testing suite be an effective test of most of the functionality in
Bugzilla.  I've already written routines using LWP to POST data to the
web server simulating a user.  If I need to modify it to use GET
instead, that is a very simple change.  I'll need to set the routines up
to perform each of the tests as different types of users (not logged in,
an admin, a regular user, a user with editbugs, ...).

My main purpose in developing this documentation and the associated
automated testing software is that I want help with testing Bugzilla for
what I think I could have impacted so I don't miss anything.  Even if
it's a rare corner case (stand on your left leg, raise your right leg to
30 degrees, raise your right hand, - you get the idea), it usually isn't
going to get caught if nobody ever tries to develop software like this.

The net result of what I'm trying to do is develop software to reduce
the burden on developers and QA testers so that as a new feature is
implemented and/or developed, the automated testing suite will handle
making sure the vast majority of Bugzilla is functioning normally.  Over
time, this testing suite can only improve if it's kept up-to-date.

Of course, some things don't make sense to automate.  Tests like
comparing graphical images to data are in that category due to the time
involved in developing the software to read the graph and compare it to
the data it's supposed to represent.  GUI "look and feel" is also in
this category.  While the testing suite may be able to assist (as it's
designed to), the ultimate decision on make/break will still be up to
the operator using the tool.

---

Kevin Benton

Perl/Bugzilla Developer

Advanced Micro Devices

The opinions stated in this communication do not necessarily reflect the
view of Advanced Micro Devices and have not been reviewed by management.
This communication may contain sensitive and/or confidential and/or
proprietary information.  Distribution of such information is strictly
prohibited without prior consent of Advanced Micro Devices.  This
communication is for the intended recipient(s) only.  If you have
received this communication in error, please notify the sender, then
destroy any remaining copies of this communication.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.bugzilla.org/pipermail/developers/attachments/20050329/9b7ae713/attachment.html>