New custom fields proposal

[Jon Wilmoth]

Thoughts/question inline...

-----Original Message-----
From: Sean McAfee [mailto:etzwane at schwag.org] 
Sent: Friday, April 04, 2003 5:34 PM
To: developers at bugzilla.org
Subject: New custom fields proposal

Hi, folks--

Enclosed is my proposal for new custom field functionality to be added
to
Bugzilla.  Please read, discuss, criticize, etc., especially the "Open
questions".

Note that this is a requirements document, and describes only WHAT I'd
like
to see implemented, not HOW it is to be implemented.  The "how" will be
left
for a future document, when we've settled on the "what".

Also note that I've omitted any reference to how custom fields will be
administered.  This is obviously important, but I found myself getting
bogged down in the details, so I decided to just go ahead and send the
part
I'd completed.


--Sean McAfee

----------------------------------------------------------------------

Definition
==========

Currently, Bugzilla offers a fixed number of named fields to describe
bugs.  These fields are shared across every component in the system.
The proposed additional functionalty is the ability to define
additional fields on a per-component basis.  These additional fields
are called "custom fields".

<feedback>
I assume your talking about "component" being a broad system module
(i.e. search, creation, reporting, etc.).  I'd suggest you use a
different term so as not to be confused with a Product component.
</feedback>

Custom fields will be available in five major varieties: integer,
float, string, date, and selection.  The first four of these types are
collectively referred to as "scalar" types.  All scalar fields can be
set to a special "undefined" value in additional to the valid values
described below.

Integer
-------

An integer custom field contains values which are integers.  Such
fields must be able to store at least the range of integers which are
typically represented as a signed four-byte quantity.

[Open question: Should integer fields be available in signed and
unsigned varieties?]

Bugzilla should signal an error condition if a user attempts to store
an integer outside of the allowed range into an integer field.

[Open question: Should Bugzilla transparently offer large-integer
semantics?]

Bugzilla may emit client-side scripting code onto any pages that
request custom-field input that will abort form submission if an
integer field's submitted value is out of range.

Float
-----

A float custom fields contains values which are floating-point
numbers.  Such fields must be able to store at least the range of
floating-point numbers which are typically represented as an
eight-byte double-precision quantity.

When a user enters a text string which is intended to be stored in a
float field, it must match the usual pattern:

*  an optional leading plus sign ('+') or minus sign ('-');
*  an integer part consisting only of digits;
*  an fractional part matching the following pattern:
   +  a period character ('.');
   +  an optional sequence of digits;
   +  an optional exponent, expressed as an 'E' or 'e' character,
         followed by an optional plus sign ('+') or minus sign ('-'),
         followed by a sequence of digits
         
Either the integer part or the fractional part may be omitted, but not
both.

Bugzilla should signal an error condition if a user attempts to store
a floating-point number outside of the allowed range into a float
field.

String
------

This type of field contains human-readable text.

[Open question: Should Bugzilla reject attempts to store arbitrary
binary data in a string field?]

Each string field is flagged as belonging to one of two varieties:
short and long.  A short string field can hold a maximum of 255
characters, and may contain no vertical whitespace characters
(carriage return, newline, or vertical tab).  It is represented on
Bugzilla's Web interface using a single-line text form element.  A
long string field can hold large amounts of text, possibly consisting
of several lines.  It is represented on Bugzilla's Web interface as a
multiline textarea form element.

Date
----

A date field contains either a date (ie, a specific day of a specific
month in a specific year), a time of day (ie, an hour from 0-23
inclusive, and a minute and second, both from 0-59 inclusive), or
both.  Each date field is flagged as containing one of these three
subsets of data.

[Open question:  Is the "time of day" subtype really necessary?]

Bugzilla should signal an error condition if a user attempts to store
an illegal date value in a date field.  Examples:

*  An invalid day of the month: March 50, February 30
*  An invalid date: February 29, 2001
*  An invalid time: 12:66:00

Date fields are represented on Bugzilla's web interface using a pair
of drop-down listbox form elements (for the month and day) and a
single-line text-entry element (for the year) for the date portion of
the field's value (if appropriate), and a single-line text entry
widget for the time portion (if appropriate).  Bugzilla should signal
an error if an invalid date or time is submitted.

[Open question: Is this too busy?  Should a simple text-entry widget
suffice for all date types?]

<feedback>
IMHO, yes...it's too busy.  If it's a date field it would be more useful
to have a popup calendar (bottom of which could also include time
elements).  I would think a date field should have a input pattern that
could vary on a field by field basis.  This can be compared to java's
simple date format
(http://java.sun.com/j2se/1.4.1/docs/api/java/text/SimpleDateFormat.html
) class or oracle's to_date fuction that allow for a pattern (i.e.
MM/dd/yyyy HH:mm:ss aa zz to define a date input valid for 04/08/2003
16:25:30 pm PDT)
</feedback>


Selection
---------

A selection field takes its value from a set of distinct string
values.  No string value in any selection field's set of valid values
may contain vertical whitespace, contain leading or trailing
whitespace, or exceed 255 characters in length.

[Open question: Should there be a limit on the size of the set of
valid strings?  If so, should this limit be global or per-component?]

<feedback>
The main issue here is screen real-estate which is a global issue.
Having unlimited select lists can throw the html tables out of wack.
I'd suggest a global limit set to the largest possible value for the UI
where custom fields will be added.
</feedback>

Each selection field comes in one of two varieties: "single-selection"
and "multiselection".  A single-selection field's value is a subset of
the field's possible values of size either zero or one.  It is
represented on Bugzilla's web interface as a drop-down listbox form
element.  A multiselection field's value is an arbitrary subset of the
field's possible values.  It is represented on Bugzilla's web
interface as a multiline listbox in which multiple simultaneous
selections are allowed.

Each selection field's set of allowed values may be specified in one
of two ways.  A "local" selection field's values are defined by the
Bugzilla administrator, and are managed by Bugzilla itself.  A
"remote" selection field's values are taken from a table in a database
which need not be the same database used by Bugzilla.  The
administrator defines the remote database in the form of a DBI connect
string, a username and password with which to log in to the other
database, a table name, and a column name.  Bugzilla generates the set
of allowed values by logging in to the remote database using the
supplied username and password, and issuing the following query:

<feedback>
While the "remote" concept is cool, I think it should receive a lower
priority and perhaps even be treated as a separate release of "custom
fields".  There's a lot of work to do as is to manage "local" custom
fields.
</feedback>

SELECT DISTINCT source_column
FROM source_table
WHERE source_column IS NOT NULL;

Bugzilla should apply the following modifications to each element of
the result set, in the order given:

*  Delete all characters after and including the first vertical
   whitespace character, if any
*  Trim leading and trailing whitespace
*  Truncate to a maximum length of 255 characters

After this, duplicate elements should be eliminated, and the remaining
elements should be sorted alphabetically before the set is presented
to the user.

[Open question: Should other sorting criteria be supported?  If so,
how?]

Since such queries may be expensive, Bugzilla may cache the results
for up to one hour.

Since any given selection field's allowed values may change over time,
due to administrator action or variation in a remote data source,
Bugzilla should not signal an error if "invalid" field values are
found in the database.  Enforcement of a selection field's valid
values is to be done only on the client side.  If a bug containing a
selection field whose value is no longer valid is edited by the user,
the invalid field's value should be presented as being the null set.

It should be possible to share the definitions of selection fields
among multiple Bugzilla components, such that a change to one field's
data source is reflected in all other fields which share that source.

[Open question:  Should the field subtypes described above really be
their own top-level types?]


Logging
=======

Whenever a custom field's value is changed, this change should be
logged.  Log information should include the identity of the user who
made the change, the time the change took place, the field's old
value, and the field's new value.


Presentation
============

Custom fields are to be presented on each Bugzilla page that presents
the standard Bugzilla fields.  These pages include at least the
following:

On "show_bug.cgi" and "post_bug.cgi", the custom fields should be
shown just prior to the list of attachments.

On "enter_bug.cgi" and "long_list.cgi", the custom fields should be
shown just after the "Description" text box.

When a bug is being created or edited, custom fields are represented
as per the field type descriptions above.  When a bug is being shown
but not edited, scalar fields and single-selection fields are
represented as simple text; multiselection fields are represented
using disabled multiline listboxes.

The query.cgi page requires more extensive modifications.

Currently query.cgi knows only about Bugzilla's standard fields.
Since all components share the same set of standard fields, these
fields can be displayed before the user has chosen a Product and
Component at the top of the page.  To add the capability to search on
custom fields, a straightforward approach would be to require the user
to first specify the projects and components within which to search,
then display the usual query page augmented with the custom fields
which exist in those components.  However, this is too invasive for
components which don't use custom fields, and also for queries which
don't care about custom fields.

Here is an alternate approach.  A button is to be added to the query
page, to the right of the "Product", "Component", and "Version"
listboxes, labelled "Show custom fields for these components".  If the
user clicks it, the query form is redisplayed, augmented with all
custom fields in all of the selected components.  If no components
were selected, or if no selected component has any custom fields, the
page is simply redisplayed as it was before.

The user may freely change the target components and press the "Show
custom fields" button again.  The query page will be redisplayed with
new form elements for the most recently chosen components.

When custom fields are being shown, a second button is to be added
below the "Show custom fields" button, labelled "Hide custom fields".
When it is clicked, the query page is redisplayed as it was
originally, without any custom field elements.

It is an error for the user to select one or more target components,
show the custom fields for those components, change the target
components, and begin a search.  In this case Bugzilla should
redisplay the query page with an appropriate error message at the
top.

Different components may define different custom fields with the same
name.  Therefore, custom field form elements for custom fields
belonging to the same component will need to be grouped together,
clearly labelled as belonging to that component.

<feedback>
I'd like to see two levels of scoping global and by product for custom
fields.  Anything else is gravy.
</feedback>

Custom string fields may be queried in the same manner as the stock
fields "Comment" and "URL".  That is, they may be matched against a
user-supplied string using any of the following predicates:

*  contains all of the words/strings
*  contains any of the words/strings
*  contains the string
*  contains the string (exact case)
*  contains all of the words
*  contains any of the words
*  matches the regexp
*  doesn't match the regexp

Custom integer and float fields may be queried against a user-supplied
string (which must look like an integer or a floating-point number,
respectively) using the following predicates:

*  is equal to
*  is less than
*  is greater than
*  is less than or equal to
*  is greater than or equal to

Custom date fields may be queried using a group of selection elements
representing the year/month/year and/or the hour/minute/second (as
appropriate) using the following predicates:

*  is exactly
*  is at or earlier than
*  is at or later than

(The open versions of the latter two inequalities--ie, strictly
earlier than or strictly later than--are probably not worth including.)

Custom single-selection selection fields may be queried against a
multiline listbox containing all valid values for the field.  Multiple
selections within this listbox are allowed.  A bug matches this query
term if this field's value is equal to ANY of the user-selected
values.

Custom multiselection selection fields may be queried against a
multiline listbox containing all valid values for the fields.
Multiple selections within this listbox are allowed.  A pair of
radiobuttons is placed next to the listbox which present two choices:
"Any" and "All".  Initially, "Any" is selected.  If the "Any" button
is selected at the time the query is initiated, then a bug matches
this query term if any of the selected values are set in this field.
If the "All" button is selected instead, then a bug matches this query
term if all of the selected values are set in this field.


<feedback>
Security - Assign access to custom fields to groups.  Assignment would
hide fields from being displayed.

Validation - Required vs. not required.  In addition to the type
checking described above, a required flag should be included in the
definition of the field.

Presentation - Error messages, Labels and references to the custom field
names (i.e. contextual help pages) should allow for the displaying of a
system admin controlled term.

Consistency - It would be nice to have a uniform approach to field
definitions (security, validation, and presentation) for both stock and
custom fields alike.

</feedback>
----------------------------------------------------------------------

----
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=jwilmoth@starbucks.com>