Custom fields and more (was Re: 2.16.5/2.17.7 release prep)

[Sean McAfee]

Gervase Markham <gerv at mozilla.org> wrote:
>Steven Suson wrote:
>>    What is the status of Custom Fields (`a la the previously available 
>> Custom Fields patch)? Will this be making it into a Stable or 
>> Development version soon? For the record, it certainly gets my vote! ;-)
>
>It still has the same status - as an unofficial patch. Concerns have 
>been raised about its architecture; it's quite possible than any custom 
>fields function Bugzilla acquires in the future will work quite differently.
>
>No-one is working on custom fields as far as I know.

I still am!

I'm sorry I haven't been more active on the developers' list lately,
but I've been working nonstop on getting people at my workplace off of
Teamshare and onto Bugzilla.  One department has been using Bugzilla
exclusively for a few months now, and has been satisfied, by and large.
Other departments are set to follow in the near future.  The flood of
bug fixes and feature requests has just recently slowed to the point that
I have some spare time to work on getting my new stuff merged back into
the main code base--pending input and approval from you folks, of course.
I've actually implemented a large number of additional features beyond
the basic custom field functionality I originally proposed, all of which
I'd like to make available to the developer community.  So, starting
from the top...

Firstly, there are the custom fields themselves.  They still work
essentially as originally described, and come in several varieties.

*  integer
*  date, with or without time
*  short string (255 characters or less; represented on web pages as
      an <input type="text"> element)
*  long string (64K characters or less; represented on web pages as
      a <textarea> element)
*  selection, single or multiple

Each field has a name that must be unique among all custom fields in
the installation.  These names follow the usual rules for C-like
identifiers.  Each field also has a more human-readable "display name"
used for HTML display which need not be unique.

Fields may belong to any number of products.  I abandoned my original
"field groups" idea, which never had much support and which subsequent
experience showed to be rather unwieldy.

Although I originally offered an administrative web interface to
manage custom fields, further development rendered it obsolete and I
have not created a replacement.  Instead, I have a number of small
command-line tools that serve roughly the same purpose.  I do need to
resort to the mysql client on occasion.

I'm particularly proud of my new bug-query implementation.  Queries
can consist of arbitrarily complicated Boolean expressions involving
AND and OR.  Individual terms can refer to stock bug fields (ie,
columns in the BUGS table), custom fields, or the special terms "any
note", "any cc", and "any commenter".  String fields can be tested
using the usual Bugzilla operators (exact, regexp, allwordssubstr,
etc) as well as "in", "not in", and "is null".  Integer and date
fields can be tested with arithmetic comparisons (<, >, etc) as well
as "in", "not in", and "is null".  Selection custom fields can be
tested with a few set operations--"any", "all", and "none"--as well as
by treating them as strings.  The expression "selection_field
string_op value" means "'label string_op value' is true for at least
one of selection_field's labels".

The results returned by a query are highly configurable.  One can
specify that some, all, or none of the custom fields associated with
each bug in the result set should be returned.  (I added the "some"
feature recently and discovered that I could speed up a query by more
than a factor of two by only returning only the two custom fields (out
of dozens) I was interested in.)  One can request that all associated
notes be returned with each bug, or not.  Likewise, one can request
that all attachments be returned with each bug, or not.  Since
fetching large attachments can be time-consuming, I also implemented a
"deferred attachment fetch" feature.  If active, only attachment IDs
are returned with each bug; attachments of interest can then be
fetched on demand.  (I don't know how widely useful this might be, but
it's a big time-saver for certain operations here.)

In addition to my work on custom fields, I've implemented a
considerable number of additional features in response to demands from
groups here at my workplace.

The most prominent additional feature is the workflow.  A workflow is
a per-product directed graph describing paths along which a bug may
move.  Nodes in the graph correspond to a bug's "state".  Each arc
pointing outwards from a state describes one of the valid transitions
a bug in that state may make.  A bug's state is described by one of
its custom fields, which is dubbed its "state field", and which must
be of the single-selection variety.  I've eliminated the "knob"
portion of the bug/edit template ("Leave as...", "Confirm bug...",
"Resolve bug...", etc).  In its place, I introduced a row of buttons
above the "Attachments" portion of the page.  Each button is labelled
with the name of one of the defined transitions away from the bug's
current state.  Pressing one of the buttons initiates the transition
of the same name.  After filling out any fields, clicking a 'Finish
Transition 'whatever'" button updates those fields and changes the
bug's state field to the new state.  A generic "Update" transition is
always available which allows the user to change any aspect of a bug,
including its state.  For any transition (including the special-case
"bug creation" transition), one may define a subset of the bug's
custom fields which are to be displayed during that transition.
Furthermore, each field may be described as "read-only" (the field is
shown but may not be altered) or "required" (the field must be set to
a nonblank value for the transition to succeed).  The names of
read-only fields are colored red.  A transition may be tagged as
requiring an attachment; the user must supply an attachment during
that transition for the transition to succeed.

Transitions are logged.  In addition to new values for any custom
fields, the user who performed the transition is logged, as is the
date/time, the bug's new state, and the bug's new owner.  A summary of
the bug's history, including the user, date, new state, and new owner
for every past transition is shown on each bug page as the bug's
"change history".  This history is also available from the bug query
interface.

The owner of a bug can change automatically as the bug traverses its
workflow.  For each state, one can arrange that the bug's owner will
change to either a) the original bug submitter, b) a specific, fixed
user, or c) a user specified by one of the bug's custom fields.  This
automatic owner-changing is advisory only; a user can always override
the automatically-generated value.

I've leveraged GraphViz's "dot" program to produce visual workflow
graphs.  A link is provided on each bug page which generates a graph
of the bug's workflow, with the bug's current state shaded red.

I've introduced a programmatic bug creation API.  This has greatly
reduced the effort involved in porting incidents over from our
Teamshare database.

I've introduced a mechanism to allow for automatic setting of certain
custom fields at various points in a bug's lifetime.  This feature was
motivated by various users' desire to retain the numbering scheme of
incidents ported over from our Teamshare server.  One of the fields in
one of our products is called "Incident Number", the same as its
original Teamshare name.  Whenever a new bug is created in this
product, its "Incident Number" field is set to a number one greater
than the highest existing "Incident Number" field of all of the bugs
in that product.

I've introduced an alternate identifying scheme for bugs.  One can
define, on a per-product basis, one integer custom field that serves
as an alternate identity number.  In many contexts, one can specify
this number as an alternative to specifying a bug's Bugzilla bug ID.
Additionally, if a bug belongs to a product for which such an
alternative ID is defined, then that ID appears on the page generated
by show_bug.cgi in place of the bug's Bugzilla ID, optionally with an
alternate prefix (eg, "Foo Dept. Case #" instead of "Bug #").

Arbitrary linking between bugs is possible.  This is orthogonal to
Bugzilla's dependencies feature.  Each bug page includes a brief form
with which one can link the displayed bug to any other, one-way or
two-way.

I've added a "quick search" feature.  This is a short form that
appears in all page footers (as part of "Preset Queries", after "My
Bugs" and all per-user named searches).  One can select a product and
search term and supply a string to match against.  Submitting this
form takes one immediately to bug_list.cgi without needing to go
through query.cgi (which can take a nontrivial amount of time to load
when many custom fields are present).

Finally, and most recently, I've introduced a feature that mimics a
Teamshare feature called a "supplementary table".  One can tag a
single-selection custom field as taking its values not from a fixed
list of labels, but from the short_desc (ie, "Summary") fields of all
of the bugs in a different product.  When a bug is displayed, an icon
appears next to the value shown for such selection fields which, when
clicked, takes one to the bug in the other product whose summary is
the original bug's field's value.  Um, that was kind of convoluted,
but I think I got the point across.


Whew!  As you can see, I've been a busy little beaver.  I'm happy to
make any or all of the fruits of my labor available to the developer
community.


--Sean