Partial custom fields design document

[MattyT]

On Fri, 2004-03-12 at 13:10, Sean McAfee wrote:

> It's been suggested that the best way to kick-start the process of getting
> custom fields functionality into the core is to start with a design document
> that all can agree on.  Makes sense to me.  Accordingly, I've started
> drawing up a document that describes the design of my latest custom fields
> patch.

I didn't read this in depth, so apologies if I missed something, but I
skimmed it, and saw what I believe to be issues I have.

Meta
----

While schema and programmatic interfaces are important, the details of
these are also uninteresting and not major.

We need a document that answers the major questions of how the system
hangs together, such as:

- how are fields added and removed?  what happens to the data on
removal?
- how are fields configured?
- how will custom fields interact with the templates - will custom
fields automatically appear without hacking?
- which existing fields will be able to become custom fields
eventually?  if some can't, why not?
- how does your system interact with I18N?  multi-language bugzillas and
the existing fields are going to need some I18N support, which must
entail some change in the way I18N works
- what are the types of fields, and what customisation can be done to
them?  eg if i wanted text fields to regexp restrictions, how would i
add this functionality?  what about more general restrictions? 
inter-field restrictions?
- general schema details - is the table metadata stored in the db?  is
there one or multiple multi-value fields per table?  are single-valued
fields implemented by a bugs table field?  how will the existing fields
be able to fit into this?
- are you going to be able to display multi-valued enumeration fields on
the bug list?  if so, how?

Also, there needs to be a rationale for the decisions, not just
details.  Otherwise everyone's going to be arguing about things you
might have already figured out.

Once we have all this, then we can move onto details.  After there's
some kind of consensus between this rabble, the details will likely be
easy in comparison.  And this is the biggest change that has ever
happened to Bugzilla, so doubly so.

So put an overall summary, then for each section, a summary, then the
details.  Mere mortals don't have the stomach to deal with all this
implementation detail at once, this will help us.

Details
-------

The idea of having all these generic tables (packing multiple fields
into one table) I find somewhat ugly.  Not only does it make a
dev's/admin's job of analysing a single field far more complicated, it
makes Bugzilla's job more complicated too - not only the SQL, but it
must either index on the field or a value, I think doing both is
difficult, thought probably possible.

The question here is whether you ever want to analyse more than one
field at once, and I'm pretty sure the answer is no for most field types
(the only obvious exception is relationship fields (for dep trees) which
I don't think you even defined, although they're not really fields in
some sense, since they're between bugs).

Given that you don't need this, I think each field should have its own
bugs field or separate table as per now.  I can understand the current
system doesn't require any schema changes, but I don't see that as a big
enough issue to warrant doing things this way.

Furthermore, I'd avoid putting "custom fields" on everything, since the
intention of any custom fields system should be to eventually subsume
almost all fields, making "custom fields" a rather useless description.

I originally had objections to adding and removing fields from the web
interface, but the more I think about it the less of a problem it
seems.  It's certainly a rare enough operation that it could be done
locally without major issues, however.

What I think we definitely do need, is to have modules controlling each
field (sub)type.  This will allow customisation through inheritance, and
hence will benefit administrators greatly, since they can plug in field
types.  Over time the field modules would gain more functionality and
hence more separate-file customisation capability.

We're never going to catch everything admins will want through a web
configuration system, but I think with bug field modules, we can get
99.9% through inheritance.

This being said, the intention is of course to make this all reasonably
easy.  Ideally you'd be able to avoid any Perl work here for simple
customisation, but there should be generous scope for Perl
customisation.

My Proposal
-----------

There's no need to do this all immediately as one patch.  But I'd want
to see us on this path.

A bug field module is placed in Bugzilla/BugFields/Field.pm.  This
contains all the code necessary for handling the field type, various
events to override, defines its configuration parameters, and so on.

It either directly inherits from a module for its field type (eg
enumeration, text, date, etc), or directly inherit from another abstract
module which inherits from this, providing some shared functionality.

The field modules are only templates, and you can instantiate them
multiple times - each time you would have a different config.

The field list (name and type) would be in the database because it's a
good idea to put the metadata with the data (eg if you copied the
database).  I don't have any objection to having a web interface here
because it allows us to gracefully handle field deletion with prompts,
and asking whether to remove the old field/table, which something like
localconfig is famous for not doing.

Each field module would define its configuration parameters and you
could then set it up on a field config page.  For a general text field,
it might allow configuring an acceptable regex, but for a more specific
one, it might be more appropriate hardcode the regex in Perl rather than
configure it.  The field configuration can be stored as name/value
strings on one table.

I would modify the templates to automatically display all the fields. 
It would know information such as the fields priority (first section,
second section, etc), type, and so on, and could use this information to
order the fields in a similar way they are now.

This is going to shift a variety of strings out of the template.  Two
possibilities come to mind about I18N.

The first is each field must have a template fragment defining these
strings, which can get used by whatever templates display this field.

The second places the strings elsewhere.  Most of the strings would be
in the field modules.  For general field modules, some of the strings
would be in the field configuration.

The first means that all I18N remains in one place, the templates.

The second however means that a module could come complete with I18N as
one file without needing a separate template, which is good for
pluggability.

The strings in the field configuration would need a separate I18N
capability for a multi-language installation, but that's an
administrator's task, not a translator's task.  This way an admin can
set all this up from one place.

I18N would be a matter of patching the module, instead of creating a
copy of the template which could get out of sync.

I would expect to see various types of fields eventually - enumeration
(eg OS), text (eg URL), date/time, allocation (eg votes), account (eg
reporter) and relationship (eg dependencies).

I would expect to leave multi-valued custom fields off of the bug list
for now, as we're likely to have subselects that can do this before we
get to hardcore porting of the old fields over.

-- 
         Matthew Tuck: Software Developer & All-Round Nice Guy        
 My Short Autobiography: 1985 Grade Bin Monitor 1990 Class Clown Award
1992 Awarded Most Likely To Spontaneously Combust 1996 Crowned Galactic
         Emperor 1998 Released From Smith Psychiatric Hospital