2.18 Goals
MattyT
matty at chariot.net.au
Thu Sep 12 07:26:28 UTC 2002
I suppose now is a good time to take a high level overview of where we
are at for 2.18 and where we might be going. As such I'll list the
projects we're all working on and the status as I see them. Apologies
if your really important project is missing, I did my best.
We should take this discussion, work out what are 2.18 hard goals, 2.18
soft goals, and 2.20 hard goals and update the roadmap as such.
Administration Rewrite (Administration Templatisation + Taint)
--------------------------------------------------------------
Bug #97706 Responsible: MattyT
This arose when I attempted to develop customisation resolutions by
copying editkeywords.cgi to editresolutions.cgi. I quickly discovered
there was substantial potential for code reuse, and set out to rewrite
the code in a reusable way. As such, the administration rewrite was
born.
The rewrite features sortkeys and inactivity for everything, and has a
schema requirement of (id, name, description, isactive, sortkey). It
does not yet support product-specific fields (but probably will once I
get up to the point of needing it to), nor does it support the more
varied needs of editgroups and editusers (it might, it might never do,
or I might make my administration system rely on a more general
administration system, having a three-tiered system for these CGIs).
Originally planned to be checked in along with customised resolutions, I
decided it would be a simpler customised resolution patch to rewrite
keywords first, giving them sortkeys and inactivity in the process.
This is bug #69267. This work is almost ready.
Another part of this is the moving of the localconfig enumerations into
the database, and giving them descriptions, sortkeys, isactive flags and
IDs, also preventing the enumeration deletion problem. This will
probably be the second part, and is bug #146104.
Then I will start thinking about product-specific fields.
The administration rewrite is fully templatised and taint mode enabled.
Hence it is a part of the templatisation + taint mode of the
administration CGIs. If we don't convert all the CGIs over for 2.18, we
will need to convert them over maintaining the old system. This is
possibly the best plan for editgroups and editusers, but we will know
better towards the end of the milestone and can wait until then.
This doesn't need to be a hard 2.18 goal, but getting as much as
possible in now will save T + TM work later.
Back End Separation
-------------------
Bug #162613 Responsible: No One/Multiple?
Touted as more as a 2.20 goal than a 2.18 goal, nevertheless the recent
massacre of CGI.pl and globals.pl has probably helped this a bit.
Bug.pm remains to be updated and used however, and we're probably still
a long way from finishing this and hence getting a nice testing suite
happening.
Zach indicated he may well be doing this for the new installation
system, that would be a very useful place to have back end separation.
Bugmail Rewrite
---------------
Various Bugs Inc. Bug #26943, #125888, #97956, #113688, #93952
Responsible: MattyT/timeless?
The mooted rewrite of processmail to include some nice new features such
as bulk change single notifications, and to be more sane.
No work has been done to my knowledge, and this is unlikely to be a 2.18
feature. However, it might be worth writing the initial bugmail
template to support these features, even if processmail does not yet use
them. This would prevent admins from having to change their templates
later.
bbaetz's modularisation of processmail (bug #124174, #140782) also
probably comes under this heading. This will help windows and probably
be done for 2.18.
Configurable Access Policies
----------------------------
Bug #110947 Responsible: No One
Touted as more of a 2.20 goal, it can probably happily stay there. The
aim is to let administrators define what access different users have to
bugs, rather than using a hardcoded policy.
It needs to be product by product, and I think a good implementation
would be to define policy records and let products refer to them, so
they can be shared between products.
This would involve removing the hardcoded system role "editbugs". It
would become a normal group on existing installations and will possibly
remain a default group for new installations.
It is blocked on the new group system.
Customised Email Reports
------------------------
Bug #6679 Responsible: No One
Pfft, nothing has been done on customising whinemail to be more useful.
Customised Fields
-----------------
Bug #91037 Responsible: Non Core
This one has been delayed for a while and really needs to be picked up
and shepherded by a core developer.
I believe it isn't "customised fields" but rather "customised
enumeration type fields". Luckily a lot of fields are simply
enumeration type (OS, platform, severity, priority, version, milestone),
as opposed to bug reference type (dependencies), text (whiteboard,
summary, URL), account (QA, assignee, CC), complicated enumeration type
(status, resolution), keywords type, allocation type (votes), or other
(bug #, alias).
Once this is in, we can start thinking about unhardcoding our
enumeration type fields, so they can be removed. We can also think
about customising other types of field.
I believe multi-select is supported, but don't know the status of
templatisation or taint mode is here. This is part of why it's
important a core developer gets involved - this is a major patch and we
need to make this this fits in with where we are heading.
A big issue with this is how they interact with templates. Requiring
admins to edit templates when fields are added or removed, is, I think,
unacceptable long term. I believe we can make our templates adjust to
new fields, given there's a fixed set of field _types_.
Getting the current patch in has I believe been touted as a hard 2.18
goal, apparently too many installations require it to upgrade, as it
would be too much work to keep redoing their installation hacks.
Customised Graphs
-----------------
Bug #105491 Responsible: Gerv
Rewrite of graphing so that you can graph anything (but not
retrospectively). This also involves moving the graphing parameters and
results into the database.
This can't be checked in before customised resolutions because it
expects resolution IDs, but they only arrive with customised
resolutions. Unfortunately, customised resolutions can't be checked in
before customised graphs either, so they need to go in at the same time.
We could split these apart, but one of them would need transitional code
that would be going away. Normally this would be OK, but it involves
schema changes.
This might need to handle stuff that isn't ID based, probably at least
customised statuses won't be for 2.18, although we could use fake-IDs
that aren't (yet) foreign keys.
Customised Reports
------------------
Bug #12282 Responsible: Gerv
Rewrite of reports.cgi to allow general bug count tables to be
generated, much like query.cgi.
The existing CGI was always bmo hardcoded and never worked properly. I
think Gerv is dropping Most Doomed and Most Recently Doomed, seemingly
even bmo doesn't use them as no one complained they didnt even work on
bmo.
Gerv just attached his first cut, and this is direly needed, it should
be a hard 2.18 goal.
I don't know whether he's planning to support stored reports for 2.18,
but this should probably be a subsequent patch since it might require
refactoring stored query code.
I have plans to support customised indentation type reports, which are
basically like query.cgi but grouped together. They would have scope
for group counts and sums, and also with the ability to only show the
top X within a group, but not for 2.18.
Another possibility not yet being explored is also attachment and/or
vote counts, so for example you could see who attached the most.
Customised Resolutions
----------------------
Bug #95434 Responsible: MattyT
This aims to allow you to define your own resolutions. There are 4
classes of resolution - { successful, unsuccessful, moved, duplicate }.
Note you can define more than one dupe and moved resolution. It also
adds inactive resolutions and sortkeys as it uses the new admin system.
It is a bit bitrotten, but this is not a major problem, the blockages
are however.
This cannot be checked in before customised graphs, since renaming
resolutions would break graphing, as the old system uses name references
outside of the database. Unfortunately it also cannot be checked in
after customised graphs, so they must be checked in together.
This also possibly cannot be checked in before quicksearch is rewritten
in perl, because the current quicksearch uses a hardcoded file called
localconfig.js which the admin needs to update manually (but this is
possibly not a problem because there is currently the same issue with
keyword renaming I believe).
Along with this, I have changed the resolution set for new installations
to { FIXED, INVALID, FEATURENOTBUG, NOTWORTHIT, DUPLICATE, WORKSFORME,
MOVED, MISSING, CAREFACTORZERO }. This is aimed at demonstrating
possibilities, not being the final set of any installation, and is
obviously not going to be the updated set of BMO once this gets in.
Also REMIND and LATER are gone for new installs.
This should be a 2.18 goal, we have waited long enough, and bmo wants
this too.
Customised Statuses
-------------------
Bug #101179 Responsible: MattyT
Not yet started, however will use the new administration system and
likely some code and lessons gained from customised resolutions.
It's probably more complicated than resolutions as you need to handle
status transitions as well, but hopefully there won't be all the stuff
blocking it that customised resolutions had.
Unlikely to make 2.18, but one can hope.
Email Interface Integration
---------------------------
Bug #94850 Responsible: Non Core?
Project to bring the email interface out of contrib and into mainline.
It would be nice if this could use a back end API rather than its own
SQL, which would mean that it would maintained better.
As such, I think this shouldn't be a 2.18 goal. We'll certainly take
improvements here, but I think that we really can't integrate and
maintain this until we have a back end it can use.
Group System Rewrite
--------------------
Bug # Responsible: Joel
The terminally delayed group system rewrite to get > 55 user groups.
The current system includes a new feature, group inheritance (groups
which are the union of other groups). There are a few arguments about
how best to determine user membership in groups. Different systems of
caching have been proposed, some faster, smaller or more flexible, but
none seems perfect. bbaetz doesn't think we even need a caching system,
at least yet.
I think this should be split off into a parity patch and the group
inheritance patch, allowing us to unblock bugs while giving us time to
work out what caching policy we should use.
At least the parity should be a hard 2.18, it is direly needed.
Localisation & Internationalisation
-----------------------------------
Bug #? Responsible: Gerv
I have no idea what Gerv is doing here, but I suppose we can claim we
have good I18N capability even if the implementation is horrid.
Long term we need to reconsider gettext or similiar, and also think
about whether we want field value names and descriptions to also be
I18Nable in the database.
I suppose the current plans for removing all user-text from the CGIs
should be a hard 2.18 goal.
I think other things can wait for future milestones.
Mail Templatisation
-------------------
Bug #125688, #148147 (? for token) Responsible: No One?
One part of the full templatisation effort requires we templatise our
mail output, including bugmail. I think we have already have a mail
template somewhere, but we need to take a high level decision how to
implement this.
One main issue is how to implement headers vs. body, either in the one
template or as two separate templates. How the mail transport rewrite
API works affects this to a degree, but I think most of us agree
separate templates are better.
The other main issue issue is how to handle escaping of high-ASCII and
so on in mail, we have bugs with this we should fix (eg bug #110692).
I'm not really familiar with mail escaping, but I understand we need to
escape headers differently from the body. Ideally this would not have
to be done in the template at all, but the mail CGIs would do it
automatically. These two things might dictate a separate-template
implementation.
Should be a hard 2.18 goal.
Mail Transport Rewrite
----------------------
Bug #84876, #65101 Responsible: JayPee/Gerv?
A rewrite of mail transport to make setup easier and to work in more
configurations. It will use some sort of Perl module, and should
therefore work under Windows, which the current system has problems with
I believe.
Will hopefully fix our sendmailnow problems?
Should be a hard 2.18 goal.
mod_perl Support
----------------
Bug #87406 Responsible: BBaetz?
This is a pretty substantial change to our architecture, in get the
performance benefits of running under Apache's mod_perl.
I don't think much has been done in the way of this so far, and I think
it should be a 2.20 goal, although new 2.18 work should attempt to be
mod_perl compliant.
New Install System
------------------
Bug #105854? Responsible: Zach
Zach's fabled new installation system. I know little about this, other
than it hopefully splits up the monolithic checksetup.pl and does some
front end/back end separation.
I also hope that it lets checksetup.pl use some of the reusable code we
have - there is currently code in checksetup.pl that also appears
elsewhere.
This seems to be stalled and I hear occasionally it's bitrotten - I
suggest it would be better to get this in via simple patches to avoid
this, ie start by moving bits of code into modules.
I don't think this should be a hard 2.18 goal, we can wait and see if it
happens.
Postgres Support
----------------
Bug #98304 Responsible: DKL
This is quite an undertaking unfortunately. Much of the PgSQL work in
under the banner of other projects.
Firstly, we need to remove all enumerations from the database as they
are MySQL-specific. This is taking place under the localconfig
enumerations part of the admin rewrite, under customised resolutions,
and under customised statuses.
It is the difficulty in getting the last in during 2.18 that may make
PgSQL support unable to arrive in 2.18. One transitional possibility is
simply to use a status_id field which has hardcoded meaning, and isn't
(yet) a foreign key.
Secondly, there is a need to handle the simple SQL dialectal issues, by
using ANSI SQL and by inserting subroutines to return the relevant
dialectical SQL. For example, regexps and date handling are different
between the databases.
Thirdly, the administration system needs to be changed to handle PgSQL.
PgSQL doesn't support some field types MySQL does, eg mediumtext I
think. One alternative is to use lowest common denominator types,
another is to be specific and emulate field types with the closest
possible field type.
Fourthly, PgSQL has a dramatically different locking & transaction
system, that is incompatible with the code we use now. I think bbaetz
is mooting a move to requiring MySQL transactions at some stage,
possibly requiring MySQL 4. This would be quite unlikely for 2.18
however, and would probably be 2.20 at the earliest.
It's probably possible to get some transitional code in which does the
appropriate transactioning when it discovers locks. This wouldn't solve
many of our lack-of-transaction problems, but it could get in for 2.18
and work under PgSQL. I'm not sure how compatible this is with bbaetz's
DBI plans, however.
Fifthly, I don't believe PgSQL supports groupset math, so the new group
system is also required.
I don't think this should be a hard 2.18 goal, but we should definitely
try hard here.
Product Group Rewrite
---------------------
Bug #147275 Responsible: Joel
The current product group system has major problems. Firstly, the
mechanism of finding product groups by name is not discoverable.
Secondly, it is inflexible, you can't share groups between products, and
you can't specify the entry parameter on a product by product basis.
Thirdly, you can't specify that a product group can never be removed.
This rewrite will solve these problems. The product group will be
defined on the product page, and can be any group. You can also specify
a different group for entry.
Further features include specifying the difference between a bug
restriction being by default, and being required and not able to be
removed, and also be not allowed.
This is all on a product by product basis. The exact features have not
been decided yet, but it will certainly be a quantum leap over what we
have.
This is blocked on the new group system.
I hope this will make 2.18, but don't think it should be a hard goal.
Shadow Database Rewrite
-----------------------
Bug #124589 Responsible: BBaetz
The current shadow database easily gets too out of sync, and can get
corrupted. This aims to rewrite the shadow database using MySQL
replication.
There will be a transition period on BMO so they can go back to the old
implementation if problems occur. But after this, the old
implementation will be dropped back into the pile of primordial sludge
it came from.
This will likely all occur before 2.18, unless someone yells. No one
has so far, as there were release notes of deprecation in 2.16 and
2.14.2 to this effect.
This is also a necessary condition for us to use DBI placeholders in
write SQL statements.
Note that PgSQL will not need a shadow database due to its MVCC
architecture, however there might be performance benefits regardless if
the shadow database is on another disk (I doubt anyone would want to do
this, however).
String Key Removal
------------------
Bug #? Responsible: MattyT/BBaetz?
We have long been using string keys for stuff such as product,
component, version and milestone.
String keys are slower and harder to rename (although the current code
does). As such, transitioning everything over to IDs is a good plan.
BBaetz has already done products and components, but has disclaimed
responibility for milestones and versions. These will probably be
picked up by me as a part of the administration rewrite.
There is probably no reason to make this a hard goal for 2.18, but it
will probably happen anyway.
System Role Rewrite
-------------------
Bug #147276 Responsible: Joel
Rather than having specific system groups, which is kludgy, Bugzilla
should have system roles that refer to groups.
Originally, these roles could be params, but eventually they will become
part of an access policy.
Not a 2.18 requirement.
Watching Rewrite
----------------
Bug #76794, #14137, #34787, #128839 Responsible: Jake/Myk?
People want to be able to watch products, components, keywords, etc.
Probably won't be 2.18, as much as I desperately want a Bugzilla product
watch.
--
Matthew Tuck: Software Developer & All-Round Nice Guy
My Short Autobiography: 1985 Grade Bin Monitor 1990 Class Clown Award
1992 Awarded Most Likely To Spontaneously Combust 1996 Crowned Galactic
Emperor 1998 Released From Smith Psychiatric Hospital
More information about the developers
mailing list