The Problems of Perl: The Future of Bugzilla

[Max Kanat-Alexander]

  [ Note: This entire post is also available in HTML on my blog:
http://avatraxiom.livejournal.com/58084.html ]

  Once upon a time, Bugzilla was an internal application at Netscape,
written in TCL. When it was open-sourced in 1998, Terry (the original
programmer), decided to re-write Bugzilla in Perl. My understanding is
that he re-wrote it in Perl because a lot of system administrators know
Perl, so that would make it easier to get contributors.

  In 1998, there were few advanced, object-oriented web scripting
languages. In fact, Perl was pretty much it. PHP was at version 3.0,
python was at version 1.5, Java was just starting to become well-known,
ruby was almost unheard of, and some people were still writing their
CGI scripts in C or C++.

  Perl has many great features, most of all the number of libraries
available and the extreme flexibility of the language. 

  However, Perl would not be my first choice for writing or maintaining
a large project, such as Bugzilla. The same flexibility that makes Perl
so powerful makes it very difficult to enforce code quality standards
or to implement modern object-oriented designs. Here are the problems:

  * Reviewing Perl code takes much longer than reviewing other
languages. Here's why:

    o There are many ways to do the same thing in Perl. It's even the
motto: "There's More Than One Way To Do It." However, this means that
each reviewer must enforce very strict code guidlines on each coder, or
must learn each coder's coding style. In the interest of consistency,
we usually pick the former. This takes more reviewer time. It's also
frustrating for contributors who are used to writing in a particular
fashion.

    o More ways to write the same thing means there are many more *bad*
ways to write code in Perl than there are in other languages. In any
language it's *possible* to do stupid things and not realize it. In
Perl it's easy. Or even when the code does what you mean it to, just
because it *works* doesn't mean it's good. My experience is that Perl
encourages people to write code that "just works," but might not be
architected appropriately. Once again, this is possible in any
language, but I think Perl makes it easier than other languages to
write bad code.

    o It's very easy to make Perl code hard to read. It uses strange
variables like $_. It relies heavily on complex regular expressions. I
don't think anybody would argue that Perl *encourages* readability.

  * Perl lacks many of the features that implement what computer
scientists call "design by contract." That is, Perl doesn't enforce
things. For example, Perl doesn't check the type of arguments to
subroutines. You can't make subroutines private in a class. Programmers
have to actually read the documentation to know that a function is
really "private" or "protected." Perl doesn't have real assertions.
(The "assert" command in C, Python, Java, or various other languages.)

    Perl's lack of enforcement is a nice feature for the casual
programmer, but for the design of large applications, you want the
programming langugae itself to do as much error-checking for you as
possible, so that you don't have to write the error-checking yourself.

  * Perl lacks a real exception mechanism. We would have to write our
own if we want one.

  * Under mod_perl, because of the design of Perl, Apache processes
grow HUGE in size. We kill them if they get up to 70MB, but even 40MB
for a single Apache process is too big. The fact that Perl never
releases memory back to the kernel is a problem.

  * Without some experience, it can be difficult to read Perl's
compiler error messages and actually then determine what's wrong.

  Since 1998 there have been many advances in programming languages.
PHP has decent object-oriented features, python has many libraries and
excellent syntax, Java has matured a lot, and Ruby is coming up in the
world quickly.

  Nowadays, almost all of our competitors have one advantage: they are
not written in perl. They can actually develop features more quickly
than we can, not because of the number of contributors they have, but
because the language they're using allows it. There are at least two
bug-trackers that I can think of off the top of my head that didn't
even *exist* in 1998 and were developed rapidly up to a point where
they could compete with Bugzilla.

  However, honestly, I like Bugzilla better than all of our
competitors. We have almost 10 years of experience writing a bug
tracker. We know what people need and want from bug-tracking software.

  But still, any of you long-term contributors to Bugzilla who also
have experience in other languages, ask yourself this question: "In all
the time I've spent working on Bugzilla in Perl, how far could I have
gotten on writing another bug tracker, from SCRATCH, in another
language?" My personal estimate is that I could have entirely
re-written Bugzilla in Python or Ruby in half the time I've been
working on it in Perl. (That would be re-writing it in a year and a
half, not an unreasonable amount of time for 80,000 lines of code or
so.)

  Nowadays, even the virtue of "lots of system administrators speak
Perl" is fading. New admins are more likely to know Python than Perl.
And in about two years from now, I'll bet people will be just as likely
to know Ruby. Perl will continue to fade in popularity, I suspect.
Already there's no doubt that far more people know PHP than know Perl.

  So the popularity argument is dead.

  One advantage that Perl has is CPAN. There are a lot of libraries
available. But then again, that's also a problem that Perl has--people
need to install SO MANY modules just to use Bugzilla. Witness all the
protesting there is from our users whenever we add new required modules
to Bugzilla, and the support questions we get about problems with CPAN.

  And even that advantage is fading. There are a lot of python modules
available now. Java has Jakarta and a lot of other modules. And Ruby
has RubyGems, which are even easier to install than CPAN modules. PHP
has PEAR, which is also very nice.

  In 1998, Perl was the right choice for a language to re-write
Bugzilla in. In 2007, though, having worked with Perl extensively for
years on the Bugzilla project, I'd say the language itself is our
greatest hinderance.

  But what can we do about it? ohloh.net says that we have 43,762 lines
of code in Bugzilla (http://www.ohloh.net/projects/3524), and I think
we might even have twice that many, if you count templates. Not to
mention POD.

  I think that the experience of Netscape and the Mozilla Project shows
us that re-writing Bugzilla from scratch and totally ignoring our old
code base would be a bad idea. If we stopped development for a year and
a half, we'd be hopelessly behind and our users would start to abandon
us in droves.

  As far as I can see, if we want to move away from Perl and move to a
language that will be better for us as time goes on, we have two
choices:

  1) Figure out a way to re-write *parts* of Bugzilla in another
language without affecting performance or greatly adding to the
complexity of installation. We could then incrementally move to another
language.

  2) Work on both projects at once--a small team working on the
re-write in another language, and the same team we have now working on
the current Perl version, up until version 3.4 or 3.6.

  If #1 is possible, I think I'd obviously prefer that. However, if it
would be extremely difficult or be somehow bad for the project overall,
then we could do #2.

  No matter which way we go, these are the steps we'd have to go
through:

  1) Make a list of every single feature of Bugzilla that would have to
be re-written in a new version.

  2) Using this list, decide on a language to use that would be the
easiest and best for implementing all of those features. We could also
decide if we want to use a web framework like Rails (Ruby) or Pylons
(Python) to eliminate us having to write some code. After all, the less
code we have to write, the better.

     I've already been experimenting with various languages, and I've
started a page here that compares their advantages and disadvantages,
from Bugzilla's perspective:

     http://wiki.mozilla.org/Bugzilla:Languages

  3) Prototype some basic features of Bugzilla in that language, to see
how easy it actually is.

  4) Prioritize the feature list of Bugzilla, to figure out what we
have to re-implement in what order.

  5) Do some design of the system so that it makes sense and is
coherent when it's done. We don't have to re-design Bugzilla at this
point--we could get stuck in that forever. And we shouldn't design it
by committee. One or two people should work on the design, and then
present it to others for review.

     However, no matter what the design is, it's important to maintain
feature and API parity with the current Perl Bugzilla--otherwise it
will be very hard to get users and extensions to upgrade.

  6) Start work, based on the design and the priority list.

  Without taking some action, I'm not sure how many more years Bugzilla
can stay alive as a product. Currently, our popularity is actually
*increasing*, as far as I can see. So we shouldn't abandon what we're
doing now. But I'm seeing more and more products come into the
bug-tracking arena, and I'm not sure that we can stay competetive for
more than a few more years if we are stuck with Perl.

	-Max
-- 
http://www.everythingsolved.com/
Competent, Friendly Bugzilla Services. And Everything Else, too.