Self-Introduction: Fergus Sullivan

Wed Aug 29 19:49:08 UTC 2007

Note: I work for Fergus and will compose a self-introduction soon...

One of the things I asked about when deciding to come work for Yahoo!  
was whether we would be sharing changes with Mozilla.  The idea got a  
very warm reception.  The only reason we haven't been sharing is  
because our code isn't quite up to snuff and there have been greater  
priorities.  We are finally starting to settle down into a steady  
state of code joy, so there has been more recent discussion of how  
and when to contribute.

On Aug 29, 2007, at 12:19 PM, Gervase Markham wrote:

> So much to ask about!
>
> Fergus Sullivan wrote:
>> == Anything else you'd like to say ==
>> Yahoo has been happily using Bugzilla for the last six years.  We  
>> now have a mandate from our top-level management to contribute to  
>> the open source effort.
>
> Wahey!
>
>> We are currently using a customized version of 2.22.
>
> Do you have an upgrade plan?

Yes, but it's slow going.  (I have been at Yahoo! since January, so  
my information may be slightly incorrect.)  There have been competing  
factions in the past that have pushed for either forking our Bugzilla  
from b.m.o, followed by at least one effort to defork.  We have  
wrestled with this question at length, and what we want to do is to  
make all the changes that made 2.22 -> 3.0 happen, but it's a large  
effort.

>
>> We serve Bugzilla on FreeBSD 6, mysql 4.1, perl 5.6.  We use a  
>> mixture of MyISAM and InnoDB.
>
> As in, different tables use different engines? Which uses which?

Basically, everything that doesn't need a full-text index is in  
InnoDB.  We are exploring how to use InnoDB for all the tables.

>
>> We have a dual-master database (we have our own proxy that can  
>> switch as needed)
>
> So MySQL supports this?

Er, no, it doesn't... :)  read "We have our own proxy" again.

>
>> and a VIP handling 3 front-end and servers and search slaves.  We  
>> have a second colo for BCP with an equivalent architecture.  We  
>> have additional machines as reporting slaves.
>
> As in, they have read-only mirrors of the DB and are used for  
> running reports?
>

Yes, we give RO-access to  people who want to run their own reports.   
It beats having to tell someone why we aren't going to add their  
spiffy feature.

>> Our database currently stands at about 1.5 million bugs and grows  
>> at about 50k per month.
>
> That's the biggest known Bugzilla database, by about 3x. GNOME is  
> at about 470,000.
>
>> We have 20,000 named accounts (one per employee, plus many  
>> internal mailing lists), of whom about 8,300 are active in any one  
>> month.
>
> LDAP? RADIUS? Or a mirror of your internal employee database into  
> the Bugzilla DB?

We have several databases from which we extract information for our  
own database.

>
>> In terms of active users and new bugs opened, our load has  
>> increased 50% in the last year.  We have 2,400 separate products  
>> in our DB.
>
> Red Hat has 7814 components in a single product, but I don't know  
> of anyone who has that many products.
>
>> In addition to myself, we have two developers and one QA assigned  
>> full time on Bugzilla.  We have another two guys working full time  
>> on it at the moment, although they may move to other tasks in a  
>> few months time.
>
> Presumably you have a testing installation?

sort of...

>
>> Speed
>>  - The core SQL within Bugzilla did not scale to meet our needs.   
>> We reached a tipping point at around 800,000 bugs.  The main cause  
>> was lock contention within the DB.  The causes and solutions would  
>> fill an academic paper.  We rewrote search.pm and greatly  
>> increased performance.  Some of the changes were trivial.  A two  
>> line change improved our "search for bugs where person X is cced"  
>> from 45 seconds to 0.4 seconds.  Other changes were vast.  The  
>> overall effect was search times have dropped from up to 45 seconds  
>> in typical usage to about six seconds.
>
> I'm certainly looking forward to seeing these! Do you have any idea  
> of how well the changes fit with our changes between 2.22 and now?
>

I think they will fit in reasonably well, although it remains to be  
seen whether anyone who doesn't have a million rows in bugs would  
notice any particular improvement.

> Was it only search that was a problem?

I have found a couple of places (unknown whether they're Mozilla or  
Yahoo! code) that would gather information about products/components/ 
etc one row at a time with separate queries.  For most installations,  
it's probably not a noticeable performance hit to do this.  For us,  
it was a killer, and I have altered those to hit the database once.

>
>> Product Management
>>  - With 2,400 different products, the vanilla product chooser does  
>> not scale for us.  We have built an entirely different version.
>
> Have you been eyeing Classifications with interest, or do they not  
> help? How do you do the UI for this?

We use classifications, but I don't think it's well integrated in our  
product-choosing workflow.  (Although I'm not a typical Bugzilla  
user, of course.)

>
>> UI
>>  - We have made many minor and a few major changes to the UI.  One  
>> interesting one is the use of a JavaScript datatable for client- 
>> side sorting of buglist results.  This greatly reduces server load.
>
> I use a Greasemonkey script for this. I agree, it needs to be a  
> core feature.
>
>>  - We have unified comments and the bug activity table.  Both  
>> appear inline in the show bug page.
>
> Do you find this useful in practice?

This is still new, so we don't have a lot of feedback about it yet.   
I think it will be very useful, especially as we allow users to  
collapse this detail.

>
> Gerv
> -
> To view or change your list settings, click here:
> <http://bugzilla.org/cgi-bin/mj_wwwusr?user=dmarshal@yahoo-inc.com>