From gerv at mozilla.org  Mon Jun 11 16:18:59 2007
From: gerv at mozilla.org (Gervase Markham)
Date: Mon, 11 Jun 2007 17:18:59 +0100
Subject: [Fwd: Bugmail is less secure than Bug views]
Message-ID: <466D75F3.8070806@mozilla.org>

This is an email forwarded with permission from the Mozilla security 
group mailing list, on the subject of the fact that Bugzilla access is 
now SSL-protected, but bugmail is still transmitted in the clear.

Gerv

-------- Original Message --------
Subject: Bugmail is less secure than Bug views
Date: Mon, 11 Jun 2007 18:50:56 +0300
From: timeless <timeless at gmail.com>
Reply-To: timeless at gmail.com
To: security-group <security-group at mozilla.org>

Ben Bucksch wrote:
> We already use SSL for bugzilla, so it's hard to get to that.

Gervase Markham wrote:
> Except that every comment is sent out as unsecured bugmail. There is no
> immediate prospect of changing this.

At some point we (myself, a Bugzilla developer, and gerv also) should
consider how to approach this.

One thing would be to have bugmails simply be envelopes indicating bug
changes to secure bugs. another would be to enable users to associate
SMIME/PGP keys with Bugzilla and ask it to SMIME encrypt mail to them.

-- That's the opt in approach. (kinda like the old days where https:
for Bugzilla was optional)

The Bugzilla I'm using commercially ostensibly requires connections
between itself and recipient SMTP servers be secured (and it's somehow
assumed that all mail that arrives at recipient servers is then
retrieved securely).

-- That's a mandatory approach. (kinda like today where https: for
Bugzilla is required)

It would be nice if the security group at some point gave input about
which path would be preferred. Or how to make such a path.

Note: irc.mozilla.org now has ircs support, and +z which enables you
to require everyone joining a channel use an SSL tunnel (and the
security group uses this feature). So given that we use SSL for real
time communication and for general security views, we probably should
look getting bugmail to a similar level.
_______________________________________________
Security-group mailing list
Security-group at mozilla.org
https://mail.mozilla.org/listinfo/security-group


From mkanat at bugzilla.org  Mon Jun 11 21:53:08 2007
From: mkanat at bugzilla.org (Max Kanat-Alexander)
Date: Mon, 11 Jun 2007 14:53:08 -0700
Subject: [Fwd: Bugmail is less secure than Bug views]
In-Reply-To: <466D75F3.8070806@mozilla.org>
References: <466D75F3.8070806@mozilla.org>
Message-ID: <20070611215533.008043FC108@help.trusthosting.net>

	Anybody who has an SSL cert and advertises TLS support on their
mail server receives mail with SSL. They can also make their POP or IMAP
connections with SSL.

	The mail is clear in their inbox, true, but I don't think this
is a real security concern.

	-Max

On Mon, 11 Jun 2007 17:18:59 +0100 Gervase Markham <gerv at mozilla.org>
wrote:
> This is an email forwarded with permission from the Mozilla security 
> group mailing list, on the subject of the fact that Bugzilla access
> is now SSL-protected, but bugmail is still transmitted in the clear.

-- 
http://www.everythingsolved.com/
Competent, Friendly Bugzilla Services. And Everything Else, too.


From gerv at mozilla.org  Tue Jun 12 09:57:58 2007
From: gerv at mozilla.org (Gervase Markham)
Date: Tue, 12 Jun 2007 10:57:58 +0100
Subject: [Fwd: Bugmail is less secure than Bug views]
In-Reply-To: <20070611215533.008043FC108@help.trusthosting.net>
References: <466D75F3.8070806@mozilla.org> <20070611215533.008043FC108@help.trusthosting.net>
Message-ID: <466E6E26.40802@mozilla.org>

Max Kanat-Alexander wrote:
> 	Anybody who has an SSL cert and advertises TLS support on their
> mail server receives mail with SSL. They can also make their POP or IMAP
> connections with SSL.

This is true but, as someone commented on the security list, there's no 
way to enforce this. In a sense, it's optional rather than mandatory 
security. It also doesn't help if people have mail forwarding, or if 
they don't have control of their SMTP server (most people).

The way to do this would, I think, be to allow people to associate a 
S/MIME or PGP key with their account, which would then be used to 
encrypt all their bugmail where the bug was in one or more groups.

Sadly, none of these look ideal:
http://search.cpan.org/search?query=smime&mode=all
We need a library where we pass it the message text and the key, and get 
the encrypted text back. Most of them seem centred around files on disk.

Gerv


From jpyeron at pdinc.us  Tue Jun 12 13:44:59 2007
From: jpyeron at pdinc.us (Jason Pyeron)
Date: Tue, 12 Jun 2007 09:44:59 -0400
Subject: [Fwd: Bugmail is less secure than Bug views]
In-Reply-To: <466E6E26.40802@mozilla.org>
Message-ID: <002601c7acf7$e24830f0$6501a8c0@MRSLAPTOP>

It has been a while since I have worked with S/MIME, it is really simple
stuff. There should be no need to depend on any new CPAN modules. Just find
a package that you can suck into bz and modify it to play nice.

What about*: http://www.mozilla.org/projects/security/pki/nss/smime/

* I have note read the code, but the text seems to fit.
 


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-                                                               -
- Jason Pyeron                      PD Inc. http://www.pdinc.us -
- Sr. Consultant                    10 West 24th Street #100    -
- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -
-                                                               -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise private information. If you
have received it in error, purge the message from your system and
notify the sender immediately.  Any other use of the email by you
is prohibited. 


-----Original Message-----
From: developers-owner at bugzilla.org [mailto:developers-owner at bugzilla.org]
On Behalf Of Gervase Markham
Sent: Tuesday, June 12, 2007 5:58
To: developers at bugzilla.org
Subject: Re: [Fwd: Bugmail is less secure than Bug views]

Max Kanat-Alexander wrote:
> 	Anybody who has an SSL cert and advertises TLS support on their
> mail server receives mail with SSL. They can also make their POP or IMAP
> connections with SSL.

This is true but, as someone commented on the security list, there's no 
way to enforce this. In a sense, it's optional rather than mandatory 
security. It also doesn't help if people have mail forwarding, or if 
they don't have control of their SMTP server (most people).

The way to do this would, I think, be to allow people to associate a 
S/MIME or PGP key with their account, which would then be used to 
encrypt all their bugmail where the bug was in one or more groups.

Sadly, none of these look ideal:
http://search.cpan.org/search?query=smime&mode=all
We need a library where we pass it the message text and the key, and get 
the encrypted text back. Most of them seem centred around files on disk.

Gerv
-
To view or change your list settings, click here:
<http://bugzilla.org/cgi-bin/mj_wwwusr?user=jpyeron at pyerotechnics.com>



From bugzilla at glob.com.au  Tue Jun 12 14:04:52 2007
From: bugzilla at glob.com.au (byron)
Date: Tue, 12 Jun 2007 22:04:52 +0800
Subject: [Fwd: Bugmail is less secure than Bug views]
In-Reply-To: <466D75F3.8070806@mozilla.org>
References: <466D75F3.8070806@mozilla.org>
Message-ID: <20070612140451.GA19797@bur.st>

> This is an email forwarded with permission from the Mozilla security 
> group mailing list, on the subject of the fact that Bugzilla access is 
> now SSL-protected, but bugmail is still transmitted in the clear.


i recon we should implement this as a bugzilla plugin -- pass the
message to an external handler after it has been packaged in mime for
modification, then deliver the modified content.

it can then use whatever's appropiate to sign the message .. call
openssl, pgp, .net, ...


-b


begin-base64 644 signature.gif
R0lGODlhbQAHAIAAAABPo////ywAAAAAbQAHAAACfAxuGAnch+Bibkn7FL1p
XgVl4Ig1jjlZRoqybgun2Cur5uOunq7u/Ipq7WIyIc7XG9JquEgumPzdlhTf
h0O83kDJaXEm8mRHwXKJy5sac7qYOpT+gtv0n+0ujQOfdqh16caWt0foBViH
N1PRMXimiLUGt3ElVimlgbllWAAAOw==


From jochen.wiedmann at gmail.com  Wed Jun 13 06:33:06 2007
From: jochen.wiedmann at gmail.com (Jochen Wiedmann)
Date: Wed, 13 Jun 2007 08:33:06 +0200
Subject: Reuse of Bugzilla Authentication
Message-ID: <e75283b10706122333w2fb39927j8fad2d9e4fd9bbeb@mail.gmail.com>

Hi,

I have a system where I host a subversion repository, a WebDAV server,
and of course, a Bugzilla instance.

I am reusing the Bugzilla accounts for authentication in subversion
and WebDAV. In order to avoid problems with the '@' character in
account names, I have also added a field "login" to the Profiles
table. (For example, my user name is "jochen.wiedmann at softwareag.com",
but my login is "jwi".) The login is used for Basic authentication.

Is that of any interest to others?

Thanks,

Jochen

-- 
"Besides, manipulating elections is under penalty of law, resulting in
a preventative effect against manipulating elections.

The german government justifying the use of electronic voting machines
and obviously  believing that we don't need a police, because all
illegal actions are forbidden.

http://dip.bundestag.de/btd/16/051/1605194.pdf


From justdave at bugzilla.org  Wed Jun 13 09:39:40 2007
From: justdave at bugzilla.org (David Miller)
Date: Wed, 13 Jun 2007 05:39:40 -0400
Subject: Reuse of Bugzilla Authentication
In-Reply-To: <e75283b10706122333w2fb39927j8fad2d9e4fd9bbeb@mail.gmail.com>
References: <e75283b10706122333w2fb39927j8fad2d9e4fd9bbeb@mail.gmail.com>
Message-ID: <466FBB5C.6060702@bugzilla.org>

Jochen Wiedmann wrote on 6/13/07 2:33 AM:

> I am reusing the Bugzilla accounts for authentication in subversion
> and WebDAV. In order to avoid problems with the '@' character in
> account names, I have also added a field "login" to the Profiles
> table. (For example, my user name is "jochen.wiedmann at softwareag.com",
> but my login is "jwi".) The login is used for Basic authentication.
> 
> Is that of any interest to others?

I think Bugzilla already has an equivalent to this...  a column intended
to be used to map to userids on an external system...  looking at the
schema, I think this is the "extern_id" column.

-- 
Dave Miller                                   http://www.justdave.net/
System Administrator, Mozilla Corporation      http://www.mozilla.com/
Project Leader, Bugzilla Bug Tracking System  http://www.bugzilla.org/


From jochen.wiedmann at gmail.com  Wed Jun 13 10:43:50 2007
From: jochen.wiedmann at gmail.com (Jochen Wiedmann)
Date: Wed, 13 Jun 2007 12:43:50 +0200
Subject: Reuse of Bugzilla Authentication
In-Reply-To: <466FBB5C.6060702@bugzilla.org>
References: <e75283b10706122333w2fb39927j8fad2d9e4fd9bbeb@mail.gmail.com>
	 <466FBB5C.6060702@bugzilla.org>
Message-ID: <e75283b10706130343s276ae880s974a2634b3a51b8a@mail.gmail.com>

On 6/13/07, David Miller <justdave at bugzilla.org> wrote:

> I think Bugzilla already has an equivalent to this...  a column intended
> to be used to map to userids on an external system...  looking at the
> schema, I think this is the "extern_id" column.

Thanks for the hint. But this isn't maintainable via the GUI, is it?


-- 
"Besides, manipulating elections is under penalty of law, resulting in
a preventative effect against manipulating elections.

The german government justifying the use of electronic voting machines
and obviously  believing that we don't need a police, because all
illegal actions are forbidden.

http://dip.bundestag.de/btd/16/051/1605194.pdf


From gerv at mozilla.org  Wed Jun 13 13:16:05 2007
From: gerv at mozilla.org (Gervase Markham)
Date: Wed, 13 Jun 2007 14:16:05 +0100
Subject: [Fwd: Bugmail is less secure than Bug views]
In-Reply-To: <20070612140451.GA19797@bur.st>
References: <466D75F3.8070806@mozilla.org> <20070612140451.GA19797@bur.st>
Message-ID: <466FEE15.5040901@mozilla.org>

byron wrote:
> i recon we should implement this as a bugzilla plugin -- pass the
> message to an external handler after it has been packaged in mime for
> modification, then deliver the modified content.

That seems entirely reasonable.

Gerv


From gerv at mozilla.org  Wed Jun 13 13:18:29 2007
From: gerv at mozilla.org (Gervase Markham)
Date: Wed, 13 Jun 2007 14:18:29 +0100
Subject: [Fwd: Bugmail is less secure than Bug views]
In-Reply-To: <002601c7acf7$e24830f0$6501a8c0@MRSLAPTOP>
References: <002601c7acf7$e24830f0$6501a8c0@MRSLAPTOP>
Message-ID: <466FEEA5.6070200@mozilla.org>

Jason Pyeron wrote:
> It has been a while since I have worked with S/MIME, it is really simple
> stuff. There should be no need to depend on any new CPAN modules. Just find
> a package that you can suck into bz and modify it to play nice.
> 
> What about*: http://www.mozilla.org/projects/security/pki/nss/smime/
> 
> * I have note read the code, but the text seems to fit.

For one thing, that requires all the keys to be stored in an NSS-style 
database. I had hoped we could find something that worked like this:

$encryped_message = encrypt($message, $key);

with $key being whatever was pasted in plain text form into the "Insert 
Key Here" textbox on the profile page.

Gerv



From justdave at bugzilla.org  Wed Jun 13 13:53:49 2007
From: justdave at bugzilla.org (David Miller)
Date: Wed, 13 Jun 2007 09:53:49 -0400
Subject: Reuse of Bugzilla Authentication
In-Reply-To: <e75283b10706130343s276ae880s974a2634b3a51b8a@mail.gmail.com>
References: <e75283b10706122333w2fb39927j8fad2d9e4fd9bbeb@mail.gmail.com>	 <466FBB5C.6060702@bugzilla.org> <e75283b10706130343s276ae880s974a2634b3a51b8a@mail.gmail.com>
Message-ID: <466FF6ED.2030807@bugzilla.org>

Jochen Wiedmann wrote on 6/13/07 6:43 AM:
> On 6/13/07, David Miller <justdave at bugzilla.org> wrote:
> 
>> I think Bugzilla already has an equivalent to this...  a column intended
>> to be used to map to userids on an external system...  looking at the
>> schema, I think this is the "extern_id" column.
> 
> Thanks for the hint. But this isn't maintainable via the GUI, is it?

Doesn't look like it currently.  That would probably be easy to fix.
Right now it looks like it's only used Environment-based auth, so if
your email on your external auth account changes Bugzilla will
automatically fix its copy the next time you log in.

perldoc Bugzilla/Auth.pm shows this in the section about $login_data_hash:

       "extern_id"
           Some string that uniquely identifies the user in an external
           account source. If this "extern_id" already exists in the
database
           with a different username, the username will be *changed* to
be the
           username specified in this $login_data.

           That is, let's my extern_id is "mkanat". I already have an
account
           in Bugzilla with the username of "mkanat at foo.com". But this time,
           when I log in, I have an extern_id of "mkanat" and a
"username" of
           "mkanat at bar.org". So now, Bugzilla will automatically change my
           username to "mkanat at bar.org" instead of "mkanat at foo.com".


-- 
Dave Miller                                   http://www.justdave.net/
System Administrator, Mozilla Corporation      http://www.mozilla.com/
Project Leader, Bugzilla Bug Tracking System  http://www.bugzilla.org/


From jpyeron at pdinc.us  Wed Jun 13 19:10:14 2007
From: jpyeron at pdinc.us (Jason Pyeron)
Date: Wed, 13 Jun 2007 15:10:14 -0400
Subject: [Fwd: Bugmail is less secure than Bug views]
In-Reply-To: <466FEEA5.6070200@mozilla.org>
Message-ID: <001101c7adee$7c6a4730$6501a8c0@MRSLAPTOP>

From: developers-owner at bugzilla.org [mailto:developers-owner at bugzilla.org]
On Behalf Of Gervase Markham
Jason Pyeron wrote:
> It has been a while since I have worked with S/MIME, it is really simple
> stuff. There should be no need to depend on any new CPAN modules. Just
find
> a package that you can suck into bz and modify it to play nice.
> 
> What about*: http://www.mozilla.org/projects/security/pki/nss/smime/
> 
> * I have note read the code, but the text seems to fit.

For one thing, that requires all the keys to be stored in an NSS-style 
database. I had hoped we could find something that worked like this:

$encryped_message = encrypt($message, $key);

with $key being whatever was pasted in plain text form into the "Insert 
Key Here" textbox on the profile page.



What I was saying, was to do just that, absorb the core S/MIME part that
does the mime parsing and sign/encryption.

Then each/any user would add their x509 public cert in their profile select
encrypt or sign on all messages


send(encrypt(sign(msg,bz.prvkey),user.pubkey))

Or 

sign(msg,bz.prvkey)

There really is nothing to it, I just wish I had more time.



From lpsolit at gmail.com  Thu Jun 14 17:49:57 2007
From: lpsolit at gmail.com (=?ISO-8859-1?Q?Fr=E9d=E9ric_Buclin?=)
Date: Thu, 14 Jun 2007 19:49:57 +0200
Subject: Custom statuses and workflow fully implemented
Message-ID: <46717FC5.5070300@gmail.com>

Hi all,

We just committed the final patches to fully support custom bug statuses
and workflow. So in case you are playing with the current code from CVS,
have a look at it and let us know if something is going wrong.

And for those who don't use CVS, this feature will be available in the
coming Bugzilla 3.1.1, which should be available in the coming weeks.

LpSolit


From mkanat at bugzilla.org  Mon Jun 25 22:38:50 2007
From: mkanat at bugzilla.org (Max Kanat-Alexander)
Date: Mon, 25 Jun 2007 15:38:50 -0700
Subject: FullText Searching: The Dilemma
Message-ID: <20070625224300.63B13868002@help.trusthosting.net>

	Full-text searching in Bugzilla is a bit of a problem.

	In MySQL, InnoDB tables don't support FULLTEXT indexes. But
MyISAM tables don't support transactions.

	So, for 3.0 we removed bugs.short_desc from the FULLTEXT
searching and now we just use a normal LIKE search. However, you can
see that that makes large installations like bugzilla.mozilla.org very
slow.

	In addition, it's very difficult to write SQL (with the current
way Search.pm works) that ranks a bug based on *all* of its comments
combined, instead of just ranking it on each comment individually. That
is, if you search for "Java", a bug with a single comment that's just
the word "Java" could actually be ranked higher than a bug with 10
comments about Java. (To you SQL experts out there: I know it's
possible, it's just not easy with the way Search.pm works.)

	Also, fulltext engines are very different between databases,
requiring us to re-implement fulltext separately for each one.

	I've looked into using an external fulltext engine, like
Lucene or something similar. The advantage is that fulltext would be the
same across every DB, and we could rank based on all comments combined.

	The problem would be that the fulltext search would no longer
be combined with the SQL search. So we'd have to first do one, and then
the other. For example, we could first search all bugs for "Java" and
then restrict them based on the other search criteria.

	Does anybody have any ideas in this department, as to how we
could do this whole full-text searching better? In a way that would
perform well, be easy to develop, and work well with our current search
architecture?

	-Max
-- 
http://www.everythingsolved.com/
Competent, Friendly Bugzilla Services. And Everything Else, too.


From kevin.benton at amd.com  Tue Jun 26 03:55:48 2007
From: kevin.benton at amd.com (Benton, Kevin)
Date: Mon, 25 Jun 2007 20:55:48 -0700
Subject: FullText Searching: The Dilemma
In-Reply-To: <20070625224300.63B13868002@help.trusthosting.net>
References: <20070625224300.63B13868002@help.trusthosting.net>
Message-ID: <D85E06673A2DFF44B221CF78D1CBB11B0151D846@SSVLEXMB1.amd.com>

Max,

I'm sure this may have been discussed before, however, I don't see any
reason why we can't split the bugs table into non-text fields stored
using InnoDB and a separate bugs_more table that holds text fields
stored in MyISAM.  This would have two benefits:  1) it minimizes risk
by splitting data across tables (versus leaving the entire table in
MyISAM), and 2) gains transactional integrity while retaining speed (the
bugs table would retain transactional status while the bugs_more table
would not).  Splitting the tables also gains a hidden benefit - it makes
it more likely that searches of non-bugs_more data will go much faster
because it's possible that records in the bugs table may be stored in a
fixed format even though we have variable-length records in the table.

Just a few thoughts... :-)


Kevin Benton
MySQL DBA #5739
Senior Software Developer
MSS Silicon Design Engineering
Advanced Micro Devices
 
The opinions stated in this communication do not necessarily reflect the
view of Advanced Micro Devices and have not been reviewed by management.
This communication may contain sensitive and/or confidential and/or
proprietary information.  Distribution of such information is strictly
prohibited without prior consent of Advanced Micro Devices.  This
communication is for the intended recipient(s) only.  If you have
received this communication in error, please notify the sender, then
destroy any remaining copies of this communication.
 
> -----Original Message-----
> From: developers-owner at bugzilla.org
[mailto:developers-owner at bugzilla.org]
> On Behalf Of Max Kanat-Alexander
> Sent: Monday, June 25, 2007 4:39 PM
> To: developers at bugzilla.org
> Subject: FullText Searching: The Dilemma
> 
> 	Full-text searching in Bugzilla is a bit of a problem.
> 
> 	In MySQL, InnoDB tables don't support FULLTEXT indexes. But
> MyISAM tables don't support transactions.
> 
> 	So, for 3.0 we removed bugs.short_desc from the FULLTEXT
> searching and now we just use a normal LIKE search. However, you can
> see that that makes large installations like bugzilla.mozilla.org very
> slow.
> 
> 	In addition, it's very difficult to write SQL (with the current
> way Search.pm works) that ranks a bug based on *all* of its comments
> combined, instead of just ranking it on each comment individually.
That
> is, if you search for "Java", a bug with a single comment that's just
> the word "Java" could actually be ranked higher than a bug with 10
> comments about Java. (To you SQL experts out there: I know it's
> possible, it's just not easy with the way Search.pm works.)
> 
> 	Also, fulltext engines are very different between databases,
> requiring us to re-implement fulltext separately for each one.
> 
> 	I've looked into using an external fulltext engine, like
> Lucene or something similar. The advantage is that fulltext would be
the
> same across every DB, and we could rank based on all comments
combined.
> 
> 	The problem would be that the fulltext search would no longer
> be combined with the SQL search. So we'd have to first do one, and
then
> the other. For example, we could first search all bugs for "Java" and
> then restrict them based on the other search criteria.
> 
> 	Does anybody have any ideas in this department, as to how we
> could do this whole full-text searching better? In a way that would
> perform well, be easy to develop, and work well with our current
search
> architecture?
> 
> 	-Max
> --
> http://www.everythingsolved.com/
> Competent, Friendly Bugzilla Services. And Everything Else, too.
> -
> To view or change your list settings, click here:
> <http://bugzilla.org/cgi-bin/mj_wwwusr?user=kevin.benton at amd.com>
> 





From mkanat at bugzilla.org  Tue Jun 26 05:12:43 2007
From: mkanat at bugzilla.org (Max Kanat-Alexander)
Date: Mon, 25 Jun 2007 22:12:43 -0700
Subject: FullText Searching: The Dilemma
In-Reply-To: <D85E06673A2DFF44B221CF78D1CBB11B0151D846@SSVLEXMB1.amd.com>
References: <20070625224300.63B13868002@help.trusthosting.net>
	<D85E06673A2DFF44B221CF78D1CBB11B0151D846@SSVLEXMB1.amd.com>
Message-ID: <20070626051656.AB706868004@help.trusthosting.net>

On Mon, 25 Jun 2007 20:55:48 -0700 "Benton, Kevin"
<kevin.benton at amd.com> wrote:
> I'm sure this may have been discussed before, however, I don't see any
> reason why we can't split the bugs table into non-text fields stored
> using InnoDB and a separate bugs_more table that holds text fields
> stored in MyISAM.

	Yes, I've considered that. I'm against it ecause only MySQL
needs that, and I don't want to have this extra table in there for
every DB just because MySQL has this limitation.

	Also, that doesn't solve the other problems.

	Basically, it just doesn't sound to me like an architecturally
ideal solution.

	-Max
-- 
http://www.everythingsolved.com/
Competent, Friendly Bugzilla Services. And Everything Else, too.


From jeffh at tragicallyleet.com  Tue Jun 26 05:52:36 2007
From: jeffh at tragicallyleet.com (Jeffrey Hulten)
Date: Mon, 25 Jun 2007 22:52:36 -0700
Subject: FullText Searching: The Dilemma
In-Reply-To: <20070626051656.AB706868004@help.trusthosting.net>
References: <20070625224300.63B13868002@help.trusthosting.net> <D85E06673A2DFF44B221CF78D1CBB11B0151D846@SSVLEXMB1.amd.com> <20070626051656.AB706868004@help.trusthosting.net>
Message-ID: <755D12FB-0359-4468-A208-D88AD97233C3@tragicallyleet.com>

An alternative, while ugly, is writing your own tokenizer and  
implementing search internal to Bugzilla.  It does solve the  
multiplatform issue.


On Jun 25, 2007, at 10:12 PM, Max Kanat-Alexander wrote:

> On Mon, 25 Jun 2007 20:55:48 -0700 "Benton, Kevin"
> <kevin.benton at amd.com> wrote:
>> I'm sure this may have been discussed before, however, I don't see  
>> any
>> reason why we can't split the bugs table into non-text fields stored
>> using InnoDB and a separate bugs_more table that holds text fields
>> stored in MyISAM.
>
> 	Yes, I've considered that. I'm against it ecause only MySQL
> needs that, and I don't want to have this extra table in there for
> every DB just because MySQL has this limitation.
>
> 	Also, that doesn't solve the other problems.
>
> 	Basically, it just doesn't sound to me like an architecturally
> ideal solution.
>
> 	-Max
> -- 
> http://www.everythingsolved.com/
> Competent, Friendly Bugzilla Services. And Everything Else, too.
> -
> To view or change your list settings, click here:
> <http://bugzilla.org/cgi-bin/mj_wwwusr?user=jeffh at tragicallyleet.com>



From mkanat at bugzilla.org  Tue Jun 26 05:58:01 2007
From: mkanat at bugzilla.org (Max Kanat-Alexander)
Date: Mon, 25 Jun 2007 22:58:01 -0700
Subject: FullText Searching: The Dilemma
In-Reply-To: <755D12FB-0359-4468-A208-D88AD97233C3@tragicallyleet.com>
References: <20070625224300.63B13868002@help.trusthosting.net>
	<D85E06673A2DFF44B221CF78D1CBB11B0151D846@SSVLEXMB1.amd.com>
	<20070626051656.AB706868004@help.trusthosting.net>
	<755D12FB-0359-4468-A208-D88AD97233C3@tragicallyleet.com>
Message-ID: <20070626060215.5E666868004@help.trusthosting.net>

On Mon, 25 Jun 2007 22:52:36 -0700 Jeffrey Hulten
<jeffh at tragicallyleet.com> wrote:
> An alternative, while ugly, is writing your own tokenizer and  
> implementing search internal to Bugzilla.  It does solve the  
> multiplatform issue.

	Well, instead of that I'd just use some external fulltext
indexer that has an API that does that for us. That's one of the
foremost options, actually--I just don't necessarily see what the best
way to integrate that into Bugzilla would be.

	-Max
-- 
http://www.everythingsolved.com/
Competent, Friendly Bugzilla Services. And Everything Else, too.


From gerv at mozilla.org  Tue Jun 26 09:39:32 2007
From: gerv at mozilla.org (Gervase Markham)
Date: Tue, 26 Jun 2007 10:39:32 +0100
Subject: FullText Searching: The Dilemma
In-Reply-To: <20070625224300.63B13868002@help.trusthosting.net>
References: <20070625224300.63B13868002@help.trusthosting.net>
Message-ID: <4680DED4.8090405@mozilla.org>

Max Kanat-Alexander wrote:
> 	In MySQL, InnoDB tables don't support FULLTEXT indexes. But
> MyISAM tables don't support transactions.

Are either of these facts going to change in the near future?

Just to be clear: different tables in the same database can use 
different storage engines?

Can we support transactions on a MyISAM table by having a shadow InnoDB 
table with 1:1 row correspondence but no content, whose rows we lock 
before accessing the real table and unlock afterwards?

This is like the bugs/bugs_more plan except that it doesn't require a 
split; bugs_shadow_locking would exist on MySQL only, and could go away 
in the future if need be without changing much code.

Another random thought: would it be possible to mirror the full text 
fields into a second MyISAM "cache" table? Again, this avoids the 
restructuring needed for a split. But perhaps it wouldn't make it any 
easier to write the search code.

> 	So, for 3.0 we removed bugs.short_desc from the FULLTEXT
> searching and now we just use a normal LIKE search. However, you can
> see that that makes large installations like bugzilla.mozilla.org very
> slow.

Only large installations? Can we fix this (thinking laterally) by 
throwing more hardware at the b.m.o. problem?

> 	I've looked into using an external fulltext engine, like
> Lucene or something similar. The advantage is that fulltext would be the
> same across every DB, and we could rank based on all comments combined.

It's Yet Another Dependency, of course. And it doesn't seem very good 
architecturally, for the reasons you mention.

Gerv


From bbaetz at acm.org  Tue Jun 26 13:57:18 2007
From: bbaetz at acm.org (Bradley Baetz)
Date: Tue, 26 Jun 2007 23:57:18 +1000
Subject: FullText Searching: The Dilemma
In-Reply-To: <4680DED4.8090405@mozilla.org>
References: <20070625224300.63B13868002@help.trusthosting.net>
	 <4680DED4.8090405@mozilla.org>
Message-ID: <99435f5b0706260657w1eff6b22ncb5d85802abadfde@mail.gmail.com>

On 26/06/07, Gervase Markham <gerv at mozilla.org> wrote:
> Max Kanat-Alexander wrote:
> >       In MySQL, InnoDB tables don't support FULLTEXT indexes. But
> > MyISAM tables don't support transactions.
>
> Are either of these facts going to change in the near future?

I believe not. Theres the new storage engine (whose name I forget),
but I wouldn't count on that for production code for a while.

> Just to be clear: different tables in the same database can use
> different storage engines?

Yes, but you get into ickyness when you try to use transactions over
them. It means that bugzilla could never do BEGIN; try something;
COMMIT and rely on <something> failing doing a rollback, for example.

> Can we support transactions on a MyISAM table by having a shadow InnoDB
> table with 1:1 row correspondence but no content, whose rows we lock
> before accessing the real table and unlock afterwards?

No, because you can't do row-level locking. I've implemented something
similar in the past with GET_LOCK - you GET_LOCK("bz_bugX"). But you
then have to have everything grab that lock, and buglist.cgi would
have to lock the whole table since it doesn't know what rows to do in
advance.

> Another random thought: would it be possible to mirror the full text
> fields into a second MyISAM "cache" table? Again, this avoids the
> restructuring needed for a split. But perhaps it wouldn't make it any
> easier to write the search code.

You can possibly do something with replication. Since everything is
statement level replication (5.1 may be different), you could
replicate the single table into another db where the table type was
different. But since you can only have one replication master in
mysql, you'd actually probably have to do it in a different mysql
instance. Its possible, but....

> >       I've looked into using an external fulltext engine, like
> > Lucene or something similar. The advantage is that fulltext would be the
> > same across every DB, and we could rank based on all comments combined.
>
> It's Yet Another Dependency, of course. And it doesn't seem very good
> architecturally, for the reasons you mention.

The 'best' option would have to be something like the old shadowdb
code did - insert changes into a separate (innodb) table, then have a
separate script put the data into a side myisam table, with format
(bug_id, key, value) using REPLACE. Remember, its not just long_desc,
its the summary field too, at least (isn't it?)

That comes will all the problems that the shadowdb had, too, of course.

Anyone know how useful mysql's trigger support is for doing that
stuff? Especially without having its next key locking get in the way?

Or, of course, just recommend that large installs use postgres+tsearch? ;)

Bradley


From dwilliss at microimages.com  Tue Jun 26 14:27:00 2007
From: dwilliss at microimages.com (Dave Williss)
Date: Tue, 26 Jun 2007 09:27:00 -0500
Subject: FullText Searching: The Dilemma
In-Reply-To: <99435f5b0706260657w1eff6b22ncb5d85802abadfde@mail.gmail.com>
References: <20070625224300.63B13868002@help.trusthosting.net>	 <4680DED4.8090405@mozilla.org> <99435f5b0706260657w1eff6b22ncb5d85802abadfde@mail.gmail.com>
Message-ID: <46812234.8070505@microimages.com>



Bradley Baetz wrote:
> Anyone know how useful mysql's trigger support is for doing that
> stuff? Especially without having its next key locking get in the way?
I've used triggers in MySQL in another database project and it worked 
pretty well.  I don't know about locking issues or how well it scales 
though, as the table the trigger was on is modified infrequently.




From gerv at mozilla.org  Tue Jun 26 14:34:09 2007
From: gerv at mozilla.org (Gervase Markham)
Date: Tue, 26 Jun 2007 15:34:09 +0100
Subject: FullText Searching: The Dilemma
In-Reply-To: <99435f5b0706260657w1eff6b22ncb5d85802abadfde@mail.gmail.com>
References: <20070625224300.63B13868002@help.trusthosting.net>	 <4680DED4.8090405@mozilla.org> <99435f5b0706260657w1eff6b22ncb5d85802abadfde@mail.gmail.com>
Message-ID: <468123E1.2000108@mozilla.org>

Bradley Baetz wrote:
> On 26/06/07, Gervase Markham <gerv at mozilla.org> wrote:
>> Are either of these facts going to change in the near future?
> 
> I believe not. Theres the new storage engine (whose name I forget),
> but I wouldn't count on that for production code for a while.

I ask only because MySQL/InnoDB engineers were mumbling in emails about 
full text searching for InnoDB as far back as early '06.

> Yes, but you get into ickyness when you try to use transactions over
> them. It means that bugzilla could never do BEGIN; try something;
> COMMIT and rely on <something> failing doing a rollback, for example.

I can't quite parse that...

>> Can we support transactions on a MyISAM table by having a shadow InnoDB
>> table with 1:1 row correspondence but no content, whose rows we lock
>> before accessing the real table and unlock afterwards?
> 
> No, because you can't do row-level locking. 

I don't quite follow. Where am I going wrong?

- Lock row 23 of InnoDB table
- Edit row 23 of MyISAM table
- Unlock row 23 of InnoDB table

InnoDB supports row level locking...

> I've implemented something
> similar in the past with GET_LOCK - you GET_LOCK("bz_bugX"). But you
> then have to have everything grab that lock, and buglist.cgi would
> have to lock the whole table since it doesn't know what rows to do in
> advance.

Why does buglist.cgi have to lock anything? All it does is display data. 
You might get some bugs being after one update, and other bugs being 
before it, but that's not the world's scariest thing.

> Anyone know how useful mysql's trigger support is for doing that
> stuff? Especially without having its next key locking get in the way?

I think I breezed past trigger-based solutions to this problem on the 
web. We're not the first people to hit it, of course :-)

> Or, of course, just recommend that large installs use postgres+tsearch? ;)

We could do that. How hard would it be to convert b.m.o.? Are there 
scripts, or do all conversions have to be done by hand? Do we know how 
MoCo IT would feel about supporting Postgres? Do we feel the Postgres 
support is stable enough? Are there other downsides?

Gerv


From lpsolit at gmail.com  Tue Jun 26 17:18:04 2007
From: lpsolit at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBCdWNsaW4=?=)
Date: Tue, 26 Jun 2007 19:18:04 +0200
Subject: FullText Searching: The Dilemma
In-Reply-To: <468123E1.2000108@mozilla.org>
References: <20070625224300.63B13868002@help.trusthosting.net>	 <4680DED4.8090405@mozilla.org> <99435f5b0706260657w1eff6b22ncb5d85802abadfde@mail.gmail.com> <468123E1.2000108@mozilla.org>
Message-ID: <46814A4C.20701@gmail.com>

>> Or, of course, just recommend that large installs use
>> postgres+tsearch? ;)
> 
> We could do that. How hard would it be to convert b.m.o.? Are there
> scripts, or do all conversions have to be done by hand? Do we know how
> MoCo IT would feel about supporting Postgres? Do we feel the Postgres
> support is stable enough? Are there other downsides?

I second this idea. Instead of using some external program (which I'm
against. We already depend on (too) many Perl modules), we could say
that big installations move to PostgreSQL if they have perf problems.
Bugzilla 2.22 and higher run on PostgreSQL as fine as it runs on MySQL.
We do QA on both DB with exactly the same tests and no failure has been
detected with PostgreSQL.

LpSolit


From ghendricks at novell.com  Tue Jun 26 17:51:54 2007
From: ghendricks at novell.com (Greg Hendricks)
Date: Tue, 26 Jun 2007 11:51:54 -0600
Subject: FullText Searching: The Dilemma
In-Reply-To: 46814A4C.20701@gmail.com
References: <20070625224300.63B13868002@help.trusthosting.net> <468123E1.2000108@mozilla.org> 46814A4C.20701@gmail.com
Message-ID: <200706261151.55050.ghendricks@novell.com>

On Tuesday 26 June 2007 11:18, Fr?d?ric Buclin wrote:
> >> Or, of course, just recommend that large installs use
> >> postgres+tsearch? ;)
> >
> > We could do that. How hard would it be to convert b.m.o.? Are there
> > scripts, or do all conversions have to be done by hand? Do we know how
> > MoCo IT would feel about supporting Postgres? Do we feel the Postgres
> > support is stable enough? Are there other downsides?
>
> I second this idea. Instead of using some external program (which I'm
> against. We already depend on (too) many Perl modules), we could say
> that big installations move to PostgreSQL if they have perf problems.
> Bugzilla 2.22 and higher run on PostgreSQL as fine as it runs on MySQL.
> We do QA on both DB with exactly the same tests and no failure has been
> detected with PostgreSQL.
>
As wonderful as I am sure Postgres is, I don't see us moving to it any time 
soon. We have commercial support for MySQL and I don't think the bosses will 
be willing to renegotiate on a new DB provider (do they even have commercial 
support for PostGres?) in the next couple years. I don't think you can 
require that "Large installations" move to it. How do you define Large? Does 
that mean it is ok to start with MySQL when you are "Small" and then Bugzilla 
warns you when you reach your 100,000th bug that you now need to switch? I 
don't think that is viable.


From lpsolit at gmail.com  Tue Jun 26 18:09:21 2007
From: lpsolit at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBCdWNsaW4=?=)
Date: Tue, 26 Jun 2007 20:09:21 +0200
Subject: FullText Searching: The Dilemma
In-Reply-To: <200706261151.55050.ghendricks@novell.com>
References: <20070625224300.63B13868002@help.trusthosting.net> <468123E1.2000108@mozilla.org> 46814A4C.20701@gmail.com <200706261151.55050.ghendricks@novell.com>
Message-ID: <46815651.9080702@gmail.com>

> support for PostGres?) in the next couple years. I don't think you can 
> require that "Large installations" move to it. How do you define Large? Does 
> that mean it is ok to start with MySQL when you are "Small" and then Bugzilla 
> warns you when you reach your 100,000th bug that you now need to switch? I 
> don't think that is viable.

I define large as in "This installation is definitely too slow". If you
notice no perf problem, with bothering moving to Pg? And some
installations remain small, in which case MySQL is fine. b.m.o has been
able to handle 380,000 bugs till now; I don't know how bad perf are
affected by the upgrade to 3.0. Max?

LpSolit


From bbaetz at acm.org  Tue Jun 26 22:17:33 2007
From: bbaetz at acm.org (Bradley Baetz)
Date: Wed, 27 Jun 2007 08:17:33 +1000
Subject: FullText Searching: The Dilemma
In-Reply-To: <468123E1.2000108@mozilla.org>
References: <20070625224300.63B13868002@help.trusthosting.net>
	 <4680DED4.8090405@mozilla.org>
	 <99435f5b0706260657w1eff6b22ncb5d85802abadfde@mail.gmail.com>
	 <468123E1.2000108@mozilla.org>
Message-ID: <99435f5b0706261517q17d5b8afw3d121101f7725fee@mail.gmail.com>

On 27/06/07, Gervase Markham <gerv at mozilla.org> wrote:
> Bradley Baetz wrote:
> > On 26/06/07, Gervase Markham <gerv at mozilla.org> wrote:
> >> Are either of these facts going to change in the near future?
> >
> > I believe not. Theres the new storage engine (whose name I forget),
> > but I wouldn't count on that for production code for a while.
>
> I ask only because MySQL/InnoDB engineers were mumbling in emails about
> full text searching for InnoDB as far back as early '06.

Yeah.

> > Yes, but you get into ickyness when you try to use transactions over
> > them. It means that bugzilla could never do BEGIN; try something;
> > COMMIT and rely on <something> failing doing a rollback, for example.
>
> I can't quite parse that...

Its not ACID, so:

eval {
  $dbh->begin;
  UPDATE
  $dbh->commit;
};
if ($@) {
  ThrowCodeError(....)
}

may leave the secondary table modified but not the first. Or a power
failure in the middle may do the same. Theres no ACID for myiasm.

> >> Can we support transactions on a MyISAM table by having a shadow InnoDB
> >> table with 1:1 row correspondence but no content, whose rows we lock
> >> before accessing the real table and unlock afterwards?
> >
> > No, because you can't do row-level locking.
>
> I don't quite follow. Where am I going wrong?
>
> - Lock row 23 of InnoDB table
> - Edit row 23 of MyISAM table
> - Unlock row 23 of InnoDB table
>
> InnoDB supports row level locking...

Firstly, you can't explicitly lock a row.

Secondly, changes made to the myisam table will show up immediately,
but the innodb ones won't.

> > I've implemented something
> > similar in the past with GET_LOCK - you GET_LOCK("bz_bugX"). But you
> > then have to have everything grab that lock, and buglist.cgi would
> > have to lock the whole table since it doesn't know what rows to do in
> > advance.
>
> Why does buglist.cgi have to lock anything? All it does is display data.
> You might get some bugs being after one update, and other bugs being
> before it, but that's not the world's scariest thing.

> > Or, of course, just recommend that large installs use postgres+tsearch? ;)
>
> We could do that. How hard would it be to convert b.m.o.? Are there
> scripts, or do all conversions have to be done by hand? Do we know how
> MoCo IT would feel about supporting Postgres? Do we feel the Postgres
> support is stable enough? Are there other downsides?

The biggest issues are postgres upgrades (dump+restore). Theres also
the issue that if you use slony for replication, schema changes have
to happen via the slony tools, which may prove interesting for
checksetup. Probably not too hard now that the sql-level stuff has
been abstracted out, but would need a fair bit of tweaking.

OTOH, maybe bugzilla doesn't need replication for performance due to
MVCC, and can use DRBD for redundancy?

Bradley


From mkanat at bugzilla.org  Tue Jun 26 23:14:50 2007
From: mkanat at bugzilla.org (Max Kanat-Alexander)
Date: Tue, 26 Jun 2007 16:14:50 -0700
Subject: FullText Searching: The Dilemma
In-Reply-To: <468123E1.2000108@mozilla.org>
References: <20070625224300.63B13868002@help.trusthosting.net>
	<4680DED4.8090405@mozilla.org>
	<99435f5b0706260657w1eff6b22ncb5d85802abadfde@mail.gmail.com>
	<468123E1.2000108@mozilla.org>
Message-ID: <20070626231904.3085D86800E@help.trusthosting.net>

On Tue, 26 Jun 2007 15:34:09 +0100 Gervase Markham <gerv at mozilla.org>
wrote:
> We could do that. How hard would it be to convert b.m.o.? Are there 
> scripts, or do all conversions have to be done by hand? Do we know
> how MoCo IT would feel about supporting Postgres? Do we feel the
> Postgres support is stable enough? Are there other downsides?

	We actually talked about it during the last upgrade, I believe.

	The problem is that currently PostgreSQL is case-sensitive for
searches, which is too much of a pain.

	If there was a way to make the whole DB use a case-insensitive
search and collation (and still be able to use indexes), we could do
that.

	-Max
-- 
http://www.everythingsolved.com/
Competent, Friendly Bugzilla Services. And Everything Else, too.


From mkanat at bugzilla.org  Tue Jun 26 23:16:46 2007
From: mkanat at bugzilla.org (Max Kanat-Alexander)
Date: Tue, 26 Jun 2007 16:16:46 -0700
Subject: FullText Searching: The Dilemma
In-Reply-To: <4680DED4.8090405@mozilla.org>
References: <20070625224300.63B13868002@help.trusthosting.net>
	<4680DED4.8090405@mozilla.org>
Message-ID: <20070626232100.691F286800E@help.trusthosting.net>

On Tue, 26 Jun 2007 10:39:32 +0100 Gervase Markham <gerv at mozilla.org>
wrote:
> Max Kanat-Alexander wrote:
> > 	In MySQL, InnoDB tables don't support FULLTEXT indexes. But
> > MyISAM tables don't support transactions.
> 
> Are either of these facts going to change in the near future?

	No. I talked to some folks at MySQLConf. Anyhow, "near future"
would mean 5.2 or later, and we couldn't require that for quite some
time.

> Just to be clear: different tables in the same database can use 
> different storage engines?

	Yes.

	We already do--longdescs is still MyISAM in 3.1. (We update it
outside of transactions.)

> Can we support transactions on a MyISAM table by having a shadow
> InnoDB table with 1:1 row correspondence but no content, whose rows
> we lock before accessing the real table and unlock afterwards?

	Not really. That's still table-locking, and table-locking is
what I'm trying to get out of with transactions.

> Another random thought: would it be possible to mirror the full text 
> fields into a second MyISAM "cache" table? Again, this avoids the 
> restructuring needed for a split. But perhaps it wouldn't make it any 
> easier to write the search code.

	That's the most commonly-suggested solution, certainly.

	-Max
-- 
http://www.everythingsolved.com/
Competent, Friendly Bugzilla Services. And Everything Else, too.


From bbaetz at acm.org  Tue Jun 26 23:30:33 2007
From: bbaetz at acm.org (Bradley Baetz)
Date: Wed, 27 Jun 2007 09:30:33 +1000
Subject: FullText Searching: The Dilemma
In-Reply-To: <20070626231904.3085D86800E@help.trusthosting.net>
References: <20070625224300.63B13868002@help.trusthosting.net>
	 <4680DED4.8090405@mozilla.org>
	 <99435f5b0706260657w1eff6b22ncb5d85802abadfde@mail.gmail.com>
	 <468123E1.2000108@mozilla.org>
	 <20070626231904.3085D86800E@help.trusthosting.net>
Message-ID: <99435f5b0706261630s2a5bef65sb7ab0e0c3c973d9e@mail.gmail.com>

On 27/06/07, Max Kanat-Alexander <mkanat at bugzilla.org> wrote:
> On Tue, 26 Jun 2007 15:34:09 +0100 Gervase Markham <gerv at mozilla.org>
> wrote:
> > We could do that. How hard would it be to convert b.m.o.? Are there
> > scripts, or do all conversions have to be done by hand? Do we know
> > how MoCo IT would feel about supporting Postgres? Do we feel the
> > Postgres support is stable enough? Are there other downsides?
>
>         We actually talked about it during the last upgrade, I believe.
>
>         The problem is that currently PostgreSQL is case-sensitive for
> searches, which is too much of a pain.
>
>         If there was a way to make the whole DB use a case-insensitive
> search and collation (and still be able to use indexes), we could do
> that.

Postgres supports functional indexes. So you create an index on
UPPER(column) and then a search WHERE UPPER(column) = 'FOO' will use
the index.

Not sure if the full text stuff does case insensitive directly.

Bradley


From mkanat at bugzilla.org  Tue Jun 26 23:31:01 2007
From: mkanat at bugzilla.org (Max Kanat-Alexander)
Date: Tue, 26 Jun 2007 16:31:01 -0700
Subject: FullText Searching: The Dilemma
In-Reply-To: <99435f5b0706261630s2a5bef65sb7ab0e0c3c973d9e@mail.gmail.com>
References: <20070625224300.63B13868002@help.trusthosting.net>
	<4680DED4.8090405@mozilla.org>
	<99435f5b0706260657w1eff6b22ncb5d85802abadfde@mail.gmail.com>
	<468123E1.2000108@mozilla.org>
	<20070626231904.3085D86800E@help.trusthosting.net>
	<99435f5b0706261630s2a5bef65sb7ab0e0c3c973d9e@mail.gmail.com>
Message-ID: <20070626233515.2403486800E@help.trusthosting.net>

On Wed, 27 Jun 2007 09:30:33 +1000 "Bradley Baetz" <bbaetz at acm.org>
wrote:
> Postgres supports functional indexes. So you create an index on
> UPPER(column) and then a search WHERE UPPER(column) = 'FOO' will use
> the index.

	Yes, I know. We use those already in a few cases. The problem
isn't that--the problem is modifying all of Bugzilla to use
$dbh->sql_istrcmp, which would also be ugly and hard to maintain.

	Remember, MySQL *doesn't* support functional indexes, and
*won't* use a normal index on LOWER(col) = 'blah';

	-Max
-- 
http://www.everythingsolved.com/
Competent, Friendly Bugzilla Services. And Everything Else, too.


From mkanat at bugzilla.org  Tue Jun 26 23:35:58 2007
From: mkanat at bugzilla.org (Max Kanat-Alexander)
Date: Tue, 26 Jun 2007 16:35:58 -0700
Subject: FullText Searching: The Dilemma
In-Reply-To: <99435f5b0706260657w1eff6b22ncb5d85802abadfde@mail.gmail.com>
References: <20070625224300.63B13868002@help.trusthosting.net>
	<4680DED4.8090405@mozilla.org>
	<99435f5b0706260657w1eff6b22ncb5d85802abadfde@mail.gmail.com>
Message-ID: <20070626234012.5827286800E@help.trusthosting.net>

On Tue, 26 Jun 2007 23:57:18 +1000 "Bradley Baetz" <bbaetz at acm.org>
wrote:
> The 'best' option would have to be something like the old shadowdb
> code did - insert changes into a separate (innodb) table, then have a
> separate script put the data into a side myisam table, with format
> (bug_id, key, value) using REPLACE. Remember, its not just long_desc,
> its the summary field too, at least (isn't it?)

	Right. It's short_desc that's the problem here, actually.
Because of all the tables in Bugzilla, I can't make the "bugs" table
MyISAM--we'd lose almost all the advantages of transactions.

> Anyone know how useful mysql's trigger support is for doing that
> stuff? Especially without having its next key locking get in the way?

	Hrm...that might be a good idea, actually. I'm not sure
triggers are available before 5.0, though, and we currently only
require 4.1.

	However, if we could get that to work, that's probably the best
solution. The only problem being the MySQL version required.

> Or, of course, just recommend that large installs use
> postgres+tsearch? ;)

	We don't even support tsearch, yet, actually.

	-Max
-- 
http://www.everythingsolved.com/
Competent, Friendly Bugzilla Services. And Everything Else, too.


From bbaetz at acm.org  Wed Jun 27 00:03:53 2007
From: bbaetz at acm.org (Bradley Baetz)
Date: Wed, 27 Jun 2007 10:03:53 +1000
Subject: FullText Searching: The Dilemma
In-Reply-To: <20070626233515.2403486800E@help.trusthosting.net>
References: <20070625224300.63B13868002@help.trusthosting.net>
	 <4680DED4.8090405@mozilla.org>
	 <99435f5b0706260657w1eff6b22ncb5d85802abadfde@mail.gmail.com>
	 <468123E1.2000108@mozilla.org>
	 <20070626231904.3085D86800E@help.trusthosting.net>
	 <99435f5b0706261630s2a5bef65sb7ab0e0c3c973d9e@mail.gmail.com>
	 <20070626233515.2403486800E@help.trusthosting.net>
Message-ID: <99435f5b0706261703r3ee1656ag9ffc1342cad049e2@mail.gmail.com>

On 27/06/07, Max Kanat-Alexander <mkanat at bugzilla.org> wrote:
> On Wed, 27 Jun 2007 09:30:33 +1000 "Bradley Baetz" <bbaetz at acm.org>
> wrote:
> > Postgres supports functional indexes. So you create an index on
> > UPPER(column) and then a search WHERE UPPER(column) = 'FOO' will use
> > the index.
>
>         Yes, I know. We use those already in a few cases. The problem
> isn't that--the problem is modifying all of Bugzilla to use
> $dbh->sql_istrcmp, which would also be ugly and hard to maintain.

Where apart from Search.pm needs to be changed, though? And even then
most of those sort of queries can be made to be fulltext queries
anyway.

Most other things (eg group names, email addresses) could be case
folded on input, although people may not like that for emails.

Bradley


From aaron.trevena at gmail.com  Wed Jun 27 09:33:20 2007
From: aaron.trevena at gmail.com (Aaron Trevena)
Date: Wed, 27 Jun 2007 10:33:20 +0100
Subject: FullText Searching: The Dilemma
In-Reply-To: <20070625224300.63B13868002@help.trusthosting.net>
References: <20070625224300.63B13868002@help.trusthosting.net>
Message-ID: <edcafa7d0706270233w54cb5b97n1c368a2c808403df@mail.gmail.com>

On 25/06/07, Max Kanat-Alexander <mkanat at bugzilla.org> wrote:
>         Does anybody have any ideas in this department, as to how we
> could do this whole full-text searching better? In a way that would
> perform well, be easy to develop, and work well with our current search
> architecture?

Is there a good reason for not implementing a reverse-index search
yourself within bugzilla ?

It's then trivial to join in to any other query conditions, doesn't
require any additional dependancies, and will work accross databases
in the same way.

A.

-- 
http://www.aarontrevena.co.uk
LAMP System Integration, Development and Hosting


From gerv at mozilla.org  Thu Jun 28 12:10:49 2007
From: gerv at mozilla.org (Gervase Markham)
Date: Thu, 28 Jun 2007 13:10:49 +0100
Subject: How To Ask Good Questions
Message-ID: <4683A549.8070506@mozilla.org>

In mozilla.support.bugzilla, we have recently been getting a large 
quantity of (what I see as) low quality requests for help.

I sometimes point people to:
http://www.catb.org/~esr/faqs/smart-questions.html
but it's very long, and hard to read and digest, particulary if English 
is not your native language.

So, I have come up with this:
http://www.gerv.net/hacking/how-to-ask-good-questions/

What do people think of the page?

I am eager to keep it to the most important things, so suggestions for 
additions will need good rationale.

And more generally, is it worth starting to point more people to a 
resource like this earlier in the process?

Gerv


From dwilliss at microimages.com  Thu Jun 28 20:09:56 2007
From: dwilliss at microimages.com (Dave Williss)
Date: Thu, 28 Jun 2007 15:09:56 -0500
Subject: [Fwd: Re: [all-list] Status Witeboard in bugzilla]
Message-ID: <46841594.3010003@microimages.com>

In the no good deed goes unpunished category...

We used to use the Status Whiteboard to keep track of which clients had 
reported a particular bug so that our support staff could notify them 
when it's fixed.  I transferred this to a custom field so that support 
could set it when creating the bug and also so that it could be called 
"Clients" instead of "Whiteboard".

Now they want it to be more like the CC list, but pulling from a 
different group of users than the CC list. The main reason for 
separating the lists is that our bugzilla is on a private LAN with no 
connection to the outside world, so we can't just have bugzilla email 
them directly.

Is there any plan to expand custom fields to allow this sort of thing or 
should I tell them it just ain't gonna happen?

Dave Williss
MicroImages, Inc.

-------- Original Message --------

Thanks.  Thats an improvement.  Not to be greedy, but is there any way 
to include client name and email address?  You could list multiple 
clients similar to the way you do QA Contact.
 
-Cindy
-----Original Message-----
*From:* all-list-bounces at microimages.com 
[mailto:all-list-bounces at microimages.com]*On Behalf Of *Dave Williss
*Sent:* Thu, June 28, 2007 2:35 PM
*To:* Everybody
*Subject:* [all-list] Status Witeboard in bugzilla

I have added a custom field called "Clients" to our bugzilla and 
transferred the contents of the Status Whiteboard field to it.

There are two advantages to this.
1. Being a custom field, I was able to click a toggle to allow it to be 
set on creation of new bugs.
2. The name makes sense for what we use it for.

The Status Whiteboard field is still there but it's now blank.  We can 
now use it for the purpose for which it was intended, which is just a 
place for quick notes about the current status of the bug.  



From gerv at mozilla.org  Thu Jun 28 20:24:13 2007
From: gerv at mozilla.org (Gervase Markham)
Date: Thu, 28 Jun 2007 21:24:13 +0100
Subject: [Fwd: Re: [all-list] Status Witeboard in bugzilla]
In-Reply-To: <46841594.3010003@microimages.com>
References: <46841594.3010003@microimages.com>
Message-ID: <468418ED.1000003@mozilla.org>

Dave Williss wrote:
> We used to use the Status Whiteboard to keep track of which clients had 
> reported a particular bug so that our support staff could notify them 
> when it's fixed.  I transferred this to a custom field so that support 
> could set it when creating the bug and also so that it could be called 
> "Clients" instead of "Whiteboard".
> 
> Now they want it to be more like the CC list, but pulling from a 
> different group of users than the CC list. The main reason for 
> separating the lists is that our bugzilla is on a private LAN with no 
> connection to the outside world, so we can't just have bugzilla email 
> them directly.

Keywords is a good field to use for this sort of thing, if your customer 
set is manageable. It seems to have roughly the semantics you want.

Gerv



From justdave at bugzilla.org  Fri Jun 29 01:45:27 2007
From: justdave at bugzilla.org (David Miller)
Date: Thu, 28 Jun 2007 21:45:27 -0400
Subject: How To Ask Good Questions
In-Reply-To: <4683A549.8070506@mozilla.org>
References: <4683A549.8070506@mozilla.org>
Message-ID: <46846437.8020808@bugzilla.org>

Gervase Markham wrote on 6/28/07 8:10 AM:
> In mozilla.support.bugzilla, we have recently been getting a large
> quantity of (what I see as) low quality requests for help.
> 
> I sometimes point people to:
> http://www.catb.org/~esr/faqs/smart-questions.html
> but it's very long, and hard to read and digest, particulary if English
> is not your native language.
> 
> So, I have come up with this:
> http://www.gerv.net/hacking/how-to-ask-good-questions/
> 
> What do people think of the page?
> 
> I am eager to keep it to the most important things, so suggestions for
> additions will need good rationale.
> 
> And more generally, is it worth starting to point more people to a
> resource like this earlier in the process?

I think this is great.  I've added a link to it to the text that is
mailed to non-subscribers who post to the support-bugzilla list via
email. (The one that tells them they need to subscribe before they can
post and warns them that it's a public forum and not a private support
channel).

That text now reads:

-----8<-----
Your message to the support-bugzilla mailing list was automatically
rejected. In order to prevent confusion from users, thinking the
support-bugzilla list address is a one-on-one support venue, non-member
posting has been turned off.

Messages to the support-bugzilla mailing list are in public (including
each sender's email address), and archived on Google Groups, at
http://groups.google.com/group/mozilla.support.bugzilla .

To post to the support-bugzilla mailing list, you must subscribe to the
mailing list. You can do so (for free, of course) by going to
https://lists.mozilla.org/listinfo/support-bugzilla .

Enter your email address (your posts will be in public).

Create a password.

Re-enter the password.

Then click on "Subscribe".

A message, asking you to confirm your subscription, will be sent to the
address you entered. Click on the link in the confirmation message. In
the resulting web page, click on "Subscribe to list support-bugzilla".

Before posting your question to the list, please take a look at
http://www.gerv.net/hacking/how-to-ask-good-questions/ for some guidance
on how to get your questions answered the most quickly and completely.

*This is an automated message*. If you have any questions about the
support-bugzilla mailing list, send them to
%(listowner)s .
-----8<-----

-- 
Dave Miller                                   http://www.justdave.net/
System Administrator, Mozilla Corporation      http://www.mozilla.com/
Project Leader, Bugzilla Bug Tracking System  http://www.bugzilla.org/


From mkanat at bugzilla.org  Fri Jun 29 03:42:01 2007
From: mkanat at bugzilla.org (Max Kanat-Alexander)
Date: Thu, 28 Jun 2007 20:42:01 -0700
Subject: How To Ask Good Questions
In-Reply-To: <4683A549.8070506@mozilla.org>
References: <4683A549.8070506@mozilla.org>
Message-ID: <20070629034618.3EBEB3FC11A@help.trusthosting.net>

On Thu, 28 Jun 2007 13:10:49 +0100 Gervase Markham <gerv at mozilla.org>
wrote:
> So, I have come up with this:
> http://www.gerv.net/hacking/how-to-ask-good-questions/
> 
> What do people think of the page?

	I think it's great. I like it way more than ESR's page, because
it's a readable length.

	-Max
-- 
http://www.everythingsolved.com/
Competent, Friendly Bugzilla Services. And Everything Else, too.


From justdave at bugzilla.org  Sat Jun 30 17:19:18 2007
From: justdave at bugzilla.org (David Miller)
Date: Sat, 30 Jun 2007 13:19:18 -0400
Subject: Bugzilla @ OSCON 2007
Message-ID: <46869096.1010709@bugzilla.org>

This was posted a few days ago on
http://www.justdave.net/dave/2007/06/25/bugzilla-oscon-2007/ but I'm
reposting here because I've gotten a lack of response.  It'd be
disappointing to have to do this all by myself. :)

We?re hoping to get some Bugzilla folks hanging out at the Mozilla booth
at OSCON this year. Mozilla is blocking off a number of 2 hour slots
throughout Wednesday and Thursday where they?ll be advertising specific
topics to be discussed at the booth, and we?ll be doing at least one of
those, and would like as many Bugzilla folks as possible there during
that time slot (probably 3:30 - 5:30pm on Thursday, but yet to be
determined). It would also be nice to generally have at least one
Bugzilla person there throughout both days (doesn?t have to be the same
person the entire time ;) ). If you?re planning to be at OSCON and are
willing to help out with staffing the booth, let me know.

-- 
Dave Miller                                   http://www.justdave.net/
System Administrator, Mozilla Corporation      http://www.mozilla.com/
Project Leader, Bugzilla Bug Tracking System  http://www.bugzilla.org/