From mkanat at bugzilla.org Tue Aug 1 00:19:21 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Mon, 31 Jul 2006 17:19:21 -0700 Subject: landfill /var/www/html clean-up Message-ID: <1154391561.2432.7.camel@localhost.localdomain> There were a lot of old test installations sitting around in landfill's /var/www/html directory. If they were old and unused (I checked by last modification date, and also if they were about closed bugs), they were moved to the archive/ directory. If for some reason anybody still needs those installations, just move then back out of /var/www/html/archive. -Max -- http://www.everythingsolved.com/ Competent, Friendly Bugzilla Services. And Everything Else, too. From kevin.benton at amd.com Tue Aug 1 23:59:30 2006 From: kevin.benton at amd.com (Benton, Kevin) Date: Tue, 1 Aug 2006 16:59:30 -0700 Subject: perltidy Message-ID: I'm wondering how others in this list feel about changing our entrance criteria for new patches to include that the submitting developer should run perltidy on the code prior to submitting it. What do others think? I currently run it with... perltidy -npro -syn -nce -i=4 -l=78 -lp -cti=1 -bar -sbl -pt=1 -sbt=1 -bbt=1 -nsfs -olc -nolq -sil=0 -nola -nokw -ndsm -aws -dws -skp -nsfp -ntqw -ibc -hsc -csc -csct=40 -csce=1 -cscw -fs -nbl -nsbl -vt=1 -vtc=0 -nsot -nsct -wbb "% + - * / x != == >= <= =~ !~ < > | & >= < = **= += *= &= <<= &&= -= /= |= >>= ||= .= %= ^= x=" -boc -cab=1 -bol -bok -bot -w filename Yes, it's a huge command line, but that's easily fixed by creating a .perltidyrc (note the leading .). If nothing else, it would help us look a bit more closely at our formatting and deviate less from certain standards. This method would also automatically end comments at the end of blocks when the ending was "far" away from the opening, but would not overwrite comments at the end of a block. Far (in this case) is defined as 6 lines or more away from the opening. By no means do I consider this to be perfect, however, I think it's a great step in the right direction in helping us to get a lot of consistency out of our code formatting and it emphasizes the vast majority of what Perl Best Practices (Conway, 2005) talks about. --- Kevin Benton Perl/Bugzilla Developer/Administrator, Perforce SCM Administrator AMD - ECSD Software Validation and Tools ? The opinions stated in this communication do not necessarily reflect the view of Advanced Micro Devices and have not been reviewed by management.? This communication may contain sensitive and/or confidential and/or proprietary information.? Distribution of such information is strictly prohibited without prior consent of Advanced Micro Devices.? This communication is for the intended recipient(s) only.? If you have received this communication in error, please notify the sender, then destroy any remaining copies of this communication. From mkanat at bugzilla.org Wed Aug 2 00:55:03 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Tue, 01 Aug 2006 17:55:03 -0700 Subject: perltidy In-Reply-To: References: Message-ID: <1154480103.2532.26.camel@localhost.localdomain> On Tue, 2006-08-01 at 16:59 -0700, Benton, Kevin wrote: > I'm wondering how others in this list feel about changing our entrance > criteria for new patches to include that the submitting developer > should run perltidy on the code prior to submitting it. It's not such a bad idea. I just ran perltidy on some of our existing files, though, and I don't totally like the output in some cases. Of course, I ran a version that must be older than the version you ran, because my version said that some of the specified switches were invalid. (The latest version available for Fedora Extras is 20031021.) For example, try running it on Bugzilla/Bug.pm and see what it outputs. In a lot of cases, it really does improve the formatting. It also makes a few mistakes in places, or changes the formatting in a way I don't think is great. I'm sure these can be changed by the appropriate command line, but I don't know them: * It adds spaces after parentheses, where we normally don't. * It adds comments like "# end unless (blah)" even to really short blocks. It's nice for long blocks, but it's unnecessary for subs and short blocks. * It reformats some $dbh-> statements in a really strange way. :-) * It violates perlstyle in that it puts the ending brace of a multi-line "if" statement on the end of the statement, and not on the next line aligned with the closing brace. It does a lot of other good reformatting, though. I picked Bugzilla/Bug.pm on purpose because it has a lot of old code, and it improved parts of it. -Max -- http://www.everythingsolved.com/ Competent, Friendly Bugzilla Services. And Everything Else, too. From kevin.benton at amd.com Wed Aug 2 15:47:45 2006 From: kevin.benton at amd.com (Benton, Kevin) Date: Wed, 2 Aug 2006 08:47:45 -0700 Subject: perltidy Message-ID: > On Tue, 2006-08-01 at 16:59 -0700, Benton, Kevin wrote: > > I'm wondering how others in this list feel about changing our entrance > > criteria for new patches to include that the submitting developer > > should run perltidy on the code prior to submitting it. > > It's not such a bad idea. > > I just ran perltidy on some of our existing files, though, and I > don't > totally like the output in some cases. Of course, I ran a version that > must be older than the version you ran, because my version said that > some of the specified switches were invalid. (The latest version > available for Fedora Extras is 20031021.) To get the latest, simply do... cpan Perl::Tidy and that will bring you up-to-date with 20060719. Those extra switches make a big difference. You may want to suggest adding Perl::Tidy to base FC support since it's such valuable tool. If nothing else, FC developers ought to consider upgrading to a much more current version since 20031021 is nearly three years old. > For example, try running it on Bugzilla/Bug.pm and see what it > outputs. > In a lot of cases, it really does improve the formatting. It also makes > a few mistakes in places, or changes the formatting in a way I don't > think is great. I'm sure these can be changed by the appropriate command > line, but I don't know them: > > * It adds spaces after parentheses, where we normally don't. That's the -pt option. The -pt=n or --paren-tightness=n parameter controls the space within parens. The example below shows the effect of the three possible values, 0, 1, and 2: if ( ( my $len_tab = length( $tabstr ) ) > 0 ) { # -pt=0 if ( ( my $len_tab = length($tabstr) ) > 0 ) { # -pt=1 (default) if ((my $len_tab = length($tabstr)) > 0) { # -pt=2 When n is 0, there is always a space to the right of a '(' and to the left of a ')'. For n=2 there is never a space. For n=1, the default, there is a space unless the quantity within the parens is a single token, such as an identifier or quoted string. ... > * It adds comments like "# end unless (blah)" even to really short > blocks. It's nice for long blocks, but it's unnecessary for subs and > short blocks. To set the length for ## end if (blah) blocks, that's -csci with a default of six lines. -csci=n, or --closing-side-comment-interval=n where "n" is the minimum number of lines that a block must have in order for a closing side comment to be added. The default value is "n=6". To illustrate: # perltidy -csci=2 -csc sub message { if ( !defined( $_[0] ) ) { print("Hello, World\n"); } ## end if ( !defined( $_[0] )) else { print( $_[0], "\n" ); } ## end else [ if ( !defined( $_[0] )) } ## end sub message Now the "if" and "else" blocks are commented. However, now this has become very cluttered. > * It reformats some $dbh-> statements in a really strange way. :-) -fs, --format-skipping This flag, which is enabled by default, causes any code between special beginning and ending comment markers to be passed to the output without formatting. The default beginning marker is #<<< and the default ending marker is #>>> but they may be changed (see next items below). Additional text may appear on these special comment lines provided that it is separated from the marker by at least one space. For example #<<< do not let perltidy touch this my @list = (1, 1, 1, 1, 2, 1, 1, 3, 3, 1, 1, 4, 6, 4, 1,); #>>> The comment markers may be placed at any location that a block com- ment may appear. If they do not appear to be working, use the -log flag and examine the .LOG file. Use -nfs to disable this feature. > * It violates perlstyle in that it puts the ending brace of a > multi-line "if" statement on the end of the statement, and not on the > next line aligned with the closing brace. I'm trying to visualize what you're saying - a multi-line if meaning that the conditionals take up more than one line or that the statements after take up more than one line or both? I'm basing my style choices primarily on Conway's Perl Best Practices with a few variances. > It does a lot of other good reformatting, though. I picked > Bugzilla/Bug.pm on purpose because it has a lot of old code, and it > improved parts of it. I agree. I picked buglist.cgi because it's just plain hairy. As Conway put it, I think that once we come up with a fixed set of params for perltidy, a lot of the style arguments will "go away" because we simply follow one standard and use software to enforce it. I think it'll also make reviewing code a lot easier because it'll be relatively easy to see if someone is following our style guidelines. It'll also put an emphasis on operations making it easier to spot bugs. For example... if ($somevar + $someothervar * $yetanothervar / 3) ... emphasizes the operations being done, making it easier to spot bugs. This configuration also enforces a max line length preventing users from creating exceedingly long lines. That also helps reviews go faster. --- Kevin Benton Perl/Bugzilla Developer/Administrator, Perforce SCM Administrator AMD - ECSD Software Validation and Tools The opinions stated in this communication do not necessarily reflect the view of Advanced Micro Devices and have not been reviewed by management. This communication may contain sensitive and/or confidential and/or proprietary information. Distribution of such information is strictly prohibited without prior consent of Advanced Micro Devices. This communication is for the intended recipient(s) only. If you have received this communication in error, please notify the sender, then destroy any remaining copies of this communication. From kevin.benton at amd.com Wed Aug 2 17:32:26 2006 From: kevin.benton at amd.com (Benton, Kevin) Date: Wed, 2 Aug 2006 10:32:26 -0700 Subject: perltidy Message-ID: > > On Tue, 2006-08-01 at 16:59 -0700, Benton, Kevin wrote: > > > I'm wondering how others in this list feel about changing our > entrance > > > criteria for new patches to include that the submitting developer > > > should run perltidy on the code prior to submitting it. > > > > It's not such a bad idea. > > > > I just ran perltidy on some of our existing files, though, and I > > don't > > totally like the output in some cases. Of course, I ran a version that > > must be older than the version you ran, because my version said that > > some of the specified switches were invalid. (The latest version > > available for Fedora Extras is 20031021.) > > To get the latest, simply do... > > cpan Perl::Tidy > > and that will bring you up-to-date with 20060719. Those extra switches > make a big difference. You may want to suggest adding Perl::Tidy to > base FC support since it's such valuable tool. If nothing else, FC > developers ought to consider upgrading to a much more current version > since 20031021 is nearly three years old. > > > For example, try running it on Bugzilla/Bug.pm and see what it > > outputs. > > In a lot of cases, it really does improve the formatting. It also > makes > > a few mistakes in places, or changes the formatting in a way I don't > > think is great. I'm sure these can be changed by the appropriate > command > > line, but I don't know them: > > > > * It adds spaces after parentheses, where we normally don't. > > That's the -pt option. > > The -pt=n or --paren-tightness=n parameter controls the space > within parens. The example below shows the effect of the three > possible values, 0, 1, and 2: > > if ( ( my $len_tab = length( $tabstr ) ) > 0 ) { # -pt=0 > if ( ( my $len_tab = length($tabstr) ) > 0 ) { # -pt=1 > (default) > if ((my $len_tab = length($tabstr)) > 0) { # -pt=2 > > When n is 0, there is always a space to the right of a '(' and to > the left of a ')'. For n=2 there is never a space. For n=1, the > default, there is a space unless the quantity within the parens > is > a single token, such as an identifier or quoted string. > > ... > > > * It adds comments like "# end unless (blah)" even to really > short > > blocks. It's nice for long blocks, but it's unnecessary for subs and > > short blocks. > > To set the length for ## end if (blah) blocks, that's -csci with a > default of six lines. > > -csci=n, or --closing-side-comment-interval=n > where "n" is the minimum number of lines that a block must > have > in order for a closing side comment to be added. The default > value is "n=6". To illustrate: > > # perltidy -csci=2 -csc > sub message { > if ( !defined( $_[0] ) ) { > print("Hello, World\n"); > } ## end if ( !defined( $_[0] )) > else { > print( $_[0], "\n" ); > } ## end else [ if ( !defined( $_[0] )) > } ## end sub message > > Now the "if" and "else" blocks are commented. However, now > this has become very cluttered. > > > * It reformats some $dbh-> statements in a really strange way. > :-) > > -fs, --format-skipping > This flag, which is enabled by default, causes any code between > special beginning and ending comment markers to be passed to the > output without formatting. The default beginning marker is #<<< > and the default ending marker is #>>> but they may be changed > (see > next items below). Additional text may appear on these special > comment lines provided that it is separated from the marker by at > least one space. For example > > #<<< do not let perltidy touch this > my @list = (1, > 1, 1, > 1, 2, 1, > 1, 3, 3, 1, > 1, 4, 6, 4, 1,); > #>>> > > The comment markers may be placed at any location that a block > com- > ment may appear. If they do not appear to be working, use the > -log > flag and examine the .LOG file. Use -nfs to disable this > feature. > > > * It violates perlstyle in that it puts the ending brace of a > > multi-line "if" statement on the end of the statement, and not on the > > next line aligned with the closing brace. > > I'm trying to visualize what you're saying - a multi-line if meaning > that the conditionals take up more than one line or that the statements > after take up more than one line or both? I'm basing my style choices > primarily on Conway's Perl Best Practices with a few variances. > > > It does a lot of other good reformatting, though. I picked > > Bugzilla/Bug.pm on purpose because it has a lot of old code, and it > > improved parts of it. > > I agree. I picked buglist.cgi because it's just plain hairy. > > As Conway put it, I think that once we come up with a fixed set of > params for perltidy, a lot of the style arguments will "go away" because > we simply follow one standard and use software to enforce it. I think > it'll also make reviewing code a lot easier because it'll be relatively > easy to see if someone is following our style guidelines. It'll also > put an emphasis on operations making it easier to spot bugs. > > For example... > > if ($somevar > + $someothervar > * $yetanothervar > / 3) > > ... emphasizes the operations being done, making it easier to spot bugs. > > This configuration also enforces a max line length preventing users from > creating exceedingly long lines. That also helps reviews go faster. I'm including the options dump from perltidy so you can see the more verbose listing of parameters I'm using right now. As I mentioned previously, I think there are things that can be improved here, but here's what I use today: $ perltidy -dop | grep -v dump # Final parameter set for this run: --add-newlines --add-semicolons --add-whitespace --backup-file-extension="bak" --blanks-before-blocks --blanks-before-comments --blanks-before-subs --block-brace-tightness=1 --block-brace-vertical-tightness=0 --nobrace-left-and-indent --brace-tightness=1 --brace-vertical-tightness=1 --brace-vertical-tightness-closing=0 --break-at-old-comma-breakpoints --break-at-old-keyword-breakpoints --break-at-old-logical-breakpoints --break-at-old-ternary-breakpoints --check-syntax --closing-brace-indentation=1 --closing-paren-indentation=1 --closing-side-comment-else-flag=1 --closing-side-comment-interval=6 --closing-side-comment-maximum-text=30 --closing-side-comment-warnings --closing-side-comments --closing-square-bracket-indentation=1 --comma-arrow-breakpoints=1 --continuation-indentation=2 --nocuddled-else --delete-old-newlines --delete-old-whitespace --nodelete-semicolons --format="tidy" --format-skipping --fuzzy-line-length --hanging-side-comments --nohtml --html-entities --html-table-of-contents --indent-block-comments --indent-columns=4 --line-up-parentheses --nologfile --long-block-line-count=8 --look-for-autoloader --look-for-selfloader --maximum-consecutive-blank-lines=1 --maximum-fields-per-table=0 --maximum-line-length=78 --minimum-space-to-comment=4 --opening-brace-always-on-right --noopening-brace-on-new-line --noopening-sub-brace-on-new-line --nooutdent-keywords --nooutdent-labels --outdent-long-comments --nooutdent-long-quotes --paren-tightness=1 --paren-vertical-tightness=1 --paren-vertical-tightness-closing=0 --pass-version-line --perl-syntax-check-flags="-c -T" --pod2html --noquiet --recombine --short-concatenation-item-length=8 --noshow-options --nospace-for-semicolon --nospace-function-paren --space-keyword-paren --square-bracket-tightness=1 --square-bracket-vertical-tightness=1 --square-bracket-vertical-tightness-closing=0 --nostack-closing-hash-brace --nostack-closing-paren --nostack-closing-square-bracket --nostack-opening-hash-brace --nostack-opening-paren --nostack-opening-square-bracket --starting-indentation-level=0 --static-block-comments --nostatic-side-comments --noswallow-optional-blank-lines --notabs --notrim-qw --valign --want-break-before="% + - * / x != == >= <= =~ !~ < > | & >= < = **= += *= &= <<= &&= -= /= |= >>= ||= .= %= ^= x=" --warning-output --- Kevin Benton Perl/Bugzilla Developer/Administrator, Perforce SCM Administrator AMD - ECSD Software Validation and Tools The opinions stated in this communication do not necessarily reflect the view of Advanced Micro Devices and have not been reviewed by management. This communication may contain sensitive and/or confidential and/or proprietary information. Distribution of such information is strictly prohibited without prior consent of Advanced Micro Devices. This communication is for the intended recipient(s) only. If you have received this communication in error, please notify the sender, then destroy any remaining copies of this communication. From mkanat at bugzilla.org Thu Aug 3 02:18:03 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Wed, 02 Aug 2006 19:18:03 -0700 Subject: checksetup.pl changes Message-ID: <1154571484.2455.46.camel@localhost.localdomain> Hey hey. So, checksetup.pl is basically a completely different file than it was three or four days ago. Here's what you need to know: Changes to the filesystem go in: Bugzilla::Install::Filesystem You'll see a subroutine there called FILESYSTEM that you can modify. If you want to delete a file or directory, you can modify the update_filesystem subroutine. You'll see that it already deletes some things, just add your thing after that. Changes to localconfig go into: Bugzilla::Install::Localconfig Adding a new variable is as simple as adding a new entry to the LOCALCONFIG_VARS constant. All new vars must be scalars. (They can be arrayrefs or hashrefs. They just shouldn't directly be arrays or hashes.) If you remove a variable from localconfig, make sure to put it into the OLD_LOCALCONFIG_VARS constant. Changes to Bugzilla's required perl modules go in: Bugzilla::Install::Requirements Changes to the database go into: Bugzilla::Install::DB In particular, the update_table_definitions function. You'll notice this function looks much nicer than the old checksetup code. Please keep it that way! :-) Read the comment at the beginning of the function to understand how it's supposed to work. You should also check out the indicate_progress function if you're going to make the code do something that takes a long time. Settings now live in Bugzilla::Install. All of the above files have POD, so you can read it. checksetup.pl itself also has some good POD. A few other changes will be happening. The initial creation of Groups will move into Bugzilla::Install, the updating of fielddefs will move...somewhere, and the creation of the admin will move into Bugzilla::Install. Basically, any raw code in checksetup will move into modules. -Max -- http://www.everythingsolved.com/ Competent, Friendly Bugzilla Services. And Everything Else, too. From mkanat at bugzilla.org Sat Aug 5 02:39:31 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Fri, 04 Aug 2006 19:39:31 -0700 Subject: Possibly moving to InnoDB Message-ID: <1154745571.2440.9.camel@localhost.localdomain> Hey guys. I'm probably going to have Bugzilla start using the InnoDB storage engine in MySQL for all tables except "bugs" and "longdescs," because those have FULLTEXT indexes (and InnoDB doesn't support FULLTEXT indexes). Before I do this, I just wanted to ask if anybody had any particular reasons that we shouldn't do it, or if there's anything we should warn users about after we switch. That is, if anybody has any experience working with InnoDB in large installations, I'd appreciate knowing about it. :-) -Max -- http://www.everythingsolved.com/ Competent, Friendly Bugzilla Services. And Everything Else, too. From bbaetz at acm.org Sat Aug 5 03:07:54 2006 From: bbaetz at acm.org (Bradley Baetz) Date: Sat, 5 Aug 2006 13:07:54 +1000 Subject: Possibly moving to InnoDB In-Reply-To: <1154745571.2440.9.camel@localhost.localdomain> References: <1154745571.2440.9.camel@localhost.localdomain> Message-ID: <99435f5b0608042007h4acfafefxa91911af728c5e71@mail.gmail.com> Warning - semi-rant follows: ;) Its *insanely* slow, although that may be because the on disk size is 2-3 times as large. You need to be careful tuning the innodb cache/buffer/etc sizes, too. Also, there are lots of deadlock problems (documented as 'features'), mostly to do with next key locking (which is also documented as a feature) eg http://www.xaprb.com/blog/2006/08/03/a-little-known-way-to-cause-a-database-deadlock/ http://bugs.mysql.com/bug.php?id=20134 just search through their bug database for 'not a bug' bugs with deadlock in the title. You can turn that off, but then you break replication unless you use 5.1 and row-level replication. They also have a really really annoying deadlock detection setup - if a query takes too long, its assumed to be due to a deadlock. So if you have any long running query (and buglist.cgi does!), a concurrent insert (which has to block if the select scans a row 'near' it, due to next-key locking) may assume its in a deadlock. This is really painful in replication setups, so there are a couple of options for the replication thread to retry a few times (eww!) Which helps a fair bit, until you run a really really long query on the slave.... Also, be wary of mixing table types in queries, especially with locking. On the flip side, I've yet to see corrupt tables with innodb. I've yet to see them with postgres either, though.... Bradley On 05/08/06, Max Kanat-Alexander wrote: > Hey guys. I'm probably going to have Bugzilla start using the InnoDB > storage engine in MySQL for all tables except "bugs" and "longdescs," > because those have FULLTEXT indexes (and InnoDB doesn't support FULLTEXT > indexes). > > Before I do this, I just wanted to ask if anybody had any particular > reasons that we shouldn't do it, or if there's anything we should warn > users about after we switch. That is, if anybody has any experience > working with InnoDB in large installations, I'd appreciate knowing about > it. :-) > > -Max From justdave at bugzilla.org Sat Aug 5 03:59:09 2006 From: justdave at bugzilla.org (David Miller) Date: Fri, 04 Aug 2006 23:59:09 -0400 Subject: checksetup.pl changes In-Reply-To: <1154571484.2455.46.camel@localhost.localdomain> References: <1154571484.2455.46.camel@localhost.localdomain> Message-ID: <44D4178D.9090508@bugzilla.org> Max Kanat-Alexander wrote on 8/2/06 10:18 PM: > Hey hey. So, checksetup.pl is basically a completely different file > than it was three or four days ago. Here's what you need to know: OK... one thing that worries me a little about this.... I seem to recall their being database changes that got triggered by the existence (or non-existence) of certain files on the filesystem, and/or some filesystem changes that depended on certain schema changes having not happened yet. a) am I really remembering that right? b) if so, how do you account for that with filesystem and database changes in different places? -- Dave Miller http://www.justdave.net/ System Administrator, Mozilla Corporation http://www.mozilla.com/ Project Leader, Bugzilla Bug Tracking System http://www.bugzilla.org/ From bugreport at peshkin.net Sat Aug 5 04:31:35 2006 From: bugreport at peshkin.net (Joel Peshkin) Date: Fri, 04 Aug 2006 21:31:35 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <1154745571.2440.9.camel@localhost.localdomain> References: <1154745571.2440.9.camel@localhost.localdomain> Message-ID: <44D41F27.4040301@peshkin.net> Max Kanat-Alexander wrote: > Hey guys. I'm probably going to have Bugzilla start using the InnoDB > storage engine in MySQL for all tables except "bugs" and "longdescs," > because those have FULLTEXT indexes (and InnoDB doesn't support FULLTEXT > indexes). > That sounds like one reason not too... I am rather leery of mixing engines. What is the compelling reason to change? From mkanat at bugzilla.org Sat Aug 5 07:22:44 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Sat, 05 Aug 2006 00:22:44 -0700 Subject: checksetup.pl changes In-Reply-To: <44D4178D.9090508@bugzilla.org> References: <1154571484.2455.46.camel@localhost.localdomain> <44D4178D.9090508@bugzilla.org> Message-ID: <1154762564.2511.2.camel@es-lappy> On Fri, 2006-08-04 at 23:59 -0400, David Miller wrote: > I seem to recall their being database changes that got triggered by the > existence (or non-existence) of certain files on the filesystem, and/or > some filesystem changes that depended on certain schema changes having > not happened yet. > > a) am I really remembering that right? Yep, there are two DB changes that depend on the filesystem. Importing Old Charts into New Charts and moving data/nomail into the DB. > b) if so, how do you account for that with filesystem and database > changes in different places? They happen during the DB phase. They're really DB changes, so Bugzilla::Install::DB takes care of them. The filesystem creation stuff happens before the DB changes, so if any DB change depends on a file's *existence*, then that's already handled. If no DB change depends on a file, then it can safely be deleted in Bugzilla::Install::Filesystem, and usually is. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From mkanat at bugzilla.org Sat Aug 5 07:23:29 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Sat, 05 Aug 2006 00:23:29 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <44D41F27.4040301@peshkin.net> References: <1154745571.2440.9.camel@localhost.localdomain> <44D41F27.4040301@peshkin.net> Message-ID: <1154762610.2511.4.camel@es-lappy> On Fri, 2006-08-04 at 21:31 -0700, Joel Peshkin wrote: > That sounds like one reason not too... I am rather leery of mixing > engines. What is the compelling reason to change? They support referential integrity and transactions. If we don't switch to them, we'll never support transactions in Bugzilla. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From jochen.wiedmann at gmail.com Sat Aug 5 10:59:18 2006 From: jochen.wiedmann at gmail.com (Jochen Wiedmann) Date: Sat, 05 Aug 2006 12:59:18 +0200 Subject: Possibly moving to InnoDB In-Reply-To: <1154745571.2440.9.camel@localhost.localdomain> References: <1154745571.2440.9.camel@localhost.localdomain> Message-ID: <44D47A06.8030501@gmail.com> Max Kanat-Alexander wrote: > Hey guys. I'm probably going to have Bugzilla start using the InnoDB > storage engine in MySQL for all tables except "bugs" and "longdescs," > because those have FULLTEXT indexes (and InnoDB doesn't support FULLTEXT > indexes). I may be wrong, but if you'd ask me for the one table, which should take part in transactions, then I'd vote for "bugs"? Reading about the problems with InnoDB: Has anyone considered to use, for example, the new Solid driver? I have worked with SolidDB in the past. The application was really heavy (much more than your average Bugzilla), transactional, and ran really good. Jochen From mkanat at bugzilla.org Sat Aug 5 11:55:16 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Sat, 05 Aug 2006 04:55:16 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <99435f5b0608042007h4acfafefxa91911af728c5e71@mail.gmail.com> References: <1154745571.2440.9.camel@localhost.localdomain> <99435f5b0608042007h4acfafefxa91911af728c5e71@mail.gmail.com> Message-ID: <1154778917.2405.24.camel@es-lappy> On Sat, 2006-08-05 at 13:07 +1000, Bradley Baetz wrote: > Its *insanely* slow, although that may be because the on disk size is > 2-3 times as large. You need to be careful tuning the innodb > cache/buffer/etc sizes, too. Is that just compared to PostgreSQL? And is that on INSERTs or SELECTs, and on what kind of load? Bugzilla's load is not exactly a typical database load, since it's very few INSERTs and long-running complex SELECTs with a lot of simultaneous simple SELECTs. I'd think that once we move to transactions, also, on something the size of bmo anything would be faster than our current architecture, which has to lock the whole table every time we want to do an INSERT. I'd be willing to take a very small performance degrade to get referential integrity and transactions, but I wouldn't accept, say, a 50% degrade in Bugzilla's overall performance. I've also heard that the storage format improved in 5.0, and takes less disk space. I know that LiveJournal and Wikipedia use InnoDB, and so do a lot of other large-scale sites, and because of the reliability issues, they mostly say that they'd choose InnoDB over MyISAM any day, except for tables that need FULLTEXT indexing. > Also, there are lots of deadlock problems (documented as 'features'), > mostly to do with next key locking (which is also documented as a > feature) > [snip] Hrm. We don't have any multi-row INSERTs that a CGI would do, that I can think of, all of our PKs are AUTO_INCREMENT integers, and we never try to insert out-of-order items. DELETEs are very rare, also, in Bugzilla, except in the logincookies table. Thankfully, we also never use SELECT FOR UPDATE or SELECT FOR INSERT, which would also cause the problem, and we only rarely insert items out of PK order. I might be concerned about this for the logincookies table; I'd have to look into it. Perhaps I should write a multi-process Bugzilla load-testing tool anyhow. > They also have a really really annoying deadlock detection setup - if > a query takes too long, its assumed to be due to a deadlock. So if you > have any long running query (and buglist.cgi does!), a concurrent > insert (which has to block if the select scans a row 'near' it, due to > next-key locking) may assume its in a deadlock. Okay. In many cases, that would actually be fine for us, as long as the timeout is long enough. MySQL would just be automatically killing the long-running queries that we kill by hand in places like bmo. > This is really painful in replication setups, Ah, I can imagine that could be difficult. > so there are a couple of options for the > replication thread to retry a few times (eww!) Which helps a fair bit, > until you run a really really long query on the slave.... There shouldn't be any really really long INSERT queries in Bugzilla, though. And if the replicator has to kill a long-running buglist SELECT statement in order to replicate an INSERT, that's fine by me. > Also, be wary of mixing table types in queries, especially with locking. Right, I'll have to look into that more. In an ideal world, I'd like to lock only the longdescs table and whatever table short_desc ends up in (since they both need a FULLTEXT index, although perhaps short_desc could live with a normal index now that it's a varchar), and let the DB engine do row-level locks on the other tables. However, I suspect that isn't possible...? > On the flip side, I've yet to see corrupt tables with innodb. I've yet > to see them with postgres either, though.... Hahaha, yeah. Obviously none of this is a problem with PostgreSQL. :-) -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From mkanat at bugzilla.org Sat Aug 5 12:22:01 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Sat, 05 Aug 2006 05:22:01 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <44D47A06.8030501@gmail.com> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> Message-ID: <1154780522.2405.27.camel@es-lappy> On Sat, 2006-08-05 at 12:59 +0200, Jochen Wiedmann wrote: > I may be wrong, but if you'd ask me for the one table, which should take > part in transactions, then I'd vote for "bugs"? Yeah, it will. We just have to either move short_desc to another table or change its index to a normal index. > Reading about the problems with InnoDB: Has anyone considered to use, > for example, the new Solid driver? [snip] I've never even heard of it. URL? Does it/will it ship with MySQL? -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From bbaetz at acm.org Sat Aug 5 12:29:41 2006 From: bbaetz at acm.org (Bradley Baetz) Date: Sat, 5 Aug 2006 22:29:41 +1000 Subject: Possibly moving to InnoDB In-Reply-To: <1154778917.2405.24.camel@es-lappy> References: <1154745571.2440.9.camel@localhost.localdomain> <99435f5b0608042007h4acfafefxa91911af728c5e71@mail.gmail.com> <1154778917.2405.24.camel@es-lappy> Message-ID: <99435f5b0608050529v51895231i3a8596f89d72b3c7@mail.gmail.com> On 05/08/06, Max Kanat-Alexander wrote: > On Sat, 2006-08-05 at 13:07 +1000, Bradley Baetz wrote: > > Its *insanely* slow, although that may be because the on disk size is > > 2-3 times as large. You need to be careful tuning the innodb > > cache/buffer/etc sizes, too. > > They also have a really really annoying deadlock detection setup - if > > a query takes too long, its assumed to be due to a deadlock. So if you > > have any long running query (and buglist.cgi does!), a concurrent > > insert (which has to block if the select scans a row 'near' it, due to > > next-key locking) may assume its in a deadlock. > > Okay. In many cases, that would actually be fine for us, as long as the > timeout is long enough. MySQL would just be automatically killing the > long-running queries that we kill by hand in places like bmo. It can be tweaked; I forget what the default is. But it doesn't kill the *running* query - it kills the blocking one. (All together now - eww!) > > so there are a couple of options for the > > replication thread to retry a few times (eww!) Which helps a fair bit, > > until you run a really really long query on the slave.... > > There shouldn't be any really really long INSERT queries in Bugzilla, > though. And if the replicator has to kill a long-running buglist SELECT > statement in order to replicate an INSERT, that's fine by me. Nope. Specifically, the case I see is a long-running reconcillation script that basically does: SELECT * FROM tab_a, tab_b WHERE tab_a.id=tab_b.a_id AND tab_a.foo = 2; SELECT * FROM tab_a, tab_c WHERE tab_a.id=tab_c.a_id AND tab_a.foo = 2; (Yes, it does make sense to do this) in a transaction. This takes up to 15-20 minutes to run, pulling a few hundred thousand rows each time. At the same time, theres a replication statment doing: UPDATE tab_a SET bar=1 WHERE id = 12345; That blocks for the 15-20 minutes, then eventually dies, stopping replication. |SLAVE START| is required to fix it. The reason is that the various selects have grabbed a share lock on the row, so the update can't run. Thats fine (well, its not really, but...), except for the timeout. See http://bugs.mysql.com/bug.php?id=8325: Fixed in 4.1.11 and 5.0.3, by introducing the SLAVE_TRANSACTION_RETRIES option (will be documented in the next days); the slave shall be started with --slave-transaction-retries=x where x>0 and then it will retry a failed transaction for x times before giving up. Of course, setting it too high may just hide real problems..... > > Also, be wary of mixing table types in queries, especially with locking. > > Right, I'll have to look into that more. In an ideal world, I'd like to > lock only the longdescs table and whatever table short_desc ends up in > (since they both need a FULLTEXT index, although perhaps short_desc > could live with a normal index now that it's a varchar), and let the DB > engine do row-level locks on the other tables. However, I suspect that > isn't possible...? I think its meant to be, but may interact with manual LOCKs a bit oddly - since you can't LOCK TABLE more than once per transaction, I'm not sure if the 'implicit' lock works. It didn't used to work, but that was a few years ago, and I may be misremembering it. > Hahaha, yeah. Obviously none of this is a problem with PostgreSQL. :-) Oh, postgres has its own issues. They just fix them rather than calling them features.... Bradley From lpsolit at gmail.com Sat Aug 5 12:52:54 2006 From: lpsolit at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBCdWNsaW4=?=) Date: Sat, 05 Aug 2006 14:52:54 +0200 Subject: Possibly moving to InnoDB In-Reply-To: <1154780522.2405.27.camel@es-lappy> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> Message-ID: <44D494A6.6090504@gmail.com> > We just have to either move short_desc to another table > or change its index to a normal index. Moving short_desc to another table really doesn't make sense. From pdemarco at zoominternet.net Sat Aug 5 14:26:02 2006 From: pdemarco at zoominternet.net (Paul) Date: Sat, 05 Aug 2006 10:26:02 -0400 Subject: checksetup.pl changes In-Reply-To: <1154762564.2511.2.camel@es-lappy> References: <1154571484.2455.46.camel@localhost.localdomain> <44D4178D.9090508@bugzilla.org> <1154762564.2511.2.camel@es-lappy> Message-ID: <1154787962.7007.0.camel@mainpc> And quips On Sat, 2006-08-05 at 00:22 -0700, Max Kanat-Alexander wrote: > On Fri, 2006-08-04 at 23:59 -0400, David Miller wrote: > > I seem to recall their being database changes that got triggered by the > > existence (or non-existence) of certain files on the filesystem, and/or > > some filesystem changes that depended on certain schema changes having > > not happened yet. > > > > a) am I really remembering that right? > > Yep, there are two DB changes that depend on the filesystem. Importing > Old Charts into New Charts and moving data/nomail into the DB. > > > b) if so, how do you account for that with filesystem and database > > changes in different places? > > They happen during the DB phase. They're really DB changes, so > Bugzilla::Install::DB takes care of them. > > The filesystem creation stuff happens before the DB changes, so if any > DB change depends on a file's *existence*, then that's already handled. > > If no DB change depends on a file, then it can safely be deleted in > Bugzilla::Install::Filesystem, and usually is. > > -Max From mkanat at bugzilla.org Sat Aug 5 15:18:08 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Sat, 05 Aug 2006 08:18:08 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <99435f5b0608050529v51895231i3a8596f89d72b3c7@mail.gmail.com> References: <1154745571.2440.9.camel@localhost.localdomain> <99435f5b0608042007h4acfafefxa91911af728c5e71@mail.gmail.com> <1154778917.2405.24.camel@es-lappy> <99435f5b0608050529v51895231i3a8596f89d72b3c7@mail.gmail.com> Message-ID: <1154791088.6998.3.camel@es-lappy> On Sat, 2006-08-05 at 22:29 +1000, Bradley Baetz wrote: > > Okay. In many cases, that would actually be fine for us, as long as the > > timeout is long enough. MySQL would just be automatically killing the > > long-running queries that we kill by hand in places like bmo. > > It can be tweaked; I forget what the default is. But it doesn't kill > the *running* query - it kills the blocking one. (All together now - > eww!) But wait, that makes sense. You mean if a SELECT is blocking everybody else from running, it kills the SELECT, right? > in a transaction. This takes up to 15-20 minutes to run, pulling a few > hundred thousand rows each time. Thankfully nothing in Bugzilla does that, except for collectstats.pl --regenerate. > Fixed in 4.1.11 and 5.0.3, by introducing the SLAVE_TRANSACTION_RETRIES option > [snip] Okay. So we should recommend that if people are going to use replication, they use at least 4.1.11 or 5.0.3. > I think its meant to be, but may interact with manual LOCKs a bit > oddly - since you can't LOCK TABLE more than once per transaction, I'm > not sure if the 'implicit' lock works. It didn't used to work, but > that was a few years ago, and I may be misremembering it. I was reading something about it. There's something about table-level locks and the implicit row-level locks caused by transactions. > Oh, postgres has its own issues. They just fix them rather than > calling them features.... Yeah. Sometimes, at least. :-) -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From mkanat at bugzilla.org Sat Aug 5 15:19:17 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Sat, 05 Aug 2006 08:19:17 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <44D494A6.6090504@gmail.com> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> <44D494A6.6090504@gmail.com> Message-ID: <1154791157.6998.6.camel@es-lappy> On Sat, 2006-08-05 at 14:52 +0200, Fr?d?ric Buclin wrote: > Moving short_desc to another table really doesn't make sense. I completely agree, from a database design perspective. If we can make it use a normal index and continue to function well in the full-text search, we'll do that. Otherwise we'll have to move it to its own bugs_short_desc table just for FULLTEXT indexing in MySQL. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From mkanat at bugzilla.org Sat Aug 5 15:19:50 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Sat, 05 Aug 2006 08:19:50 -0700 Subject: checksetup.pl changes In-Reply-To: <1154787962.7007.0.camel@mainpc> References: <1154571484.2455.46.camel@localhost.localdomain> <44D4178D.9090508@bugzilla.org> <1154762564.2511.2.camel@es-lappy> <1154787962.7007.0.camel@mainpc> Message-ID: <1154791190.6998.8.camel@es-lappy> On Sat, 2006-08-05 at 10:26 -0400, Paul wrote: > And quips Right, and quips. That also happens during the database-updates phase. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From bugreport at peshkin.net Sat Aug 5 17:14:50 2006 From: bugreport at peshkin.net (Joel Peshkin) Date: Sat, 05 Aug 2006 10:14:50 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <1154791088.6998.3.camel@es-lappy> References: <1154745571.2440.9.camel@localhost.localdomain> <99435f5b0608042007h4acfafefxa91911af728c5e71@mail.gmail.com> <1154778917.2405.24.camel@es-lappy> <99435f5b0608050529v51895231i3a8596f89d72b3c7@mail.gmail.com> <1154791088.6998.3.camel@es-lappy> Message-ID: <44D4D20A.6020408@peshkin.net> Can we handle the move to InnoDB the same way as any other db-compatibility change? (Support both MyISAM and InnoDB) From justdave at bugzilla.org Sat Aug 5 22:02:08 2006 From: justdave at bugzilla.org (David Miller) Date: Sat, 05 Aug 2006 18:02:08 -0400 Subject: Possibly moving to InnoDB In-Reply-To: <1154780522.2405.27.camel@es-lappy> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> Message-ID: <44D51560.30602@bugzilla.org> Max Kanat-Alexander wrote on 8/5/06 8:22 AM: > On Sat, 2006-08-05 at 12:59 +0200, Jochen Wiedmann wrote: >> I may be wrong, but if you'd ask me for the one table, which should take >> part in transactions, then I'd vote for "bugs"? > > Yeah, it will. We just have to either move short_desc to another table > or change its index to a normal index. > >> Reading about the problems with InnoDB: Has anyone considered to use, >> for example, the new Solid driver? [snip] > > I've never even heard of it. URL? > > Does it/will it ship with MySQL? The Solid guys had a booth at OSCON and were bragging about them hooking up with MySQL... http://news.com.com/M/2100-1012_3-6061443.html http://dev.soliddb.com/ -- Dave Miller http://www.justdave.net/ System Administrator, Mozilla Corporation http://www.mozilla.com/ Project Leader, Bugzilla Bug Tracking System http://www.bugzilla.org/ From jeffh at tragicallyleet.com Sun Aug 6 02:32:02 2006 From: jeffh at tragicallyleet.com (Jeffrey Hulten) Date: Sat, 5 Aug 2006 19:32:02 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <44D51560.30602@bugzilla.org> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> <44D51560.30602@bugzilla.org> Message-ID: <2F6DEDF9-9953-4E42-93E2-81FF29690A06@tragicallyleet.com> The other possibility is to make the indexing of the full text be programmatical instead of a function of the database operation. In other words, index it ourselves. On Aug 5, 2006, at 3:02 PM, David Miller wrote: > Max Kanat-Alexander wrote on 8/5/06 8:22 AM: >> On Sat, 2006-08-05 at 12:59 +0200, Jochen Wiedmann wrote: >>> I may be wrong, but if you'd ask me for the one table, which >>> should take part in transactions, then I'd vote for "bugs"? >> Yeah, it will. We just have to either move short_desc to another >> table >> or change its index to a normal index. >>> Reading about the problems with InnoDB: Has anyone considered to >>> use, for example, the new Solid driver? [snip] >> I've never even heard of it. URL? >> Does it/will it ship with MySQL? > > The Solid guys had a booth at OSCON and were bragging about them > hooking up with MySQL... > > http://news.com.com/M/2100-1012_3-6061443.html > > http://dev.soliddb.com/ > > -- > Dave Miller http://www.justdave.net/ > System Administrator, Mozilla Corporation http://www.mozilla.com/ > Project Leader, Bugzilla Bug Tracking System http://www.bugzilla.org/ > - > To view or change your list settings, click here: > From jochen.wiedmann at gmail.com Sun Aug 6 04:11:04 2006 From: jochen.wiedmann at gmail.com (Jochen Wiedmann) Date: Sun, 06 Aug 2006 06:11:04 +0200 Subject: Possibly moving to InnoDB In-Reply-To: <2F6DEDF9-9953-4E42-93E2-81FF29690A06@tragicallyleet.com> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> <44D51560.30602@bugzilla.org> <2F6DEDF9-9953-4E42-93E2-81FF29690A06@tragicallyleet.com> Message-ID: <44D56BD8.5040502@gmail.com> Dave Miller wrote: > The Solid guys had a booth at OSCON and were bragging about them > hooking up with MySQL... You didn't like them, did you? :-) From justdave at bugzilla.org Sun Aug 6 05:09:52 2006 From: justdave at bugzilla.org (David Miller) Date: Sun, 06 Aug 2006 01:09:52 -0400 Subject: Possibly moving to InnoDB In-Reply-To: <44D56BD8.5040502@gmail.com> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> <44D51560.30602@bugzilla.org> <2F6DEDF9-9953-4E42-93E2-81FF29690A06@tragicallyleet.com> <44D56BD8.5040502@gmail.com> Message-ID: <44D579A0.9090208@bugzilla.org> Jochen Wiedmann wrote on 8/6/06 12:11 AM: > Dave Miller wrote: > >> The Solid guys had a booth at OSCON and were bragging about them >> hooking up with MySQL... > > You didn't like them, did you? :-) I have no opinion, I didn't talk to them, and I haven't investigated their engine yet. That was just a statement of fact. :) They didn't leave me with any kind of negative impression at OSCON if that's what you were inferring. -- Dave Miller http://www.justdave.net/ System Administrator, Mozilla Corporation http://www.mozilla.com/ Project Leader, Bugzilla Bug Tracking System http://www.bugzilla.org/ From mkanat at bugzilla.org Sun Aug 6 07:01:40 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Sun, 06 Aug 2006 00:01:40 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <44D4D20A.6020408@peshkin.net> References: <1154745571.2440.9.camel@localhost.localdomain> <99435f5b0608042007h4acfafefxa91911af728c5e71@mail.gmail.com> <1154778917.2405.24.camel@es-lappy> <99435f5b0608050529v51895231i3a8596f89d72b3c7@mail.gmail.com> <1154791088.6998.3.camel@es-lappy> <44D4D20A.6020408@peshkin.net> Message-ID: <1154847700.2598.0.camel@es-lappy> On Sat, 2006-08-05 at 10:14 -0700, Joel Peshkin wrote: > Can we handle the move to InnoDB the same way as any other > db-compatibility change? (Support both MyISAM and InnoDB) No. We need all databases to support transactions before we can implement transaction support. I'd also ideally like all databases to support referential integrity before I check in referential integrity support. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From mkanat at bugzilla.org Sun Aug 6 07:03:31 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Sun, 06 Aug 2006 00:03:31 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <44D51560.30602@bugzilla.org> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> <44D51560.30602@bugzilla.org> Message-ID: <1154847812.2598.3.camel@es-lappy> On Sat, 2006-08-05 at 18:02 -0400, David Miller wrote: > The Solid guys had a booth at OSCON and were bragging about them hooking > up with MySQL... > > http://news.com.com/M/2100-1012_3-6061443.html > > http://dev.soliddb.com/ Okay. In any case it's a beta, and it looks like it only supports MySQL 5.0.22 and later. So that's an unlikely candidate for us at this time. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From bbaetz at acm.org Sun Aug 6 07:16:40 2006 From: bbaetz at acm.org (Bradley Baetz) Date: Sun, 6 Aug 2006 17:16:40 +1000 Subject: Possibly moving to InnoDB In-Reply-To: <1154791088.6998.3.camel@es-lappy> References: <1154745571.2440.9.camel@localhost.localdomain> <99435f5b0608042007h4acfafefxa91911af728c5e71@mail.gmail.com> <1154778917.2405.24.camel@es-lappy> <99435f5b0608050529v51895231i3a8596f89d72b3c7@mail.gmail.com> <1154791088.6998.3.camel@es-lappy> Message-ID: <99435f5b0608060016h34856796v47db3edd5c49f05@mail.gmail.com> On 06/08/06, Max Kanat-Alexander wrote: > On Sat, 2006-08-05 at 22:29 +1000, Bradley Baetz wrote: > > > Okay. In many cases, that would actually be fine for us, as long as the > > > timeout is long enough. MySQL would just be automatically killing the > > > long-running queries that we kill by hand in places like bmo. > > > > It can be tweaked; I forget what the default is. But it doesn't kill > > the *running* query - it kills the blocking one. (All together now - > > eww!) > > But wait, that makes sense. You mean if a SELECT is blocking everybody > else from running, it kills the SELECT, right? No. It kills the other one - ie the one that is not running. > > in a transaction. This takes up to 15-20 minutes to run, pulling a few > > hundred thousand rows each time. > > Thankfully nothing in Bugzilla does that, except for collectstats.pl > --regenerate. Yeah, but I've had it happen with concurrent update/delete. We 'fixed' it by changing delete to select first then delete if there was a match > Okay. So we should recommend that if people are going to use > replication, they use at least 4.1.11 or 5.0.3. It still doesn't help, though - it just makes it less likely, since by incresing the time before theres an error, you've decreased the number of times an error will occur. Bradley From bbaetz at acm.org Sun Aug 6 07:22:06 2006 From: bbaetz at acm.org (Bradley Baetz) Date: Sun, 6 Aug 2006 17:22:06 +1000 Subject: Possibly moving to InnoDB In-Reply-To: <1154847812.2598.3.camel@es-lappy> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> <44D51560.30602@bugzilla.org> <1154847812.2598.3.camel@es-lappy> Message-ID: <99435f5b0608060022w17df4f79rcabceb20a787da0a@mail.gmail.com> On 06/08/06, Max Kanat-Alexander wrote: > On Sat, 2006-08-05 at 18:02 -0400, David Miller wrote: > > The Solid guys had a booth at OSCON and were bragging about them hooking > > up with MySQL... > > > > http://news.com.com/M/2100-1012_3-6061443.html > > > > http://dev.soliddb.com/ > > Okay. In any case it's a beta, and it looks like it only supports MySQL > 5.0.22 and later. So that's an unlikely candidate for us at this time. I don't suppose dropping MySQL support is an option? ;) Bradley From mkanat at bugzilla.org Sun Aug 6 15:39:36 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Sun, 06 Aug 2006 08:39:36 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <99435f5b0608060016h34856796v47db3edd5c49f05@mail.gmail.com> References: <1154745571.2440.9.camel@localhost.localdomain> <99435f5b0608042007h4acfafefxa91911af728c5e71@mail.gmail.com> <1154778917.2405.24.camel@es-lappy> <99435f5b0608050529v51895231i3a8596f89d72b3c7@mail.gmail.com> <1154791088.6998.3.camel@es-lappy> <99435f5b0608060016h34856796v47db3edd5c49f05@mail.gmail.com> Message-ID: <1154878777.2608.6.camel@es-lappy> On Sun, 2006-08-06 at 17:16 +1000, Bradley Baetz wrote: > > But wait, that makes sense. You mean if a SELECT is blocking everybody > > else from running, it kills the SELECT, right? > > No. It kills the other one - ie the one that is not running. That's pretty silly. > Yeah, but I've had it happen with concurrent update/delete. We 'fixed' > it by changing delete to select first then delete if there was a match Odd. Thankfully we rarely have a concurrent UPDATE/DELETE. We occasionally have some concurrent INSERT/DELETEs, though (like logincookies). > > Okay. So we should recommend that if people are going to use > > replication, they use at least 4.1.11 or 5.0.3. > > It still doesn't help, though - it just makes it less likely, since by > increasing the time before theres an error, you've decreased the number > of times an error will occur. Yeah. I mean, I know that many huge sites use replication just fine on InnoDB, including LiveJournal and Wikipedia, so some of it probably depends on the structure of the application. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From gerv at mozilla.org Mon Aug 7 09:09:34 2006 From: gerv at mozilla.org (Gervase Markham) Date: Mon, 07 Aug 2006 10:09:34 +0100 Subject: perltidy In-Reply-To: References: Message-ID: <44D7034E.6060702@mozilla.org> Benton, Kevin wrote: > I'm wondering how others in this list feel about changing our > entrance criteria for new patches to include that the submitting > developer should run perltidy on the code prior to submitting it. I like the idea, if we can get a set of command-line flags which are close enough to the existing style we've developed. If we are going to do this, should we have a flag day where a group of people run perltidy on the existing codebase, with the agreed flags, and check in the result? Gerv From fergus at yahoo-inc.com Mon Aug 7 10:17:09 2006 From: fergus at yahoo-inc.com (Fergus Sullivan) Date: Mon, 07 Aug 2006 03:17:09 -0700 Subject: perltidy In-Reply-To: <44D7034E.6060702@mozilla.org> References: <44D7034E.6060702@mozilla.org> Message-ID: <44D71325.3040100@yahoo-inc.com> Gervase Markham wrote: > If we are going to do this, should we have a flag day where a group of > people run perltidy on the existing codebase, with the agreed flags, and > check in the result? > Downside here is that it would cause a lot of diffs when comparing pre-tidy code with post-tidy code. From gerv at mozilla.org Mon Aug 7 14:43:18 2006 From: gerv at mozilla.org (Gervase Markham) Date: Mon, 07 Aug 2006 15:43:18 +0100 Subject: perltidy In-Reply-To: <44D71325.3040100@yahoo-inc.com> References: <44D7034E.6060702@mozilla.org> <44D71325.3040100@yahoo-inc.com> Message-ID: <44D75186.8030507@mozilla.org> Fergus Sullivan wrote: > Downside here is that it would cause a lot of diffs when comparing > pre-tidy code with post-tidy code. If we get the options close enough to our house style, the changes might not be all that significant. Perhaps we could try, and see what size the diff was. Although, it does seem odd to say "This is our preferred style. But we haven't changed all the code to match because that would, er, involve changing the code." Gerv From myk at mozilla.org Mon Aug 7 18:01:54 2006 From: myk at mozilla.org (Myk Melez) Date: Mon, 07 Aug 2006 11:01:54 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <2F6DEDF9-9953-4E42-93E2-81FF29690A06@tragicallyleet.com> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> <44D51560.30602@bugzilla.org> <2F6DEDF9-9953-4E42-93E2-81FF29690A06@tragicallyleet.com> Message-ID: <44D78012.7030604@mozilla.org> Jeffrey Hulten wrote: > The other possibility is to make the indexing of the full text be > programmatical instead of a function of the database operation. In > other words, index it ourselves. Right. One way to do this would be to duplicate both summary and comment data in a separate table specifically designed for fulltext indexing, so that both bugs and longdescs can get transactions and referential integrity, i.e.: CREATE TABLE bug_descriptions ( bug_id INT NOT NULL, summary VARCHAR(255) NOT NULL, comment TEXT ); Besides freeing those tables to participate in InnoDB goodness (or SolidDB goodness, for that matter), the other advantage to this approach is that it resolves a longstanding problem with our fulltext indexes (because of MySQL's inability to include columns from different tables in a fulltext index) that reduces their utility (even with our hacks to partially work around the problem). The cost, however, is disk space. Otherwise, we could roll our own fulltext engine or use something like Senna . -myk -------------- next part -------------- An HTML attachment was scrubbed... URL: From lpsolit at gmail.com Mon Aug 7 21:11:55 2006 From: lpsolit at gmail.com (=?ISO-8859-1?Q?Fr=E9d=E9ric_Buclin?=) Date: Mon, 07 Aug 2006 23:11:55 +0200 Subject: perltidy In-Reply-To: <44D75186.8030507@mozilla.org> References: <44D7034E.6060702@mozilla.org> <44D71325.3040100@yahoo-inc.com> <44D75186.8030507@mozilla.org> Message-ID: <44D7AC9B.80409@gmail.com> > Although, it does seem odd to say "This is our preferred style. But we > haven't changed all the code to match because that would, er, involve > changing the code." No, this makes perfect sense. Bitrotting some patches awaiting review only because we added or removed some whitespaces or moved a bracket on its own line, etc... is a waste of time for everybody: for the developer writing the patch as well as for the reviewer wasting his time reviewing a patch which no longer applies correctly. And of course, in case you have to backport your patch to branches, the probability that your patch doesn't apply cleanly on branches is higher because we did some minor changes on the trunk. I'm against this kind of "cleanup" of the existing code. I spent a lot of time backporting security patches lately, so I know how painful this can be. LpSolit From lpsolit at gmail.com Mon Aug 7 22:03:08 2006 From: lpsolit at gmail.com (=?ISO-8859-1?Q?Fr=E9d=E9ric_Buclin?=) Date: Tue, 08 Aug 2006 00:03:08 +0200 Subject: Bugzilla meeting on Tuesday, August 8, 2006 at 18:00 GMT (11:00 PDT) Message-ID: <44D7B89C.9020904@gmail.com> Reminder: We will have another (short?) Bugzilla meeting tomorrow (Tuesday) at 18:00 GMT (11:00 PDT). LpSolit From fergus at yahoo-inc.com Mon Aug 7 22:23:32 2006 From: fergus at yahoo-inc.com (Fergus Sullivan) Date: Mon, 07 Aug 2006 15:23:32 -0700 Subject: perltidy In-Reply-To: <44D7AC9B.80409@gmail.com> References: <44D7034E.6060702@mozilla.org> <44D71325.3040100@yahoo-inc.com> <44D75186.8030507@mozilla.org> <44D7AC9B.80409@gmail.com> Message-ID: <44D7BD64.2040901@yahoo-inc.com> Fr?d?ric Buclin wrote: >> Although, it does seem odd to say "This is our preferred style. But we >> haven't changed all the code to match because that would, er, involve >> changing the code." >> > > No, this [avoiding perltidy?] makes perfect sense. Bitrotting some patches awaiting review > only because we added or removed some whitespaces or moved a bracket on > its own line, etc... is a waste of time for everybody: for the developer > writing the patch as well as for the reviewer wasting his time reviewing > a patch which no longer applies correctly. And of course, in case you > have to backport your patch to branches, the probability that your patch > doesn't apply cleanly on branches is higher because we did some minor > changes on the trunk There are places I've worked where adjusting white space/code formatting was considered almost a sackable offence. In the case of Bugzilla it seems a major blocking point has been lack of sufficient resources to review patches. From Tomas.Kopal at altap.cz Tue Aug 8 07:38:56 2006 From: Tomas.Kopal at altap.cz (Tomas Kopal) Date: Tue, 08 Aug 2006 09:38:56 +0200 Subject: Possibly moving to InnoDB In-Reply-To: <44D78012.7030604@mozilla.org> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> <44D51560.30602@bugzilla.org> <2F6DEDF9-9953-4E42-93E2-81FF29690A06@tragicallyleet.com> <44D78012.7030604@mozilla.org> Message-ID: <44D83F90.4050103@altap.cz> Well, I haven't contributed to bugzilla for some time due to my current time constraints, but I am still following the discussion, and this one caught my attention :-). On 7.8.2006 20:01, Myk Melez wrote: > Jeffrey Hulten wrote: >> The other possibility is to make the indexing of the full text be >> programmatical instead of a function of the database operation. In >> other words, index it ourselves. > Right. One way to do this would be to duplicate both summary and > comment data in a separate table specifically designed for fulltext > indexing, so that both bugs and longdescs can get transactions and > referential integrity, i.e.: ... > > Otherwise, we could roll our own fulltext engine or use something like > Senna . > > -myk > I would like to support this way of thinking, as removing the dependency on native MySQL fulltext search would also remove (probably the last?) dependency on MySQL specific behavior. I mean, if we solve the full text search problem in DB agnostic way, we'd get not only greater independence on MySQL DB engine (btw., according to the solid web, the solidDB engine does not support fulltext indexing), but also full text search on other DBs like Postgres (and Oracle etc. in the future). The only requirement is this to be DB agnostic, so Senna probably does not fit... Back into read-only mode now... :-) Tomas From jochen.wiedmann at gmail.com Tue Aug 8 07:53:54 2006 From: jochen.wiedmann at gmail.com (Jochen Wiedmann) Date: Tue, 8 Aug 2006 09:53:54 +0200 Subject: Possibly moving to InnoDB In-Reply-To: <44D83F90.4050103@altap.cz> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> <44D51560.30602@bugzilla.org> <2F6DEDF9-9953-4E42-93E2-81FF29690A06@tragicallyleet.com> <44D78012.7030604@mozilla.org> <44D83F90.4050103@altap.cz> Message-ID: On 8/8/06, Tomas Kopal wrote: > I would like to support this way of thinking, as removing the dependency > on native MySQL fulltext search would also remove (probably the last?) > dependency on MySQL specific behavior. > I mean, if we solve the full text search problem in DB agnostic way, > we'd get not only greater independence on MySQL DB engine (btw., > according to the solid web, the solidDB engine does not support fulltext > indexing), but also full text search on other DBs like Postgres (and > Oracle etc. in the future). > The only requirement is this to be DB agnostic, so Senna probably does > not fit... Following this way, you might as well use no DB at all. Implementing classical DB tasks for yourself is surely the wrong way to go. Jochen -- My wife Mary and I have been married for forty-seven years and not once have we had an argument serious enough to consider divorce; murder, yes, but divorce, never. (Jack Benny) From bbaetz at acm.org Tue Aug 8 07:59:59 2006 From: bbaetz at acm.org (Bradley Baetz) Date: Tue, 8 Aug 2006 17:59:59 +1000 Subject: Possibly moving to InnoDB In-Reply-To: <44D78012.7030604@mozilla.org> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> <44D51560.30602@bugzilla.org> <2F6DEDF9-9953-4E42-93E2-81FF29690A06@tragicallyleet.com> <44D78012.7030604@mozilla.org> Message-ID: <99435f5b0608080059v1380a08ak6ac7b9337978417@mail.gmail.com> On 08/08/06, Myk Melez wrote: > Right. One way to do this would be to duplicate both summary and comment > data in a separate table specifically designed for fulltext indexing, so > that both bugs and longdescs can get transactions and referential integrity, > i.e.: Or we could use a database that doesn't suck. > The other advantage to this approach is that it > resolves a longstanding problem with our fulltext indexes (because of > MySQL's inability to include columns from different tables in a fulltext > index) that reduces their utility (even with our hacks to partially work > around the problem). Or we could... :) > Otherwise, we could roll our own fulltext engine or use something like > Senna. I don't really think we want to go down that path. Postgres has one, oracle has one, mysql has one. The different syntax issues only happens when doing 'complicated' stuff - simple 'find word in substring' can just be handled by bugzilla. If a user tries multiple fulltext queries, and the backend db doesn't support them, we can't help. (Note that postgres' tsearch2 is actually implented via a separate table and triggers. Thats a bit ugly, but we don't have to care - it Just Works) Bradley From Tomas.Kopal at altap.cz Tue Aug 8 08:14:30 2006 From: Tomas.Kopal at altap.cz (Tomas Kopal) Date: Tue, 08 Aug 2006 10:14:30 +0200 Subject: Possibly moving to InnoDB In-Reply-To: References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> <44D51560.30602@bugzilla.org> <2F6DEDF9-9953-4E42-93E2-81FF29690A06@tragicallyleet.com> <44D78012.7030604@mozilla.org> <44D83F90.4050103@altap.cz> Message-ID: <44D847E6.4020607@altap.cz> On 8.8.2006 9:53, Jochen Wiedmann wrote: > On 8/8/06, Tomas Kopal wrote: > >> I would like to support this way of thinking, as removing the dependency >> on native MySQL fulltext search would also remove (probably the last?) >> dependency on MySQL specific behavior. >> I mean, if we solve the full text search problem in DB agnostic way, >> we'd get not only greater independence on MySQL DB engine (btw., >> according to the solid web, the solidDB engine does not support fulltext >> indexing), but also full text search on other DBs like Postgres (and >> Oracle etc. in the future). >> The only requirement is this to be DB agnostic, so Senna probably does >> not fit... > > Following this way, you might as well use no DB at all. > > Implementing classical DB tasks for yourself is surely the wrong way to go. > > > Jochen > Well, full text search engine is by no means classical DB task. E.g. at least SQL standard does not mention anything about full text search... But it can by built on top of the DB. And we can (and should) surely use SQL DB for underlying storage for the engine... Tomas From mkanat at bugzilla.org Tue Aug 8 08:25:14 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Tue, 08 Aug 2006 01:25:14 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <99435f5b0608080059v1380a08ak6ac7b9337978417@mail.gmail.com> References: <1154745571.2440.9.camel@localhost.localdomain> <44D47A06.8030501@gmail.com> <1154780522.2405.27.camel@es-lappy> <44D51560.30602@bugzilla.org> <2F6DEDF9-9953-4E42-93E2-81FF29690A06@tragicallyleet.com> <44D78012.7030604@mozilla.org> <99435f5b0608080059v1380a08ak6ac7b9337978417@mail.gmail.com> Message-ID: <1155025515.31022.4.camel@localhost.localdomain> On Tue, 2006-08-08 at 17:59 +1000, Bradley Baetz wrote: > > Otherwise, we could roll our own fulltext engine or use something like > > Senna. > > I don't really think we want to go down that path. [snip] Yeah, I think it's probably not necessary to roll our own fulltext engine at this time. The solution I'm thinking of is this: Make the index on bugs.short_desc a normal index. Instead of doing a fulltext match against bugs.short_desc, do a substring match. And just add "1.0" to the relevance for any bug that has the word in the title. This would solve bug 347721, and it would also handle one of my biggest pet peeves with fulltext search--that it often ranks things very low even if they have the exact word in the summary. Anyhow, if you look at the code now, you'll see that we only use the fulltext index on short_desc to determine the relevance. For the actual searching we use a REGEXP, so my idea wouldn't really change much anyhow (just the order results are returned in). -Max -- http://www.everythingsolved.com/ Competent, Friendly Bugzilla Services. And Everything Else, too. From mkanat at bugzilla.org Tue Aug 8 08:58:15 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Tue, 08 Aug 2006 01:58:15 -0700 Subject: InnoDB Testing, Discussion Results Message-ID: <1155027496.31022.20.camel@localhost.localdomain> I did some performance testing of InnoDB vs. MyISAM on Bugzilla, using my load-testing script from bug 347520. As far as I can tell, under heavy load there's little difference between the speed of MyISAM and the speed of InnoDB. This is probably because we still use whole-table locks in the Bugzilla code. Looking into it (using MySQL's slow query log), I did find that under high numbers of simultaneous users, certain buglist searches do spend a lot of time waiting for WRITE locks on the database to release. That is, every time somebody uses process_bug or post_bug, it's going to slow down queries. If we were under transactions, this would end. So InnoDB probably is a big plus in that area. I've read very good things about the stability of InnoDB, and for our uses its speed seems entirely comparable to the speed of MyISAM. Overall, I see more benefit in moving to InnoDB than I see problems. From our discussion I've learned some valuable things, and I'm sure that many of them would be of interest to our users, and should be mentioned in the Release Notes. And it's true that we will have certain hurdles to jump in making our InnoDB support perfect, but we always have hurdles to jump when making improvements. Also, for those interested, here's the bug with the patch that actually changes Bugzilla to use InnoDB: https://bugzilla.mozilla.org/show_bug.cgi?id=347475 I'm going to wait a day or two, and then probably commit it, unless the approvers object, or unless somebody explains a very definite reason that moving to InnoDB would cause more harm than good for the majority of our users. -Max -- http://www.everythingsolved.com/ Competent, Friendly Bugzilla Services. And Everything Else, too. From lpsolit at gmail.com Tue Aug 8 11:28:21 2006 From: lpsolit at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBCdWNsaW4=?=) Date: Tue, 08 Aug 2006 13:28:21 +0200 Subject: {Spam?} Re: InnoDB Testing, Discussion Results In-Reply-To: <1155027496.31022.20.camel@localhost.localdomain> References: <1155027496.31022.20.camel@localhost.localdomain> Message-ID: <44D87555.5090401@gmail.com> > I'm going to wait a day or two, and then probably commit it, unless the > approvers object, or unless somebody explains a very definite reason > that moving to InnoDB would cause more harm than good for the majority > of our users. We should talk about this during our Bugzilla meeting today. Deciding to commit such changes while the thread is still not closed doesn't sound right to me. LpSolit From mkanat at bugzilla.org Tue Aug 8 12:53:46 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Tue, 08 Aug 2006 05:53:46 -0700 Subject: {Spam?} Re: InnoDB Testing, Discussion Results In-Reply-To: <44D87555.5090401@gmail.com> References: <1155027496.31022.20.camel@localhost.localdomain> <44D87555.5090401@gmail.com> Message-ID: <1155041626.31022.25.camel@localhost.localdomain> On Tue, 2006-08-08 at 13:28 +0200, Fr?d?ric Buclin wrote: > We should talk about this during our Bugzilla meeting today. Deciding to > commit such changes while the thread is still not closed doesn't sound > right to me. That's why I was waiting a day or two. :-) I think most of the actual discussion of the thread's issue is complete, though. We have been mostly discussing some side issues at the end of it. -Max -- http://www.everythingsolved.com/ Competent, Friendly Bugzilla Services. And Everything Else, too. From bbaetz at acm.org Tue Aug 8 13:33:25 2006 From: bbaetz at acm.org (Bradley Baetz) Date: Tue, 8 Aug 2006 23:33:25 +1000 Subject: InnoDB Testing, Discussion Results In-Reply-To: <1155027496.31022.20.camel@localhost.localdomain> References: <1155027496.31022.20.camel@localhost.localdomain> Message-ID: <99435f5b0608080633ud0a039ara717f9673d8f26bc@mail.gmail.com> On 08/08/06, Max Kanat-Alexander wrote: > Looking into it (using MySQL's slow query log), I did find that under > high numbers of simultaneous users, certain buglist searches do spend a > lot of time waiting for WRITE locks on the database to release. That is, > every time somebody uses process_bug or post_bug, it's going to slow > down queries. If we were under transactions, this would end. We shouldn't have WRITE locks for any period of time. If you're not doing replication, what you'll find is that readers block writers - ie you're waiting for WRITE locks, because someone else has a read log. You may want to test that - it will improve, but you'll still have row level exclusive locks (yes, its more complicated than that, but for our purposes...) Bradley From mkanat at bugzilla.org Tue Aug 8 14:52:41 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Tue, 08 Aug 2006 07:52:41 -0700 Subject: Possible Fulltext Search Improvements Message-ID: <1155048762.3843.3.camel@localhost.localdomain> Hey hey. So, I'm working on fulltext search a bit, and I was wondering if anybody out there would like to help me out with some testing. Basically, I've made some modifications, but I don't yet know if they're actually *better* than what we had before. So what I'd like is if a few people could apply these patches to their test installation, and then try doing some fulltext searches that they might actually do, and tell me if they like the results better or worse than how old Bugzilla does it: https://bugzilla.mozilla.org/show_bug.cgi?id=287170 https://bugzilla.mozilla.org/show_bug.cgi?id=347864 You have to apply both patches from both bugs, in the order that I pasted them above. Note that you'll have to run checksetup.pl after you apply the patches, and it will modify your DB, so you should only do this in a testing environment, or a copy of your production environment. Just give me your subjective opinion--are the search results "better?" If they are better, what's better about them? If they're worse, what's worse about them? -Max -- http://www.everythingsolved.com/ Competent, Friendly Bugzilla Services. And Everything Else, too. From kevin.benton at amd.com Tue Aug 8 19:54:48 2006 From: kevin.benton at amd.com (Benton, Kevin) Date: Tue, 8 Aug 2006 12:54:48 -0700 Subject: InnoDB Testing, Discussion Results Message-ID: > -----Original Message----- > From: developers-owner at bugzilla.org [mailto:developers-owner at bugzilla.org] > On Behalf Of Max Kanat-Alexander > Sent: Tuesday, August 08, 2006 2:58 AM > To: developers at bugzilla.org > Subject: InnoDB Testing, Discussion Results > > > I did some performance testing of InnoDB vs. MyISAM on Bugzilla, > using > my load-testing script from bug 347520. > > As far as I can tell, under heavy load there's little difference > between the speed of MyISAM and the speed of InnoDB. This is probably > because we still use whole-table locks in the Bugzilla code. > > Looking into it (using MySQL's slow query log), I did find that > under > high numbers of simultaneous users, certain buglist searches do spend a > lot of time waiting for WRITE locks on the database to release. That is, > every time somebody uses process_bug or post_bug, it's going to slow > down queries. If we were under transactions, this would end. So InnoDB > probably is a big plus in that area. > > I've read very good things about the stability of InnoDB, and for > our > uses its speed seems entirely comparable to the speed of MyISAM. > Overall, I see more benefit in moving to InnoDB than I see problems. > > From our discussion I've learned some valuable things, and I'm sure > that many of them would be of interest to our users, and should be > mentioned in the Release Notes. And it's true that we will have certain > hurdles to jump in making our InnoDB support perfect, but we always have > hurdles to jump when making improvements. > > Also, for those interested, here's the bug with the patch that > actually > changes Bugzilla to use InnoDB: > > https://bugzilla.mozilla.org/show_bug.cgi?id=347475 > > I'm going to wait a day or two, and then probably commit it, unless > the > approvers object, or unless somebody explains a very definite reason > that moving to InnoDB would cause more harm than good for the majority > of our users. Where InnoDB suffers that MyISAM doesn't is that MyISAM is specifically designed for high-speed queries. This is especially important for "big-wigs" in corporate America. It seems to me that we need to give administrators the ability to decide which is more important - faster searches, or data integrity if the server goes off-line during a write. I don't think it's fair for us to make that decision for administrators by default without giving them the option to choose. Loosing fulltext indexes in some cases is clearly not worth it when searching full bug histories (attachments, for example), or other text fields. On the other hand, if the installation rarely does searching, InnoDB seems to make a lot of sense if the goal is to make sure no data is lost. Please be aware, however, that preventing data loss can be accomplished in an alternative way - by turning on the binlog. That allows administrators to replay any action that modifies the server without respect to storage engine. It also seems to me to be a bad idea to make tables that very rarely change suffer from the overhead of transactional storage engines. That seems to me to be another administrator-based decision. Personally, I would hope that in the process of making it possible to use InnoDB or MyISAM, we would also work to make it possible to use NDBCluster so installations that need row-level-locking *and* high availability could get it. The only problem with NDBCluster that could prevent an admin from using it is in 5.0 and before, NDBCluster must have enough RAM to load entire tables. I would hope that we'd give admins the ability to do this at a few different levels - semi-globally, selectively by pre-programmed group, or in a per-table basis (for the most advanced admins). --- Kevin Benton Perl/Bugzilla Developer/Administrator, Perforce SCM Administrator AMD - ECSD Software Validation and Tools The opinions stated in this communication do not necessarily reflect the view of Advanced Micro Devices and have not been reviewed by management. This communication may contain sensitive and/or confidential and/or proprietary information. Distribution of such information is strictly prohibited without prior consent of Advanced Micro Devices. This communication is for the intended recipient(s) only. If you have received this communication in error, please notify the sender, then destroy any remaining copies of this communication. From gerv at mozilla.org Tue Aug 8 20:06:53 2006 From: gerv at mozilla.org (Gervase Markham) Date: Tue, 08 Aug 2006 21:06:53 +0100 Subject: {Spam?} Re: InnoDB Testing, Discussion Results In-Reply-To: <1155041626.31022.25.camel@localhost.localdomain> References: <1155027496.31022.20.camel@localhost.localdomain> <44D87555.5090401@gmail.com> <1155041626.31022.25.camel@localhost.localdomain> Message-ID: <44D8EEDD.2080607@mozilla.org> Max Kanat-Alexander wrote: > That's why I was waiting a day or two. :-) I think most of the actual > discussion of the thread's issue is complete, though. We have been > mostly discussing some side issues at the end of it. If some developers feel they haven't finished discussing it yet, then we haven't finished discussing it yet :-) This is a nuclear power plant, not a bike shed. Gerv From kevin.benton at amd.com Tue Aug 8 20:09:56 2006 From: kevin.benton at amd.com (Benton, Kevin) Date: Tue, 8 Aug 2006 13:09:56 -0700 Subject: Possibly moving to InnoDB Message-ID: > > > It can be tweaked; I forget what the default is. But it doesn't kill > > > the *running* query - it kills the blocking one. (All together now - > > > eww!) > > > > But wait, that makes sense. You mean if a SELECT is blocking > everybody > > else from running, it kills the SELECT, right? > > No. It kills the other one - ie the one that is not running. This can be configured by determining the isolation level in the server. See http://dev.mysql.com/doc/refman/4.1/en/innodb-transaction-isolation.html for more information. --- Kevin Benton Perl/Bugzilla Developer/Administrator, Perforce SCM Administrator AMD - ECSD Software Validation and Tools The opinions stated in this communication do not necessarily reflect the view of Advanced Micro Devices and have not been reviewed by management. This communication may contain sensitive and/or confidential and/or proprietary information. Distribution of such information is strictly prohibited without prior consent of Advanced Micro Devices. This communication is for the intended recipient(s) only. If you have received this communication in error, please notify the sender, then destroy any remaining copies of this communication. From mkanat at bugzilla.org Tue Aug 8 20:10:28 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Tue, 08 Aug 2006 13:10:28 -0700 Subject: InnoDB Testing, Discussion Results In-Reply-To: References: Message-ID: <1155067828.2396.10.camel@es-lappy> On Tue, 2006-08-08 at 12:54 -0700, Benton, Kevin wrote: > It seems to me that we need to give > administrators the ability to decide That's not possible--transactions have to be supported by all our databases or we can't use them. > Losing fulltext indexes in some cases [snip] We won't lose fulltext indexes. > is clearly not worth it when searching full bug > histories (attachments, for example), We don't have fulltext indexes on those fields now. > Personally, I would hope that in the process of making it possible to > use InnoDB or MyISAM, we would also work to make it possible to use > NDBCluster Yeah, I suspect InnoDB support would also give us pretty easy NDBCluster support. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From kevin.benton at amd.com Tue Aug 8 20:16:28 2006 From: kevin.benton at amd.com (Benton, Kevin) Date: Tue, 8 Aug 2006 13:16:28 -0700 Subject: InnoDB Testing, Discussion Results Message-ID: > On 08/08/06, Max Kanat-Alexander wrote: > > Looking into it (using MySQL's slow query log), I did find that > under > > high numbers of simultaneous users, certain buglist searches do spend a > > lot of time waiting for WRITE locks on the database to release. That is, > > every time somebody uses process_bug or post_bug, it's going to slow > > down queries. If we were under transactions, this would end. > > We shouldn't have WRITE locks for any period of time. If you're not > doing replication, what you'll find is that readers block writers - ie > you're waiting for WRITE locks, because someone else has a read log. > > You may want to test that - it will improve, but you'll still have row > level exclusive locks (yes, its more complicated than that, but for > our purposes...) If we do this, we have to be especially careful to make sure we release locks explicitly using mod_perl. --- Kevin Benton Perl/Bugzilla Developer/Administrator, Perforce SCM Administrator AMD - ECSD Software Validation and Tools The opinions stated in this communication do not necessarily reflect the view of Advanced Micro Devices and have not been reviewed by management. This communication may contain sensitive and/or confidential and/or proprietary information. Distribution of such information is strictly prohibited without prior consent of Advanced Micro Devices. This communication is for the intended recipient(s) only. If you have received this communication in error, please notify the sender, then destroy any remaining copies of this communication. From kevin.benton at amd.com Tue Aug 8 20:32:16 2006 From: kevin.benton at amd.com (Benton, Kevin) Date: Tue, 8 Aug 2006 13:32:16 -0700 Subject: InnoDB Testing, Discussion Results Message-ID: > On Tue, 2006-08-08 at 12:54 -0700, Benton, Kevin wrote: > > It seems to me that we need to give > > administrators the ability to decide > > That's not possible--transactions have to be supported by all our > databases or we can't use them. I agree, but that doesn't mean that we have to use the transaction code when making updates based on administrator configuration. I think that in the long-term, we should think seriously about moving to libraries to update the back-end so we can create those libraries to handle all the databases we care to support. > > Losing fulltext indexes in some cases [snip] > > We won't lose fulltext indexes. > > > is clearly not worth it when searching full bug > > histories (attachments, for example), > > We don't have fulltext indexes on those fields now. Some of us have customized our installations to use fulltext indexes on fields that didn't have them previously. > > Personally, I would hope that in the process of making it possible to > > use InnoDB or MyISAM, we would also work to make it possible to use > > NDBCluster > > Yeah, I suspect InnoDB support would also give us pretty easy > NDBCluster support. I agree. --- Kevin Benton Perl/Bugzilla Developer/Administrator, Perforce SCM Administrator AMD - ECSD Software Validation and Tools The opinions stated in this communication do not necessarily reflect the view of Advanced Micro Devices and have not been reviewed by management. This communication may contain sensitive and/or confidential and/or proprietary information. Distribution of such information is strictly prohibited without prior consent of Advanced Micro Devices. This communication is for the intended recipient(s) only. If you have received this communication in error, please notify the sender, then destroy any remaining copies of this communication. From kevin.benton at amd.com Tue Aug 8 20:23:44 2006 From: kevin.benton at amd.com (Benton, Kevin) Date: Tue, 8 Aug 2006 13:23:44 -0700 Subject: {Spam?} Re: InnoDB Testing, Discussion Results Message-ID: Being that I'm in the MySQL DBA class right now, and with the knowledge I've gained so far, I'm not convinced that moving to InnoDB is a good idea yet. I would ask that we wait till at least next week to make any decision about this. While it is 10 years old and stable, it doesn't necessarily accomplish the goals of corporations under some circumstances. --- Kevin Benton Perl/Bugzilla Developer/Administrator, Perforce SCM Administrator AMD - ECSD Software Validation and Tools The opinions stated in this communication do not necessarily reflect the view of Advanced Micro Devices and have not been reviewed by management. This communication may contain sensitive and/or confidential and/or proprietary information. Distribution of such information is strictly prohibited without prior consent of Advanced Micro Devices. This communication is for the intended recipient(s) only. If you have received this communication in error, please notify the sender, then destroy any remaining copies of this communication. > -----Original Message----- > From: developers-owner at bugzilla.org [mailto:developers-owner at bugzilla.org] > On Behalf Of Gervase Markham > Sent: Tuesday, August 08, 2006 2:07 PM > To: developers at bugzilla.org > Subject: Re: {Spam?} Re: InnoDB Testing, Discussion Results > > Max Kanat-Alexander wrote: > > That's why I was waiting a day or two. :-) I think most of the > actual > > discussion of the thread's issue is complete, though. We have been > > mostly discussing some side issues at the end of it. > > If some developers feel they haven't finished discussing it yet, then we > haven't finished discussing it yet :-) > > This is a nuclear power plant, not a bike shed. > > Gerv > - > To view or change your list settings, click here: > > From mkanat at bugzilla.org Tue Aug 8 20:28:12 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Tue, 08 Aug 2006 13:28:12 -0700 Subject: {Spam?} Re: InnoDB Testing, Discussion Results In-Reply-To: <44D8EEDD.2080607@mozilla.org> References: <1155027496.31022.20.camel@localhost.localdomain> <44D87555.5090401@gmail.com> <1155041626.31022.25.camel@localhost.localdomain> <44D8EEDD.2080607@mozilla.org> Message-ID: <1155068892.2396.11.camel@es-lappy> On Tue, 2006-08-08 at 21:06 +0100, Gervase Markham wrote: > Max Kanat-Alexander wrote: > > That's why I was waiting a day or two. :-) I think most of the actual > > discussion of the thread's issue is complete, though. We have been > > mostly discussing some side issues at the end of it. > > If some developers feel they haven't finished discussing it yet, then we > haven't finished discussing it yet :-) Yeah, no, I agree. :-) We talked about it more at the meeting, too. We're going to look into it more. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From jochen.wiedmann at gmail.com Tue Aug 8 22:15:39 2006 From: jochen.wiedmann at gmail.com (Jochen Wiedmann) Date: Wed, 09 Aug 2006 00:15:39 +0200 Subject: InnoDB Testing, Discussion Results In-Reply-To: References: Message-ID: <44D90D0B.9040104@gmail.com> Benton, Kevin wrote: > If we do this, we have to be especially careful to make sure we release > locks explicitly using mod_perl. Do we? Isn't this part of the transaction logic that they are released at rollback or commit? From bbaetz at acm.org Tue Aug 8 23:24:52 2006 From: bbaetz at acm.org (Bradley Baetz) Date: Wed, 9 Aug 2006 09:24:52 +1000 Subject: Possibly moving to InnoDB In-Reply-To: References: Message-ID: <99435f5b0608081624m7d98d6faie45875b8cfba69c8@mail.gmail.com> On 09/08/06, Benton, Kevin wrote: > > No. It kills the other one - ie the one that is not running. > > This can be configured by determining the isolation level in the server. Well... Yes, if you change the isolation mode there aren't the locks. But you either need isolation or locks for data integrity, so thats not really a solution. Bradley From mkanat at bugzilla.org Wed Aug 9 02:37:02 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Tue, 08 Aug 2006 19:37:02 -0700 Subject: Possibly moving to InnoDB In-Reply-To: <99435f5b0608081624m7d98d6faie45875b8cfba69c8@mail.gmail.com> References: <99435f5b0608081624m7d98d6faie45875b8cfba69c8@mail.gmail.com> Message-ID: <1155091022.2342.1.camel@es-lappy> On Wed, 2006-08-09 at 09:24 +1000, Bradley Baetz wrote: > Well... Yes, if you change the isolation mode there aren't the locks. > But you either need isolation or locks for data integrity, so thats > not really a solution. True. Thankfully, though, our longest-running query (buglist.cgi) would have no problem running READ UNCOMMITTED. I'd have to read up more on exactly how the deadlock code works, but perhaps that would help. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From kevin.benton at amd.com Fri Aug 11 15:23:35 2006 From: kevin.benton at amd.com (Benton, Kevin) Date: Fri, 11 Aug 2006 08:23:35 -0700 Subject: Automated Table Optimizations Message-ID: I just learned that doing either CHECK TABLE or ANALYZE TABLE can do a lot for us in optimizing disk space utilization as well as indexing of tables in MySQL. This can improve performance of the back-end in tables that could potentially suffer from fragmenting (usually due to deletes followed by subsequent adds). I think it'd be a good idea to recommend that users implement a cron job that does these operations on a weekly-monthly basis in an optimization section of the manual. I also think it'd be good to document (if we don't already) how to set up MySQL replication in a similar section. What do others think about this? --- Kevin Benton Perl/Bugzilla Developer/Administrator, Perforce SCM Administrator AMD - ECSD Software Validation and Tools ? The opinions stated in this communication do not necessarily reflect the view of Advanced Micro Devices and have not been reviewed by management.? This communication may contain sensitive and/or confidential and/or proprietary information.? Distribution of such information is strictly prohibited without prior consent of Advanced Micro Devices.? This communication is for the intended recipient(s) only.? If you have received this communication in error, please notify the sender, then destroy any remaining copies of this communication. From mkanat at bugzilla.org Fri Aug 11 22:15:56 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Fri, 11 Aug 2006 15:15:56 -0700 Subject: Automated Table Optimizations In-Reply-To: References: Message-ID: <1155334556.2440.7.camel@localhost.localdomain> On Fri, 2006-08-11 at 08:23 -0700, Benton, Kevin wrote: > I just learned that doing either CHECK TABLE or ANALYZE TABLE can do a > lot for us in optimizing disk space utilization as well as indexing of > tables in MySQL. Yep. And on PostgreSQL, VACUUM ANALYZE (or VACUUM FULL ANALYZE, if you have more time). It's even more important on Postgres than on MySQL. > I think it'd be a good idea to recommend that users implement a cron > job that does these operations on a weekly-monthly basis in an > optimization section of the manual. I also considered adding a ./checksetup.pl --optimize switch. There's a bug filed on it. > I also think it'd be good to document (if we don't already) how to set > up MySQL replication in a similar section. What do others think about > this? Sounds like a good idea. :-) -Max -- http://www.everythingsolved.com/ Competent, Friendly Bugzilla Services. And Everything Else, too. From justdave at bugzilla.org Fri Aug 11 23:06:43 2006 From: justdave at bugzilla.org (David Miller) Date: Fri, 11 Aug 2006 19:06:43 -0400 Subject: Automated Table Optimizations In-Reply-To: References: Message-ID: <44DD0D83.8020507@bugzilla.org> Benton, Kevin wrote on 8/11/06 11:23 AM: > I just learned that doing either CHECK TABLE or ANALYZE TABLE can do a lot for us in optimizing disk space utilization as well as indexing of tables in MySQL. This can improve performance of the back-end in tables that could potentially suffer from fragmenting (usually due to deletes followed by subsequent adds). I think it'd be a good idea to recommend that users implement a cron job that does these operations on a weekly-monthly basis in an optimization section of the manual. I also think it'd be good to document (if we don't already) how to set up MySQL replication in a similar section. What do others think about this? Yeah, we've had this going on b.m.o for a while. [root at mrdb01 cron.weekly]# cat analyze-bugzilla-tables #!/bin/sh # # Analyze the bugzilla tables to keep the indexes performant mysql bugs -e "analyze table `mysql bugs --skip-column-names -e 'show tables' | tr '\n' ',' | sed 's/,$//'`" -- Dave Miller http://www.justdave.net/ System Administrator, Mozilla Corporation http://www.mozilla.com/ Project Leader, Bugzilla Bug Tracking System http://www.bugzilla.org/ From lpsolit at gmail.com Mon Aug 14 14:00:11 2006 From: lpsolit at gmail.com (=?ISO-8859-1?Q?Fr=E9d=E9ric_Buclin?=) Date: Mon, 14 Aug 2006 16:00:11 +0200 Subject: Minutes of our last Bugzilla meeting (August 8, 2006) Message-ID: <44E081EB.8070704@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We had a Bugzilla meeting on August 8, 2006: http://bugzilla.glob.com.au/irc/?c=bugzilla-meeting&a=date&s=8+Aug+2006&e=9+Aug+2006 * We plan to fix 6 security bugs in our next set of releases. 4 of them have patches awaiting review (/me pokes justdave, myk and mkanat), and the last 2 still need someone to work on them. * Due to these security bugs and QA tests, we won't be ready to release before the end of the month. * justdave and jhulten will try to contact a guy from MySQL to know what is best between MyISAM and InnoDB for our DB schema. This follows a long and interesting discussion we had during the meeting, where many questions and worries appeared. Till all our questions are answered, we are going to postpone the decision to move to InnoDB. * Oracle informed us that they were still interested in having Bugzilla 3.0 working with an Oracle DB. They plan to assign two Oracle developers on it. * Supporting Oracle means we need an Oracle installation on landfill (for reviews and testing patches). But we need some Oracle patches to install it on RHEL 4, and these patches are not free. justdave will have a look if it's possible to get them anyway. * No negative feedback about new module owners so far. Our new policy is available at http://wiki.mozilla.org/Bugzilla:Owners * No new task from the roadmap http://wiki.mozilla.org/Bugzilla:Roadmap has been done for several weeks now, mainly because we are waiting for reviews. Flag code rewrite, mod_perl support and custom fields should be complete on time though, i.e. before we freeze for 3.0. About other tasks, we don't know yet, but don't be too optimistic. * A non-critical question is: which version number should we use for our next development snapshot: 2.23.3, 2.99.1 or something else? The reason for this question is that we are now officially going to release Bugzilla 3.0 (instead of Bugzilla 2.24). We don't agree about what this version number should be, but that's a minor issue which we will fix internally. :) * We plan to release Bugzilla 3.0rc1 around Christmas. We suspect we will need a second RC in Q1 2007 with Bugzilla 3.0 final a few weeks after RC2. * Our next meeting will take place on Tuesday, August 22, at 18:00 GMT (11:00 PDT, 20:00 CEST). LpSolit PS: All minutes are also available on the web: http://wiki.mozilla.org/Bugzilla:Meetings#Summaries_of_previous_meetings -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFE4IHquG3TRgecPlsRAi3kAJ9ilDOyBsV9wEObGDNf5u5OmIi7YgCcDuFe kQRWK0P4QaqlVW7nwReT/kQ= =EWlZ -----END PGP SIGNATURE----- From gerv at mozilla.org Mon Aug 14 14:50:24 2006 From: gerv at mozilla.org (Gervase Markham) Date: Mon, 14 Aug 2006 15:50:24 +0100 Subject: Minutes of our last Bugzilla meeting (August 8, 2006) In-Reply-To: <44E081EB.8070704@gmail.com> References: <44E081EB.8070704@gmail.com> Message-ID: <44E08DB0.4020304@mozilla.org> Fr?d?ric Buclin wrote: > * Oracle informed us that they were still interested in having Bugzilla > 3.0 working with an Oracle DB. They plan to assign two Oracle developers > on it. > > * Supporting Oracle means we need an Oracle installation on landfill > (for reviews and testing patches). But we need some Oracle patches to > install it on RHEL 4, and these patches are not free. justdave will have > a look if it's possible to get them anyway. So I guess the question this raises is: are the Bugzilla developers as a group free software people or open source people? That is to say, do we do Bugzilla as Free/Open Source because we believe that's the way software should be ("free software"), or because pragmatically it's what works ("open source")? If we are free software people, why exactly are we helping a proprietary database vendor sell more copies of its product? After all, they won't even make their OS patches free software, thereby asking us to use non-free software to help them. Even if we are "open source" pragmatists, I don't think anyone's ever argued that Bugzilla _needs_ Oracle. The biggest Bugzilla in the world runs pretty well on MySQL, and I know some people who think PostgreSQL is even better. ;-) Another pragmatic point: in the past, features which have been implemented but which none of the development team use have turned out to be maintenance albatrosses. Say Oracle get Bugzilla 3.0 working with their database, then the developers get reassigned. Then a few Bugzilla users start using the Oracle support. Who's responsible for keeping it working? We got PostgreSQL support because core developers wanted to use Postgres. Are there any who want to use Oracle? (And why?) Gerv From bugreport at peshkin.net Mon Aug 14 15:22:24 2006 From: bugreport at peshkin.net (Joel Peshkin) Date: Mon, 14 Aug 2006 08:22:24 -0700 Subject: Minutes of our last Bugzilla meeting (August 8, 2006) In-Reply-To: <44E08DB0.4020304@mozilla.org> References: <44E081EB.8070704@gmail.com> <44E08DB0.4020304@mozilla.org> Message-ID: <44E09530.1090301@peshkin.net> I would think Oracle would have a very strong motivation to make it easy for us to support their DB. It's not so much an issue that Bugzilla will sell Oracle by itself. If an Oracle shop wants Bugzilla, currently, they get to learn about all of Oracle's competitors. With Oracle support, they continue to be the only DB in that shop. -Joel From lpsolit at gmail.com Mon Aug 14 15:26:43 2006 From: lpsolit at gmail.com (=?ISO-8859-1?Q?Fr=E9d=E9ric_Buclin?=) Date: Mon, 14 Aug 2006 17:26:43 +0200 Subject: Minutes of our last Bugzilla meeting (August 8, 2006) In-Reply-To: <44E08DB0.4020304@mozilla.org> References: <44E081EB.8070704@gmail.com> <44E08DB0.4020304@mozilla.org> Message-ID: <44E09633.5090600@gmail.com> > If we are free software people, why exactly are we helping a proprietary > database vendor sell more copies of its product? After all, they won't > even make their OS patches free software, thereby asking us to use > non-free software to help them. That's one of the comments I made during the meeting. If we have to pay, I disagree to review anything related to Oracle. > to be maintenance albatrosses. Say Oracle get Bugzilla 3.0 working with > their database, then the developers get reassigned. Then a few Bugzilla > users start using the Oracle support. Who's responsible for keeping it > working? mkanat \o/ LpSolit From kevin.benton at amd.com Mon Aug 14 15:53:10 2006 From: kevin.benton at amd.com (Benton, Kevin) Date: Mon, 14 Aug 2006 08:53:10 -0700 Subject: Minutes of our last Bugzilla meeting (August 8, 2006) Message-ID: > Fr?d?ric Buclin wrote: > > * Oracle informed us that they were still interested in having Bugzilla > > 3.0 working with an Oracle DB. They plan to assign two Oracle developers > > on it. > > > > * Supporting Oracle means we need an Oracle installation on landfill > > (for reviews and testing patches). But we need some Oracle patches to > > install it on RHEL 4, and these patches are not free. justdave will have > > a look if it's possible to get them anyway. I would think that since Oracle is willing to commit two developers to it, that they'd be willing to give us what we need to make it work on RHEL 4. > So I guess the question this raises is: are the Bugzilla developers as a > group free software people or open source people? That is to say, do we > do Bugzilla as Free/Open Source because we believe that's the way > software should be ("free software"), or because pragmatically it's what > works ("open source")? > > If we are free software people, why exactly are we helping a proprietary > database vendor sell more copies of its product? After all, they won't > even make their OS patches free software, thereby asking us to use > non-free software to help them. > > Even if we are "open source" pragmatists, I don't think anyone's ever > argued that Bugzilla _needs_ Oracle. The biggest Bugzilla in the world > runs pretty well on MySQL, and I know some people who think PostgreSQL > is even better. ;-) > > Another pragmatic point: in the past, features which have been > implemented but which none of the development team use have turned out > to be maintenance albatrosses. Say Oracle get Bugzilla 3.0 working with > their database, then the developers get reassigned. Then a few Bugzilla > users start using the Oracle support. Who's responsible for keeping it > working? > > We got PostgreSQL support because core developers wanted to use > Postgres. Are there any who want to use Oracle? (And why?) It seems that having a poll on the bugzilla.org website would be appropriate here. Are we chasing a needle in the haystack or would this be a significant part of our userbase? It seems that the poll could look something like this: What database(s) does your company use? [_] DB2 [_] Microsoft SQL [_] MySQL [_] Oracle [_] PostgreSQL [_] Other [___________________________] --- Kevin Benton Perl/Bugzilla Developer/Administrator, Perforce SCM Administrator AMD - ECSD Software Validation and Tools The opinions stated in this communication do not necessarily reflect the view of Advanced Micro Devices and have not been reviewed by management. This communication may contain sensitive and/or confidential and/or proprietary information. Distribution of such information is strictly prohibited without prior consent of Advanced Micro Devices. This communication is for the intended recipient(s) only. If you have received this communication in error, please notify the sender, then destroy any remaining copies of this communication. From MDaskalov at technologica.com Mon Aug 14 17:10:33 2006 From: MDaskalov at technologica.com (Michael Daskalov) Date: Mon, 14 Aug 2006 20:10:33 +0300 Subject: Minutes of our last Bugzilla meeting (August 8, 2006) Message-ID: <15F26D0D9E18E24583D912511D668FF80100A3EE@exchange.ad.tlogica.com> Hi guys, As I watched your discussion on http://bugzilla.glob.com.au/irc/?c=bugzilla-meeting&a=date&s=8+Aug+2006& e=9+Aug+2006 I saw that Oracle 9i version was mentioned. This is a quite ancient version which was originally released for RedHat Advanced Server 2.1, and it is going to be desupported soon. I would propose that Oracle Bugzilla support be based on Oracle10g R2 (version is 10.2.0.1) . You CAN and everybody else CAN download a FREE (as cost) version of Oracle called Oracle XE (Oracle Express Edition). It comes as an RPM (or .deb) that doesn't need patches to run on RHEL 4. This version has some limitations (it support only 1 processor, only 1 GB RAM, and only 4GB of user data per machine). But is is legally free and royality free. It doesn't have paid support. If you are interested you can read more on http://www.oracle.com/technology/software/products/database/xe/index.htm l or you can download "Oracle Database 10g Express Edition (Universal)" for the Unicode version from http://www.oracle.com/technology/software/products/database/xe/htdocs/10 2xelinsoft.html Regards, Mihail Daskalov * Oracle informed us that they were still interested in having Bugzilla 3.0 working with an Oracle DB. They plan to assign two Oracle developers on it. * Supporting Oracle means we need an Oracle installation on landfill (for reviews and testing patches). But we need some Oracle patches to install it on RHEL 4, and these patches are not free. justdave will have a look if it's possible to get them anyway. Regards, Mihail Daskalov -----Original Message----- From: developers-owner at bugzilla.org [mailto:developers-owner at bugzilla.org] On Behalf Of Frederic Buclin Sent: Monday, August 14, 2006 5:00 PM To: developers at bugzilla.org Cc: jhulten at tragicallyleet.com Subject: Minutes of our last Bugzilla meeting (August 8, 2006) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We had a Bugzilla meeting on August 8, 2006: http://bugzilla.glob.com.au/irc/?c=bugzilla-meeting&a=date&s=8+Aug+2006& e=9+Aug+2006 * We plan to fix 6 security bugs in our next set of releases. 4 of them have patches awaiting review (/me pokes justdave, myk and mkanat), and the last 2 still need someone to work on them. * Due to these security bugs and QA tests, we won't be ready to release before the end of the month. * justdave and jhulten will try to contact a guy from MySQL to know what is best between MyISAM and InnoDB for our DB schema. This follows a long and interesting discussion we had during the meeting, where many questions and worries appeared. Till all our questions are answered, we are going to postpone the decision to move to InnoDB. * Oracle informed us that they were still interested in having Bugzilla 3.0 working with an Oracle DB. They plan to assign two Oracle developers on it. * Supporting Oracle means we need an Oracle installation on landfill (for reviews and testing patches). But we need some Oracle patches to install it on RHEL 4, and these patches are not free. justdave will have a look if it's possible to get them anyway. * No negative feedback about new module owners so far. Our new policy is available at http://wiki.mozilla.org/Bugzilla:Owners * No new task from the roadmap http://wiki.mozilla.org/Bugzilla:Roadmap has been done for several weeks now, mainly because we are waiting for reviews. Flag code rewrite, mod_perl support and custom fields should be complete on time though, i.e. before we freeze for 3.0. About other tasks, we don't know yet, but don't be too optimistic. * A non-critical question is: which version number should we use for our next development snapshot: 2.23.3, 2.99.1 or something else? The reason for this question is that we are now officially going to release Bugzilla 3.0 (instead of Bugzilla 2.24). We don't agree about what this version number should be, but that's a minor issue which we will fix internally. :) * We plan to release Bugzilla 3.0rc1 around Christmas. We suspect we will need a second RC in Q1 2007 with Bugzilla 3.0 final a few weeks after RC2. * Our next meeting will take place on Tuesday, August 22, at 18:00 GMT (11:00 PDT, 20:00 CEST). LpSolit PS: All minutes are also available on the web: http://wiki.mozilla.org/Bugzilla:Meetings#Summaries_of_previous_meetings -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFE4IHquG3TRgecPlsRAi3kAJ9ilDOyBsV9wEObGDNf5u5OmIi7YgCcDuFe kQRWK0P4QaqlVW7nwReT/kQ= =EWlZ -----END PGP SIGNATURE----- - To view or change your list settings, click here: From mkanat at bugzilla.org Mon Aug 14 17:48:29 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Mon, 14 Aug 2006 10:48:29 -0700 Subject: Minutes of our last Bugzilla meeting (August 8, 2006) In-Reply-To: <44E08DB0.4020304@mozilla.org> References: <44E081EB.8070704@gmail.com> <44E08DB0.4020304@mozilla.org> Message-ID: <1155577709.3588.10.camel@es-lappy> On Mon, 2006-08-14 at 15:50 +0100, Gervase Markham wrote: > So I guess the question this raises is: are the Bugzilla developers as a > group free software people or open source people? I think we're all individuals, working together on a project. I'd like to see more people use Bugzilla, as a project, because I like Bugzilla. When I was working with Oracle last time they were working with us, they were really nice and helpful people. I see nothing wrong with helping them out a bit in return by reviewing their patches. And it also helps us out, because larger and larger companies can start using Bugzilla. Particularly as we're going to have mod_perl support in 3.0, it'll be a great benefit for companies who only use Oracle to be able to use Bugzilla. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From gerv at mozilla.org Mon Aug 14 20:34:13 2006 From: gerv at mozilla.org (Gervase Markham) Date: Mon, 14 Aug 2006 21:34:13 +0100 Subject: Minutes of our last Bugzilla meeting (August 8, 2006) In-Reply-To: <1155577709.3588.10.camel@es-lappy> References: <44E081EB.8070704@gmail.com> <44E08DB0.4020304@mozilla.org> <1155577709.3588.10.camel@es-lappy> Message-ID: <44E0DE45.6030803@mozilla.org> Max Kanat-Alexander wrote: > On Mon, 2006-08-14 at 15:50 +0100, Gervase Markham wrote: >> So I guess the question this raises is: are the Bugzilla developers as a >> group free software people or open source people? > > I think we're all individuals, working together on a project. > > I'd like to see more people use Bugzilla, as a project, because I like > Bugzilla. You're an open source pragmatist, then :-) I didn't make the split in order to make value judgements. However, if it turns out that the majority of us are free software people, we might want to re-evaluate our Oracle strategy. Gerv From gerv at mozilla.org Mon Aug 14 20:37:10 2006 From: gerv at mozilla.org (Gervase Markham) Date: Mon, 14 Aug 2006 21:37:10 +0100 Subject: Minutes of our last Bugzilla meeting (August 8, 2006) In-Reply-To: <44E09530.1090301@peshkin.net> References: <44E081EB.8070704@gmail.com> <44E08DB0.4020304@mozilla.org> <44E09530.1090301@peshkin.net> Message-ID: <44E0DEF6.6090605@mozilla.org> Joel Peshkin wrote: > I would think Oracle would have a very strong motivation to make it easy > for us to support their DB. It's not so much an issue that Bugzilla > will sell Oracle by itself. If an Oracle shop wants Bugzilla, > currently, they get to learn about all of Oracle's competitors. With > Oracle support, they continue to be the only DB in that shop. Right. If our main aim for Bugzilla is to be popular (pragmatism; an "open source" viewpoint) then it absolutely makes sense to support Oracle. (Free software people tend to care less about popularity; see RMS's comments in the latter half of this essay: http://www.gnu.org/licenses/why-not-lgpl.html) Gerv From jeffh at tragicallyleet.com Mon Aug 14 21:49:50 2006 From: jeffh at tragicallyleet.com (Jeffrey Hulten) Date: Mon, 14 Aug 2006 14:49:50 -0700 Subject: Minutes of our last Bugzilla meeting (August 8, 2006) In-Reply-To: <44E0DEF6.6090605@mozilla.org> References: <44E081EB.8070704@gmail.com> <44E08DB0.4020304@mozilla.org> <44E09530.1090301@peshkin.net> <44E0DEF6.6090605@mozilla.org> Message-ID: <4E5947B8-134E-43FB-871B-F26FC597B2AD@tragicallyleet.com> From what I have seen there is a fair amount of pragmatism in this group. If there wasn't why would we approve any change that is not needed by the Mozilla projects. I worked for a large financial institution as a database engineer and MySQL was NOT allowed by policy because of (what were in my mind, short sighted) interpretations of regulations and the support issue. They allowed M$SQL, Oracle and DB@ on the mainframe. From my perspective we are in a great position of being an open source 'gateway drug' for larger companies. Some companies made exceptions for MySQL or didn't care or the installation happened under the radar. Removing barriers from entry (even if we think they are stupid barriers) increases market-share which increases each of our personal capital and attracts more new developers (like me!). And if my pragmatic response means I am not revolutionary enough for RMS, so be it. On Aug 14, 2006, at 1:37 PM, Gervase Markham wrote: > Joel Peshkin wrote: >> I would think Oracle would have a very strong motivation to make >> it easy >> for us to support their DB. It's not so much an issue that Bugzilla >> will sell Oracle by itself. If an Oracle shop wants Bugzilla, >> currently, they get to learn about all of Oracle's competitors. With >> Oracle support, they continue to be the only DB in that shop. > > Right. If our main aim for Bugzilla is to be popular (pragmatism; an > "open source" viewpoint) then it absolutely makes sense to support > Oracle. > > (Free software people tend to care less about popularity; see RMS's > comments in the latter half of this essay: > http://www.gnu.org/licenses/why-not-lgpl.html) > > Gerv > > - > To view or change your list settings, click here: > From ralf.beckers at chipvision.com Tue Aug 15 07:42:41 2006 From: ralf.beckers at chipvision.com (Ralf Beckers) Date: Tue, 15 Aug 2006 09:42:41 +0200 Subject: Test Management System Message-ID: <44E17AF1.4090508@chipvision.com> Ahoi, I'm looking for a add-on to bugzilla, to help our qa/qc stuff to organize their test tasks. Currently, we do it manually with bugzilla only. Till now, I found testopia (former testrunner) and got some information about litmus. Why was litmus implemented with the target to replace testopia? Is there any reason again testopia (it looked very promising)? TIA, Ralf -- Ralf Beckers Lead Development Engineer ChipVision Design Systems AG Fritz-Bock-Strasse 5 - 26121 Oldenburg - Germany phone +49-441-35042-360 fax +49-441-35042-350 email ralf.beckers at chipvision.com www.chipvision.com From lpsolit at gmail.com Tue Aug 15 08:14:33 2006 From: lpsolit at gmail.com (=?ISO-8859-15?Q?Fr=E9d=E9ric_Buclin?=) Date: Tue, 15 Aug 2006 10:14:33 +0200 Subject: Test Management System In-Reply-To: <44E17AF1.4090508@chipvision.com> References: <44E17AF1.4090508@chipvision.com> Message-ID: <44E18269.304@gmail.com> > Why was litmus implemented with the target to replace testopia? Is there > any reason again testopia (it looked very promising)? Litmus has never been implemented with the goal to replace Testopia. Nor the opposite. As far as I know, Litmus is very specific to Mozilla products while Testopia is to be used by anyone, for example your QA team. So I would suggest that you use Testopia instead. I just checked, Testopia 1.0 RC1 is now available. If you have more questions about Litmus, you could contact Zach Lipton (zach at zachlipton.com) directly (he maintains this product). About Testopia, you can contact Gregary Hendricks (ghendricks at novell.com), the new maintainer of Testopia. Fr?d?ric Buclin From ralf.beckers at chipvision.com Tue Aug 15 08:31:16 2006 From: ralf.beckers at chipvision.com (Ralf Beckers) Date: Tue, 15 Aug 2006 10:31:16 +0200 Subject: Test Management System In-Reply-To: <44E18269.304@gmail.com> References: <44E17AF1.4090508@chipvision.com> <44E18269.304@gmail.com> Message-ID: <44E18654.3060003@chipvision.com> Ahoi, Fr?d?ric Buclin schrieb: > the opposite. As far as I know, Litmus is very specific to Mozilla > products while Testopia is to be used by anyone, for example your QA > team. So I would suggest that you use Testopia instead. I just checked, > Testopia 1.0 RC1 is now available. Being not specialized on a specific product group is a good reason to use testopia :) I'll give it a try on our test server :) Thanks for your answer, Ralf -- Ralf Beckers Lead Development Engineer ChipVision Design Systems AG Fritz-Bock-Strasse 5 - 26121 Oldenburg - Germany phone +49-441-35042-360 fax +49-441-35042-350 email ralf.beckers at chipvision.com www.chipvision.com From mkanat at bugzilla.org Tue Aug 15 16:36:08 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Tue, 15 Aug 2006 09:36:08 -0700 Subject: Minutes of our last Bugzilla meeting (August 8, 2006) In-Reply-To: <44E0DE45.6030803@mozilla.org> References: <44E081EB.8070704@gmail.com> <44E08DB0.4020304@mozilla.org> <1155577709.3588.10.camel@es-lappy> <44E0DE45.6030803@mozilla.org> Message-ID: <1155659769.2356.5.camel@es-lappy> On Mon, 2006-08-14 at 21:34 +0100, Gervase Markham wrote: > I didn't make the split in order to make value judgements. However, if > it turns out that the majority of us are free software people, we might > want to re-evaluate our Oracle strategy. Um, and also our Windows support? :-) I don't think we have to worry about this. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From jon at vmware.com Tue Aug 15 16:39:08 2006 From: jon at vmware.com (Jonathan Schatz) Date: Tue, 15 Aug 2006 09:39:08 -0700 Subject: Bugzilla contract at VMware Message-ID: <0D466187AAD7714792D27594194D17F7B21BCE@PA-EXCH01.vmware.com> Hi, VMware is seeking a bugzilla hacker for a 6 month contract with the possibility of converting to full-time. Our current environment uses three seperate bugzilla instances for resume tracking, bug tracking, and trouble ticket tracking. This particular project will be working on the trouble ticketing system. Our entire company is moving to use this particular instance, and a lot of work needs to be done on it. It's a fork from 2.18.x with some minor customizations on the back end and in the templates. You'll probably also be working on our bug tracking instance, which is a heavily customized fork of the 2.17.x branch with some really cool features that I'd like to eventually try and get rolled back into the main branch (but that's another story :-)). The ideal candidate for this position will be in the SF Bay area. The work will be done in one of our Palo Alto offices, but we may move to our San Francisco office, so transportation to either location shouldn't be an issue for you. Besides intimate familiarity with the bugzilla code base, we want someone with strong OO perl and mod_perl skills. Perforce and LDAP experience are preferred, but if you have all of the above and are lacking one of those we might still be able to make do. If this interests you, please email your resume and a short cover letter. This project has already started, so this position would start as soon as possible. I don't have compensation information yet from my manager, but it should be decided on shortly. Thanks, -jon ps - if jobs are off-topic on this list now I apologize From mkanat at bugzilla.org Fri Aug 18 16:48:22 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Fri, 18 Aug 2006 09:48:22 -0700 Subject: Groups Architecture Message-ID: <1155919702.2333.5.camel@es-lappy> There's been a lot of discussion in bug 240086, and Gerv rightfully pointed out that we should take it to the mailing list. Here's a link to the bug: https://bugzilla.mozilla.org/show_bug.cgi?id=240086 And here's a link to my latest comment on the bug, which is where I proposed a sort of re-architecture of the groups system instead of what the current patch does: https://bugzilla.mozilla.org/show_bug.cgi?id=240086#c69 So, discuss? :-) -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From gerv at mozilla.org Fri Aug 18 16:59:46 2006 From: gerv at mozilla.org (Gervase Markham) Date: Fri, 18 Aug 2006 17:59:46 +0100 Subject: Groups Architecture In-Reply-To: <1155919702.2333.5.camel@es-lappy> References: <1155919702.2333.5.camel@es-lappy> Message-ID: <44E5F202.1010704@mozilla.org> Max Kanat-Alexander wrote: > So, discuss? :-) The bug is unclear at first glance as to exactly what this large patch does. Would anyone who's read it all and understands the situation explain: - How things work now - How the patch would change them - How Max proposes to change them instead - Whether Max will have a patch for his ideas any time soon? ;-) Gerv From mkanat at bugzilla.org Sat Aug 19 00:00:55 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Fri, 18 Aug 2006 17:00:55 -0700 Subject: Webservice as a Stable API Message-ID: <1155945655.2333.47.camel@es-lappy> So, we're just about to have the basic framework for our Webservice (our XML-RPC interface) checked-in: https://bugzilla.mozilla.org/show_bug.cgi?id=webservices One of the most valuable parts of having an XML-RPC interface is that it allows external programs to interact with Bugzilla without having to know the internals of Bugzilla code. It also ought to allow external programs to interact with all Bugzillas in the same way, even different versions of Bugzilla. (Obviously starting with 3.0 and going on.) An API is most useful when it doesn't change between versions. This we call a stable API. Or, it can change, but any changes won't break old code. Making a stable API is something we've never had to do before in Bugzilla (although our Bug XML comes close), but I think it's a worthy goal for our web service interface. The things required of a stable API are: 1) A function's required parameters don't change. For example, if we have do_foo($foo_id), we can add an optional $reason argument later, but we can't add any more *required* arguments to do_foo(). So even if we allow do_foo($foo_id, $reason), we still have to accept do_foo($foo_id). An interesting side effect of this has to do with *removing* required parameters. We can remove a required parameter, but if we do, the function's parameter list can *never* get longer without some trickery. For example: do_bar($bar_id, $return_value) (where both are required) becomes do_bar($bar_id). Now what if we want to add a second argument to do_bar again? We can't, because old code will still be passing $return_value into that second argument. (Basically, the workaround is to use a third argument.) We can avoid the entire problem of removing arguments by *always* using named parameters when there is more than one argument. Named parameters are much easier all around, anyhow. 2) Nothing disappears in the function's return values, and the returned data types don't change. For example: we return a hash with 'foo', 'bar', and 'baz'. We cannot ever stop returning any of those items. If 'foo' is an integer, it must stay an integer for all of eternity. However, we could add a 'qux' item later, because that doesn't break old code. 3) Functions don't disappear. 4) If anything is unstable, it's clearly noted in the docs. This does mean that if we mark anything as stable, it should only be after careful review and testing. In my latest patch against the webservice code, I've defined both STABLE and EXPERIMENTAL as ways of describing functions, where EXPERIMENTAL means "we think it's stable but we're still testing it out". Anything labeled "EXPERIMENTAL" should become "STABLE" by a final release. My patch is here: https://bugzilla.mozilla.org/show_bug.cgi?id=349256 -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From myk at mozilla.org Sat Aug 19 00:08:19 2006 From: myk at mozilla.org (Myk Melez) Date: Fri, 18 Aug 2006 17:08:19 -0700 Subject: Webservice as a Stable API In-Reply-To: <1155945655.2333.47.camel@es-lappy> References: <1155945655.2333.47.camel@es-lappy> Message-ID: <44E65673.4070105@mozilla.org> Max Kanat-Alexander wrote: > We can avoid the entire problem of removing arguments by *always* using > named parameters when there is more than one argument. Named parameters > are much easier all around, anyhow. > Shouldn't we use named parameters even when there's just one argument (in case we add more arguments later)? -myk From bugreport at peshkin.net Sat Aug 19 14:24:49 2006 From: bugreport at peshkin.net (Joel Peshkin) Date: Sat, 19 Aug 2006 07:24:49 -0700 Subject: Groups Architecture In-Reply-To: <1155919702.2333.5.camel@es-lappy> References: <1155919702.2333.5.camel@es-lappy> Message-ID: <44E71F31.7010803@peshkin.net> Max Kanat-Alexander wrote: > There's been a lot of discussion in bug 240086, and Gerv rightfully > pointed out that we should take it to the mailing list. > > Here's a link to the bug: > > https://bugzilla.mozilla.org/show_bug.cgi?id=240086 > > And here's a link to my latest comment on the bug, which is where I > proposed a sort of re-architecture of the groups system instead of what > the current patch does: > > https://bugzilla.mozilla.org/show_bug.cgi?id=240086#c69 > > So, discuss? :-) > > -Max > It would actually not be immensely hard to make the code able to be configured for AND groups or OR groups, but it would be extremely hard to automate migration. First of all, I believe that the only place where there is "more than one lock" on a door is in bug visibility. All of the other places I can think of are single locks where AND groups are equivalent to OR groups. The current check for visibility is done by joining a bug list with.... LEFT JOIN bug_group_map ON bug_group_map.bug_id = bugs.id AND bug_group_map.group_id NOT IN( $list_of_groups_you_are_in) WHERE bug_group_map.group_id IS NULL If we want to invert that, we make it.... LEFT JOIN bug_group_map ON bug_group_map.bug_id = bugs.id AND bug_group_map.group_id IN( $list_of_groups_you_are_in) WHERE bug_group_map.group_id IS NOT NULL [and we can either let the database optimize it or drop the where clause and the "left"] This search logic **should** only exist in CanSeeBug and in Search.pm. Migrating sites from one system to another is a bit harder. If you have 3 locks on a door, you have to issue new keys to people that had every one of the 7 possible combinations of keys to the door and you have to replace the 3 locks with 7 locks. From mkanat at bugzilla.org Sat Aug 19 18:02:43 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Sat, 19 Aug 2006 11:02:43 -0700 Subject: Webservice as a Stable API In-Reply-To: <44E65673.4070105@mozilla.org> References: <1155945655.2333.47.camel@es-lappy> <44E65673.4070105@mozilla.org> Message-ID: <1156010563.3170.1.camel@es-lappy> On Fri, 2006-08-18 at 17:08 -0700, Myk Melez wrote: > Shouldn't we use named parameters even when there's just one argument > (in case we add more arguments later)? That makes sense. That would solve the whole parameters problem. We could just say that functions always take named parameters. In XML-RPC terms, this would mean that the parameters are always a . So let's say that parameters for our API functions should always be named. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From mkanat at bugzilla.org Sat Aug 19 18:05:19 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Sat, 19 Aug 2006 11:05:19 -0700 Subject: Groups Architecture In-Reply-To: <44E71F31.7010803@peshkin.net> References: <1155919702.2333.5.camel@es-lappy> <44E71F31.7010803@peshkin.net> Message-ID: <1156010719.3170.5.camel@es-lappy> On Sat, 2006-08-19 at 07:24 -0700, Joel Peshkin wrote: > Migrating sites from one system to another is a bit harder. If you have > 3 locks on a door, you have to issue new keys to people that had every > one of the 7 possible combinations of keys to the door and you have to > replace the 3 locks with 7 locks. Yeah, that's what I was thinking of doing, was during migration just creating combination groups. It would be a bit chaotic for some installations, but most large installations that I've worked with rarely have a bug in more than one group (because of the AND nature of groups). And we'd make sure to have checksetup.pl print out all the created combination groups. Rather than have a choice between AND and OR groups, I'd rather just have OR groups. -Max -- http://www.everythingsolved.com/ Everything Solved: Competent, Friendly Bugzilla and Linux Services From lpsolit at gmail.com Sat Aug 19 18:26:55 2006 From: lpsolit at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBCdWNsaW4=?=) Date: Sat, 19 Aug 2006 20:26:55 +0200 Subject: Webservice as a Stable API In-Reply-To: <1156010563.3170.1.camel@es-lappy> References: <1155945655.2333.47.camel@es-lappy> <44E65673.4070105@mozilla.org> <1156010563.3170.1.camel@es-lappy> Message-ID: <44E757EF.10607@gmail.com> > So let's say that parameters for our API functions should always be named. Sounds good to me. LpSolit From bugreport at peshkin.net Sat Aug 19 19:09:39 2006 From: bugreport at peshkin.net (Joel Peshkin) Date: Sat, 19 Aug 2006 12:09:39 -0700 Subject: Groups Architecture In-Reply-To: <1156010719.3170.5.camel@es-lappy> References: <1155919702.2333.5.camel@es-lappy> <44E71F31.7010803@peshkin.net> <1156010719.3170.5.camel@es-lappy> Message-ID: <44E761F3.4090400@peshkin.net> Max Kanat-Alexander wrote: > On Sat, 2006-08-19 at 07:24 -0700, Joel Peshkin wrote: > >> Migrating sites from one system to another is a bit harder. If you have >> 3 locks on a door, you have to issue new keys to people that had every >> one of the 7 possible combinations of keys to the door and you have to >> replace the 3 locks with 7 locks. >> > > Yeah, that's what I was thinking of doing, was during migration just > creating combination groups. It would be a bit chaotic for some > installations, but most large installations that I've worked with rarely > have a bug in more than one group (because of the AND nature of groups). > And we'd make sure to have checksetup.pl print out all the created > combination groups. > Perhaps we could prevail on Karl or Erik to write a quick script to identify the number of combinations in use and post the results?? From gerv at mozilla.org Mon Aug 21 08:59:59 2006 From: gerv at mozilla.org (Gervase Markham) Date: Mon, 21 Aug 2006 09:59:59 +0100 Subject: Groups Architecture In-Reply-To: <1156010719.3170.5.camel@es-lappy> References: <1155919702.2333.5.camel@es-lappy> <44E71F31.7010803@peshkin.net> <1156010719.3170.5.camel@es-lappy> Message-ID: <44E9760F.5050409@mozilla.org> Max Kanat-Alexander wrote: > Rather than have a choice between AND and OR groups, I'd rather just > have OR groups. That would certainly be much better - but we'd need a good idea of what deep pools we were throwing people into before we removed AND support. One option might be to tell the admin "migrating will create 37 new combination groups, as follows:" with the count of bugs in each. Then, if there are 20 of them with a single bug each, those are probably mis-categorisations. The admin could use the UI to fix them and simplify things before migrating. Gerv From gerv at mozilla.org Mon Aug 21 09:15:11 2006 From: gerv at mozilla.org (Gervase Markham) Date: Mon, 21 Aug 2006 10:15:11 +0100 Subject: Webservice as a Stable API In-Reply-To: <1155945655.2333.47.camel@es-lappy> References: <1155945655.2333.47.camel@es-lappy> Message-ID: <44E9799F.7010101@mozilla.org> Max Kanat-Alexander wrote: > We can avoid the entire problem of removing arguments by *always* using > named parameters when there is more than one argument. Named parameters > are much easier all around, anyhow. As Myk says, this sounds like a good idea to me. > For example: we return a hash with 'foo', 'bar', and 'baz'. We cannot > ever stop returning any of those items. If 'foo' is an integer, it must > stay an integer for all of eternity. However, we could add a 'qux' item > later, because that doesn't break old code. On the same principle, should we make all return values hashes? Or is that overkill? > 3) Functions don't disappear. But if we end up with a big backwards-compatibility problem, we can always add doFoo2(). > This does mean that if we mark anything as stable, it should only be > after careful review and testing. In my latest patch against the > webservice code, I've defined both STABLE and EXPERIMENTAL as ways of > describing functions, where EXPERIMENTAL means "we think it's stable but > we're still testing it out". I would hope everything is EXPERIMENTAL at the moment! > Anything labeled "EXPERIMENTAL" should > become "STABLE" by a final release. I don't agree (reasonably strongly). This effectively makes all interface stabilisation block the release. I think it's entirely fine to ship final releases with things still labelled EXPERIMENTAL. That just tells people not to use them if they want to be sure their code won't break. We should not rush stabilisation of an interface. Gerv From mkanat at bugzilla.org Mon Aug 21 16:32:14 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Mon, 21 Aug 2006 09:32:14 -0700 Subject: Webservice as a Stable API In-Reply-To: <44E9799F.7010101@mozilla.org> References: <1155945655.2333.47.camel@es-lappy> <44E9799F.7010101@mozilla.org> Message-ID: <1156177934.2422.2.camel@localhost.localdomain> On Mon, 2006-08-21 at 10:15 +0100, Gervase Markham wrote: > On the same principle, should we make all return values hashes? Or is > that overkill? That seems like a good idea to me, too. > > 3) Functions don't disappear. > > But if we end up with a big backwards-compatibility problem, we can > always add doFoo2(). Good point. > I would hope everything is EXPERIMENTAL at the moment! It is. :-) Or its undocumented, which makes it UNSTABLE. > > Anything labeled "EXPERIMENTAL" should > > become "STABLE" by a final release. > > I don't agree (reasonably strongly). This effectively makes all > interface stabilisation block the release. I think it's entirely fine to > ship final releases with things still labelled EXPERIMENTAL. That just > tells people not to use them if they want to be sure their code won't > break. We should not rush stabilisation of an interface. Okay, fair enough. Although I think release time is the time to *consider* stabilization. I think we should have some point where it's somewhat forced (or at least forced to be considered), since otherwise we could just gleefully mark everything EXPERIMENTAL forever and keep on changing it. :-) -Max -- http://www.everythingsolved.com/ Competent, Friendly Bugzilla Services. And Everything Else, too. From lpsolit at gmail.com Mon Aug 21 17:20:21 2006 From: lpsolit at gmail.com (=?ISO-8859-1?Q?Fr=E9d=E9ric_Buclin?=) Date: Mon, 21 Aug 2006 19:20:21 +0200 Subject: Bugzilla meeting Tuesday, August 22nd at 11:00 PDT (18:00 GMT, 20:00 CEST) Message-ID: <44E9EB55.2080805@gmail.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Reminder: We will meet again tomorrow, Tuesday 22 at 11:00 PDT (18:00 GMT) in the #bugzilla-meeting channel. The agenda can be seen at http://wiki.mozilla.org/Bugzilla:Meetings See you tomorrow, LpSolit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFE6etVuG3TRgecPlsRAl4jAJ9yfpH1PR7PaaSdWYxFj4bNAjcO9QCfYwna 93qfWLFT7XiyP0ZCJRjQr2E= =mvAq -----END PGP SIGNATURE----- From jeffh at tragicallyleet.com Tue Aug 22 06:55:27 2006 From: jeffh at tragicallyleet.com (Jeffrey Hulten) Date: Mon, 21 Aug 2006 23:55:27 -0700 Subject: Webservice as a Stable API In-Reply-To: <1156177934.2422.2.camel@localhost.localdomain> References: <1155945655.2333.47.camel@es-lappy> <44E9799F.7010101@mozilla.org> <1156177934.2422.2.camel@localhost.localdomain> Message-ID: <21EB3729-5300-4967-A5B8-243C522D941E@tragicallyleet.com> The way I have seen it handled elsewhere to change the url. For instance... http://bugzilla.mozilla.org/api/v1 then when we learn a bunch and want to change we move to v2. Support one version back of the api as depricated and go from there. On Aug 21, 2006, at 9:32 AM, Max Kanat-Alexander wrote: > On Mon, 2006-08-21 at 10:15 +0100, Gervase Markham wrote: >> On the same principle, should we make all return values hashes? Or is >> that overkill? > > That seems like a good idea to me, too. > >>> 3) Functions don't disappear. >> >> But if we end up with a big backwards-compatibility problem, we can >> always add doFoo2(). > > Good point. > >> I would hope everything is EXPERIMENTAL at the moment! > > It is. :-) Or its undocumented, which makes it UNSTABLE. > >>> Anything labeled "EXPERIMENTAL" should >>> become "STABLE" by a final release. >> >> I don't agree (reasonably strongly). This effectively makes all >> interface stabilisation block the release. I think it's entirely >> fine to >> ship final releases with things still labelled EXPERIMENTAL. That >> just >> tells people not to use them if they want to be sure their code won't >> break. We should not rush stabilisation of an interface. > > Okay, fair enough. Although I think release time is the time to > *consider* stabilization. I think we should have some point where it's > somewhat forced (or at least forced to be considered), since otherwise > we could just gleefully mark everything EXPERIMENTAL forever and > keep on > changing it. :-) > > -Max > -- > http://www.everythingsolved.com/ > Competent, Friendly Bugzilla Services. And Everything Else, too. > > - > To view or change your list settings, click here: > From mkanat at bugzilla.org Tue Aug 22 19:06:25 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Tue, 22 Aug 2006 12:06:25 -0700 Subject: Webservice as a Stable API In-Reply-To: <21EB3729-5300-4967-A5B8-243C522D941E@tragicallyleet.com> References: <1155945655.2333.47.camel@es-lappy> <44E9799F.7010101@mozilla.org> <1156177934.2422.2.camel@localhost.localdomain> <21EB3729-5300-4967-A5B8-243C522D941E@tragicallyleet.com> Message-ID: <1156273585.2452.14.camel@localhost.localdomain> On Mon, 2006-08-21 at 23:55 -0700, Jeffrey Hulten wrote: > The way I have seen it handled elsewhere to change the url. For > instance... > > http://bugzilla.mozilla.org/api/v1 > > then when we learn a bunch and want to change we move to v2. Support > one version back of the api as depricated and go from there. That would work, also. We could do xmlrpc.cgi?v=2 when we want to modify it. Of course, one of the major differences between us and a lot of SOAP providers is that we're a shipping product (where they're normally a service, like SalesForce). -Max -- http://www.everythingsolved.com/ Competent, Friendly Bugzilla Services. And Everything Else, too. From dwilliss at microimages.com Mon Aug 28 14:01:11 2006 From: dwilliss at microimages.com (Dave Williss) Date: Mon, 28 Aug 2006 09:01:11 -0500 Subject: favicon Message-ID: <44F2F727.6060609@microimages.com> OK, this may seem like a silly request, but can we add the favicon.ico from the bugzilla.org site to the actual bugzilla distribution? I use Firefox as my main browser and it would be nice to be able to identify bugzilla pages by the icon in the tab. Dave Williss MicroImages, Inc. From lpsolit at gmail.com Mon Aug 28 14:31:23 2006 From: lpsolit at gmail.com (=?ISO-8859-1?Q?Fr=E9d=E9ric_Buclin?=) Date: Mon, 28 Aug 2006 16:31:23 +0200 Subject: favicon In-Reply-To: <44F2F727.6060609@microimages.com> References: <44F2F727.6060609@microimages.com> Message-ID: <44F2FE3B.8010007@gmail.com> > OK, this may seem like a silly request, but can we add the favicon.ico > from the bugzilla.org site to the actual bugzilla distribution? This "silly" request is bug 132383 on b.m.o. We probably can as soon as justdave approves it. LpSolit From mkanat at bugzilla.org Sat Aug 26 00:55:16 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Fri, 25 Aug 2006 17:55:16 -0700 Subject: Bugzilla Calendar Message-ID: <1156553716.5394.17.camel@localhost.localdomain> Hey everybody. I've created a calendar for Bugzilla in Google Calendar. It contains the dates of all our meetings, and all the tentative dates for releases, freezes, and release candidates. It's a public calendar, so you can subscribe to it: ICal: http://www.google.com/calendar/ical/ih4fh7ks327mou7e3hifikokco at group.calendar.google.com/public/basic.ics XML: http://www.google.com/calendar/feeds/ih4fh7ks327mou7e3hifikokco at group.calendar.google.com/public/basic HTML: http://www.google.com/calendar/embed?src=ih4fh7ks327mou7e3hifikokco%40group.calendar.google.com -Max -- http://www.everythingsolved.com/ Competent, Friendly Bugzilla Services. And Everything Else, too. From luis at tieguy.org Tue Aug 22 11:15:48 2006 From: luis at tieguy.org (Luis Villa) Date: Tue, 22 Aug 2006 07:15:48 -0400 Subject: Webservice as a Stable API In-Reply-To: <21EB3729-5300-4967-A5B8-243C522D941E@tragicallyleet.com> References: <1155945655.2333.47.camel@es-lappy> <44E9799F.7010101@mozilla.org> <1156177934.2422.2.camel@localhost.localdomain> <21EB3729-5300-4967-A5B8-243C522D941E@tragicallyleet.com> Message-ID: <2cb10c440608220415i74b70301g71c2e4f6de338b8c@mail.gmail.com> On 8/22/06, Jeffrey Hulten wrote: > The way I have seen it handled elsewhere to change the url. For > instance... > > http://bugzilla.mozilla.org/api/v1 > > then when we learn a bunch and want to change we move to v2. Support > one version back of the api as depricated and go from there. This is what Amazon does, IIRC. > On Aug 21, 2006, at 9:32 AM, Max Kanat-Alexander wrote: > > > On Mon, 2006-08-21 at 10:15 +0100, Gervase Markham wrote: > >> On the same principle, should we make all return values hashes? Or is > >> that overkill? > > > > That seems like a good idea to me, too. > > > >>> 3) Functions don't disappear. > >> > >> But if we end up with a big backwards-compatibility problem, we can > >> always add doFoo2(). > > > > Good point. > > > >> I would hope everything is EXPERIMENTAL at the moment! > > > > It is. :-) Or its undocumented, which makes it UNSTABLE. > > > >>> Anything labeled "EXPERIMENTAL" should > >>> become "STABLE" by a final release. > >> > >> I don't agree (reasonably strongly). This effectively makes all > >> interface stabilisation block the release. I think it's entirely > >> fine to > >> ship final releases with things still labelled EXPERIMENTAL. That > >> just > >> tells people not to use them if they want to be sure their code won't > >> break. We should not rush stabilisation of an interface. > > > > Okay, fair enough. Although I think release time is the time to > > *consider* stabilization. I think we should have some point where it's > > somewhat forced (or at least forced to be considered), since otherwise > > we could just gleefully mark everything EXPERIMENTAL forever and > > keep on > > changing it. :-) > > > > -Max > > -- > > http://www.everythingsolved.com/ > > Competent, Friendly Bugzilla Services. And Everything Else, too. > > > > - > > To view or change your list settings, click here: > > > > - > To view or change your list settings, click here: > > From kevin.benton at amd.com Tue Aug 29 20:31:08 2006 From: kevin.benton at amd.com (Benton, Kevin) Date: Tue, 29 Aug 2006 13:31:08 -0700 Subject: Bugzilla Calendar Message-ID: Wouldn't it be cool if they included an RSS feed right off the HTML version of the calendar? :) --- Kevin Benton Perl/Bugzilla Developer/Administrator, Perforce SCM Administrator AMD - ECSD Software Validation and Tools The opinions stated in this communication do not necessarily reflect the view of Advanced Micro Devices and have not been reviewed by management. This communication may contain sensitive and/or confidential and/or proprietary information. Distribution of such information is strictly prohibited without prior consent of Advanced Micro Devices. This communication is for the intended recipient(s) only. If you have received this communication in error, please notify the sender, then destroy any remaining copies of this communication. > -----Original Message----- > From: developers-owner at bugzilla.org [mailto:developers-owner at bugzilla.org] > On Behalf Of Max Kanat-Alexander > Sent: Friday, August 25, 2006 6:55 PM > To: developers at bugzilla.org > Subject: Bugzilla Calendar > > > Hey everybody. I've created a calendar for Bugzilla in Google > Calendar. > It contains the dates of all our meetings, and all the tentative dates > for releases, freezes, and release candidates. > > It's a public calendar, so you can subscribe to it: > > ICal: > > http://www.google.com/calendar/ical/ih4fh7ks327mou7e3hifikokco at group.cal en > dar.google.com/public/basic.ics > > XML: > > http://www.google.com/calendar/feeds/ih4fh7ks327mou7e3hifikokco at group.ca le > ndar.google.com/public/basic > > HTML: > > http://www.google.com/calendar/embed?src=ih4fh7ks327mou7e3hifikokco%40gr ou > p.calendar.google.com > > -Max > -- > http://www.everythingsolved.com/ > Competent, Friendly Bugzilla Services. And Everything Else, too. > > - > To view or change your list settings, click here: > > From mkanat at bugzilla.org Tue Aug 29 21:13:38 2006 From: mkanat at bugzilla.org (Max Kanat-Alexander) Date: Tue, 29 Aug 2006 14:13:38 -0700 Subject: License Block for New Files Message-ID: <1156886018.2457.48.camel@localhost.localdomain> So, just so everybody knows, the MPL in fact *does* always require the "Initial Developer" block in the license notice. That means that for whoever first writes a file, they have to be listed in that Initial Developer block. -Max -- http://www.everythingsolved.com/ Competent, Friendly Bugzilla Services. And Everything Else, too.