memory usage in mod_perl

[Bill Barry]

Max Kanat-Alexander wrote:
> On Wed, 17 Jan 2007 08:22:41 -0500 (EST) "Jake" <jake at bugzilla.org>
> wrote:
>   
>> According to that link, Perl won't return it to the kernel, but it
>> will reuse it. So unless there's a ton of memory all being used at
>> the exact same time on a thread, it doesn't seem like even a long
>> buglist should cause memory usage to climb to 350MB.
>>     
Dependency graphs might be able to do it (especially the show all 
dependencies on all bugs one). They hit the server pretty hard.
> 	That 350MB is every single memory allocation made during the
> page call, and all of the HTML and template code needed to display that
> thing.
>   
I thought it was every memory allocation across all page calls at the 
time. mod_perl causes everything to run within the same process, and 
doesn't do very optimistic GC (if any at all).
> 	I just checked on landfill, and loading every bug that's in
> landfill (4726 bugs) on a single buglist page causes Apache to allocate
> 40MB of memory on top of whatever it already has for that process.
>   
How about the Long Summary?
> 	Although I would be interested to know the breakdown of memory
> usage of such a buglist. But without an understanding of perl's
> internals (which I mostly lack), it's hard to write a script that will
> explain that to me, when the memory is not allocated to any particular
> variable.
>   
Would getting everything from within each namespace help? I am not sure 
how to get non allocated memory that perl is using, but getting stuff 
pointed at by variables will at least provide hints where we should 
consider optimizing wouldn't it? We would need to put some sort of watch 
script in at various points of each cgi and actually use it (on bmo) for 
a little while to profile. I don't think these sort of issues will be 
found nearly as quickly (if ever) on any smaller or less used Bugzilla.