bzr fast-import performance slows over time

[Brian de Alwis]

Hi Ian.

On 13-Apr-2009, at 9:37 PM, Ian Clatworthy wrote:
> I'm seeing the same thing on pretty much every import. I'm yet to look
> into why. As you're seeing, for really deep histories, the pack times
> get huge so that's part of the story, but certainly not the only part.


It looks like there is a memory leak somewhere in bzr or fast-import,  
and much of the time comes from swapping and thrashing.  The machine  
has about 8GB of memory, and 4GB of swap.

My previous attempts to import the NetBSD git repository are  
eventually ended with the python process being killed. I finally  
twigged this morning that processes are often killed on Linux when the  
machine runs out of memory.

Today I tried running the import again (the fast-import checkpointing  
+ resuming is great), with a parallel recurring 'ps vx'.  The python  
process' %MEM just continually increases, although the rate of  
increase does slow over time.  Here's a snapshot after about 5 minutes:

   PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
28680 pts/1    DN     3:13  38717     2 8181413 5719900 73.8  
python2.5 /staff/bsd178/installs/bzr-1.14rc1/bin/bzr fast-import -

So it's now accounting for 8GB of memory.  After about 45 minutes:

28680 pts/1    RN    25:01  50241     2 10162361 6321832 81.6  
python2.5 /staff/bsd178/installs/bzr-1.14rc1/bin/bzr fast-import -

It's at 10GB.  At this point the memory used does continue to  
increase, but very slowly.  Top reports about 24% is used by the  
process, and 74% is idle.  procinfo -D doesn't report any swapping at  
this point.  Using vmstat, there's quite a lot of I/O activity, with  
many blocks being read in and written out.

After approx 2 hours (this is roughly when it started processing new  
commits):

28680 pts/1    RN    79:34  58876     2 10403445 6521684 84.2  
python2.5 /staff/bsd178/installs/bzr-1.14rc1/bin/bzr fast-import -

After approx 5 hours:

28680 pts/1    RN   238:19  59351     2 11024053 7124072 92.0  
python2.5 /staff/bsd178/installs/bzr-1.14rc1/bin/bzr fast-import -

Brian.

-- 
"Amusement to an observing mind is study." - Benjamin Disraeli