Too much data being downloaded with sftp

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 9/10/2010 10:40 AM, John Arbash Meinel wrote:
> On 9/9/2010 9:56 PM, Martin Pool wrote:
>> Hi Brent,
> 
>> The short answer is if at all possible you should install bzr on the
>> server and use bzr+ssh instead; most of our performance work is now
>> focussed on that not sftp or ftp.
> 
>> From your logs the problem may be that something is overflowing a
>> cache so it's repeatedly reading the same object from the server.  You
>> could file a bug on this at <https://bugs.launchpad.net/bzr/>.  This
>> sounds familiar but I can't find a dupe.
> 
> It would certainly be good to understand why we aren't caching that
> object. It looks like a GroupCompressBlock that is bigger than we
> expect. Perhaps you just have a single file that is significant in size?
> (Though it would have to have several revisions or we wouldn't try to
> download it repeatedly...)
> 
> John
> =:->
> 

Just another thought. Maybe it is very redundant data that compresses
very well with zlib (consider 50MB of all NULLs.). It might be possible
that we see the uncompressed size of the GCBlock and decide it is too
big to hold in memory for now. I'd have to see the details.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkyKUboACgkQJdeBCYSNAANHLACfQN9jRk8iA2yEUdh4D7KjGBes
XVYAoKeMw5QOVySsjCmjFNzReVsCZTZ3
=Jhjz
-----END PGP SIGNATURE-----