[MERGE][1.8?] Make 'bzr co' more memory sensitive

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vincent Ladeuil wrote:
...

>     john> +        """For the given keys, split them up based on their prefix.
>     john> +
>     john> +        To keep memory pressure somewhat under control, split the
>     john> +        requests back into per-file-id requests, otherwise "bzr co"
>     john> +        extracts the full tree into memory before writing it to disk.
>     john> +        This should be revisited if _get_content_maps() can ever cross
>     john> +        file-id boundaries.
> 
> Can you write a test guarding from regression on that particular
> case ?
> 
> BB:tweak
> 
> Convert that to approve from me if the test above is really too
> hard to get right, but in that case, add a FIXME.

I don't really know how to write a correct test (that maximum memory is
constrained).

We could ....

1) Use a feature to see if /proc/PID/status is available (which is what
- -Dmemory uses).

2) Create a 'sparse file'. Either on disk or in a fake MemoryTransport,
which could return a file with 10MB of un-interesting lines (all letter
a, etc). And then commit many of them.

3) Then do a checkout of 100 10MB files and assert that peak memory
consumption is < 100MB. (If we unpacked everything at once, it would
peak at 1GB). (run_bzr_subprocess('bzr co -Dmemory') and assert that the
output has "VmPeak XXXX" with a value less than 100MB.

By using trivially compressible files, they shouldn't take up a lot of
disk space, and the in-memory constraint is also trivial, as you can
re-use the same string so it doesn't take much to *create* the file.

It feels a bit... ugly, but it isn't the worst way to do it.

What do you think?
> 
> On the performance implications, I'd say that this fix should go
> in *even* if it makes the operation slower on smaller cases. We
> can't reasonably pretend it's a performance regression since for
> large projects the actual implementation is not viable.
> 
>       Vincent
> 


A lot of people have noticed this in 1.7. I think we had 3-4 bug reports
on it. Do we consider it enough of a regression that we should push this
into 1.8 final?

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkjvo5EACgkQJdeBCYSNAAPVAACaAnjbCpkkxse4uCu64A5PFgGE
/usAoIaF424KwDpIb+CaWLS1vX0lXO7T
=92sz
-----END PGP SIGNATURE-----