[RFC] Allow compressed requests

[John Arbash Meinel]

I've been thinking about the bzr workflow using transports, and I
realized that we use get() for plain text files (like knit indexes,
branch format, etc). And we use readv() for the other files (.knit data).

So I was thinking that we could actually change our requests, such that
if we are doing a plain get() we make a request and allow compression.
And if we do a readv() then we don't do compression.

It should be fairly easy to do for http (both pycurl and urllib), and it
also would work for the smart server.

Basically, this would cut out a lot of our download bandwidth for knit
indexes. It also would help for 'revision-history'.

In a simple test, I just copied all of the knit indices from
.bzr/repository into a temp directory, and then compressed all of them.
The results are:

normal: 4.6MB
.gz: 1.8MB
.bz2: 1.6MB

And for bzr.dev's revision-history:
110K revision-history
 30K revision-history.bz2
 34K revision-history.gz

So we are talking a bandwidth savings of about 3:1. And this is the
indices, which is what we generally download completely, whenever we
need to access them.

For implementation, either we could add a hint parameter to get() which
we discussed in the past. But actually, I was thinking that we don't
really need it. Instead we can use get_uncompressed() or maybe
_get_uncompressed() which is what readv() would use, and just switch the
default to compressing if the transport supports it.

The default implementation would just return get(), but all of the
existing transports would be able to do the right thing. They would, of
course, always return uncompressed bytes.

The only concern I have, is that sometimes asking for a transfer
encoding of compressed would get confused whether or not the actual
content was compressed. I guess that was a content encoding versus
transfer encoding, or something like that.

Anyway, it seemed like something that could be quite beneficial, and
just wondered what people thought.

John
=:->

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20061206/6b93c7fb/attachment.pgp