[RFC] ztransport plugin

[Alexander Belchenko]

I'm working on plugin to create transport with transparent 
gzipping/ungzipping weave files. I named it ztransport.

Background:
1) I have dial-up internet connection. *It's definitely my personal 
problem*. But when I need to pull some bzr.dev branch -- it every time 
very hard work. Each time when I need actually pull new revisions bzr 
download inventory.weave that have size of more than 2.5 MB. Plus 
additional weaves for changed files. It's very hard. Especially when 
connection breaks during pull.
2) I have not much free space on my site. *It's definitely my personal 
problem*. But sometimes people ask me why I not publish my working 
bzr.dev branch. Main problem is I have poor internet connection. So 
frequent upload changed weaves (at least inventory.weave that have size 
~2.5 MB) is hard to me.

As I say it's all my personal problems.

I'm thinking about this and one day I try to compress .bzr directory 
with zip. Result was very encouraging: size reduced down to 4 times. And 
weave files was compressed very well. Because most of them have big size 
and regural structure.

So I start to write plugin to check my idea to compress weaves on the 
fly when I do push/pull operations. I wrote first draft version of 
ztransport plugin that working with local transport. Simply to check my 
idea. This transport use raw prefix 'zfile://' that simply stripped off 
in constructor of ZLocalTransport class (inherited from LocalTransport).
In .put and .get methods I do transparent gzipping/ungzipping all files 
with names that endswith('.weave'). It's simple and dirty hack simply to 
check my idea.

After pushing my bzr.dev branch with this plugin I got this numbers:

pushed 3094 revisions (this number showed on progress bar during copying)

pure push (with LocalTransport):
	push duration:	35 minutes
	size of resulting .bzr dir:	20504215 bytes
	size of resulting .bzr/weaves dir:	16297167 bytes
	size of resulting .bzr/inventory.weave file:	2544162 bytes

push with 'zfile://' as destination:
	push duration:	48 minutes
	size of resulting .bzr dir:	4872308 bytes
	size of resulting .bzr/weaves dir:	2708340 bytes
	size of resulting .bzr/inventory.weave file:	530843 bytes

Overall compress ratio is about 1/4. And for inventory.weave this is 
about 1/5.

And reweaving each weave each time during pushing consuming much more 
time than gzipping/ungzipping weaves each time. So I think my experiment 
was successful.

The best thing with this approach is full support by this transport both 
gzipped and ungzipped weaves transparently (but probably not so fast as 
LocalTransport).

I want to create analogue ztransport for HttpTransport. But using 
modified transports with prefixes like 'zfile://' or 'zhttp://' is bad 
idea. Especially with regard to my previous post about bug with 
'file://' prefix handling.

So I think it worth to extend my plugin with zpush/zpull/zbranch/zmerge 
commands. One more: push command store last push location in main bzr 
configuration dir in branches.conf file. But pull use location from 
.bzr/parent file and prefix 'zfile://' is lost (probably because 
transport.base store path without zfile-prefix).

I'd like to hear some suggestion about improving existing ztransport and 
future ZHttpTransport. What you think about z-prefixed urls or paths 
(zfile/zhhtp) and z-prefixed command?

If in the future I will publish my branch with gzipped weaves is this 
will not to be ugly or unusable for others?


And BTW, you could branch ztransport plugin here:

bzr branch http://bzr.onembedding.com/bzr.win/plugins/ztransport

--
Alexander