streaming push alpha-testing

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> Andrew and I have landed a bunch of work over the last week to do
> streaming push. We'd love to get some feedback before we freeze for
> release - at the moment the protocol isn't frozen in stone, but once its
> in a release its hard[er] to change.
> 
> How to test:
> 
>  * get a copy of bzr.dev on your server
>  * with a matching copy on your client
>  * build both ('make' is sufficient)
>  * push to a bzr url using bzr.dev on both ends. If your server is
>    a bzr+ssh server, you need to specify BZR_REMOTE_PATH. For instance:
>    BZR_REMOTE_PATH=/home/rob/local/bin/bzr ~/source/bzr.dev/bzr push
>    bzr+ssh://host/tmp/demo
> 
> I think we really are most concerned about unexpected regressions with
> this code. I know of one possible regression with stacked branches and
> bzr.dev which jml reported, but if there are others I would be
> surprised. 
> 
> For me, pushing up a small (1 commit) branch stacked on bzr.dev I see a
> modest improvement:

So one issue I have. You removed some of the progress information, and
don't really provide a way to provide it.

This hasn't been a problem in the past, because Pack=>Pack fetching used
Packer, which had its own progress notes as part of _copy_nodes[_graph].
However, while testing stuff like gc/chk fetching [which always use the
generic path], we don't have any indication of how many texts, etc, we
are copying.

The way I did it (at one point) was to wrap "get_record_stream(keys)" in
a helper that just did:
 for idx, record in get_record_stream(keys, ...):
  pb.update('copying', idx, len(keys))

I realize this may seem trivial, but having no progress information for
~10min is a non trivial change.

Do you have any idea how to hook in a reasonable progress structure into
this code? I'm concerned that with it being a multi-pass/try-to-stream
and not think too much in advance system, we won't have a way to present
any idea of how far along we are. Sure we would still have the "copied
XX bytes" progress, but honestly I think we can do better.

I think having "copying inventory 124" is worthwhile (even if we don't
know how many total we will copy, showing how many we *have* copied is
reasonable). Even further, though, the current code *does* know how many
"keys" it will be requesting.

Should we be putting "get_nested_progress_bar()" into the
get_record_stream code?

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmi1RkACgkQJdeBCYSNAAPwdQCeOuzZP4Yt1lI7/iMRFMn+gmds
iF0AoIi4puT8z7BfuRug3fN6mI4gZWWA
=QR5m
-----END PGP SIGNATURE-----