Problems with new streaming API

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

John Arbash Meinel wrote:
> Martin Pool wrote:
>> On 17/01/2008, John Arbash Meinel <john at arbash-meinel.com> wrote:
> 
>>> Can you attach to a running python interpreter with pdb?
>> I would suggest that you ssh to the server host, start bzr listening
>> on a tcp socket (on your presumed safe home network), then you can
>> interrupt it with Ctrl-\ to get into pdb.  This assumes it is not
>> something ssh-specific, which seems unlikely here.
> 
>> Or if you just interrupt it, you might get a stack trace in the
>> server's .bzr.log.
> 
> 
> Actually, it seems to be an interaction with SSH. When I used "bzr://"
> it only took 3.3minutes again. I just started another "bzr+ssh://" and
> it has been sitting there for 10 minutes.
> 
> It may be something with --inet or it may require ssh, I'm still
> investigating, though it seems like it will take a while for this one to
> finish. I'm trying to leave it alone to get an accurate -Dhpss -Dtiming
> result. At the moment it seems like the time is spent between
> "Repository.stream_revisions_chunked()" and when it gets the "ok" back.
> That might just be that the log hasn't flushed yet.
> 
> John
> =:->
> 

So this one finally finished, but with a very weird result. Here is a
simple excerpt:
8.301 hpss call:   'Repository.stream_revisions_chunked',
              'home/jameinel/dev/bzr/', 'pqm at pqm.ubuntu...
1198.439    result:   1190.134s  'error', "Generic bzr smart protocol
              error: bad request 'Repository.stream_revisions_chunked'"

Now, that first line is 900,000 characters long (it is the full set of
revisions).

Something is wrong about it taking 1200 seconds just to return a "not
valid request" response.

Which makes me feel like the problem lies in the request parser.
Probably because the request are not length prefixed. So I imagine it is
reading it 1 character at a time. I imagine I'm accidentally connecting
to bzr 1.0.0 in that request (since it is on the system search path,
rather than my local search path.)

So I'm issuing the request again, this time using BZR_REMOTE_PATH to
ensure I'm connecting to the right one.

I'm still getting a *lot* of cycles waiting for the response. And right
now it is spinning on:
futex(0x81f0890, FUTEX_WAKE, 1)         = 0
brk(0x957d000)                          = 0x957d000
futex(0x81f0890, FUTEX_WAKE, 1)         = 0
brk(0x95ea000)                          = 0x95ea000
futex(0x81f0890, FUTEX_WAKE, 1)         = 0
futex(0x81f0890, FUTEX_WAKE, 1)         = 0


Because this seems to be an interaction with bzr+ssh (since bzr did not
show it). I'm thinking that reading 1 character at a time is costing us
a lot when tunneled over ssh. And when you have to pay that cost *
900,000 characters we just fall over and die.

Now, that doesn't explain why it didn't die for the .stream_revisions()
request, so it could be something else.

I'm keeping the client-side -Dhpss -Dtiming runs if someone wants to
look at them. (I suppose I could also keep a --lsprof output while I'm
at it, since it shouldn't add much overhead.)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHj74aJdeBCYSNAAMRAo2DAJ97s6Wzm8aAMLcwVDYf35+yFPyWxQCggaAo
encufKJGhylXKfxgQNR1lzc=
=CpKA
-----END PGP SIGNATURE-----