transport layer and multiple connections

[John A Meinel]

Matthieu Moy wrote:
> John A Meinel <john at arbash-meinel.com> writes:
> 
>> The reason we didn't use a specific dictionary is because we wanted
>> finer control over whether connections are kept alive or not. 
> 
> I also realize that my patch is probably not multi-thread safe. A
> server may have to be multi-threaded, and in this case, we need either
> a mutex mechanism, or one transport per thread.
> 
>> Long term bzrlib needs to be designed that it can be loaded for a
>> long time. It may be used in a GUI (or even a server) that stays
>> open for a long time, rather than a command line program that runs
>> frequently.
> 
> It isn't stupid to keep the connection open for some time in a server
> either. I've been working with baz behind a firewal + overloaded HTTP
> proxy, and for most operations, opening the connection was the longest
> thing. Having a persistant pool of connections, that would stay in
> opened for say, a few minutes would be a considerable performance gain
> in this case. "bzr service" and "bzr shell" would be helpfull for
> that.
> 
> So, the problem is harder than I thought, but it still would be nice
> to solve it.
> 

Well, I'm not worrying about multi-threading yet, but I think you can
look at my changes from here:
http://bzr.arbash-meinel.com/branches/bzr/sftp-fix/

I had to pull Robey's changes, because the old SftpTransport was sending
the wrong username and password (it interpreted things as
sftp://pass:user@host/ instead of sftp://user:pass@host/).

My basic changes are to include a weakref dictionary so that if there is
a connection currently active for a given host, username and port, it
will be re-used.
So doing:
t = get_transport()
t = t.clone('..')
Should not try to reconnect.

Now, the case with push is more complicated, because it is willing to
create a directory if it finds that it cannot connect. (Which I somewhat
disagree with).
Because the connection is not held, I would not be able to guarantee
that it wouldn't have to reconnect.
On the plus side, it seems it sometimes takes a couple steps before a
weakref dictionary lets go of it's items. At least in my testing in the
interpreter, I could delete the last reference, and the dictionary
wouldn't update right away.
So it *may* connect only once in the above situation.

But it should handle the case where you do:

b = Branch.open_containing(path_1)
b2 = Branch.open_containing(path_2)

Where the paths are both to the same host. Which I think is useful,
especially since it lets SftpTransport optimize for the copy_to() case,
when the remote host is the same. I'm trying to work on that, but so far
I don't see that Paramiko supports a remote copy operation.
Though I'm pretty sure the sftp client does. I didn't find it in the
IETF spec, either, though.

http://carmen.cselt.it/internet-drafts/draft-ietf-secsh-filexfer-10.txt

John
=:->

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 249 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20051120/ac615c6c/attachment.pgp