[step 11] request: 12 steps towards a high performance server

[Robert Collins]

On Fri, 2006-09-15 at 11:03 -0500, John Arbash Meinel wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Andrew Bennetts wrote:
> > On Wed, Sep 13, 2006 at 02:46:30PM -0500, John Arbash Meinel wrote:
> 
> ...
> 
> >> ^- So the opening request is a plain 'hello' in order to get the 'ok +
> >> version number'? I would rather avoid things that other protocols might
> >> want to use.
> > 
> > In general servers and clients have to cope with being accidentally connected to
> > something speaking another protocol.  The worst that will typically happen is
> > both sides wait indefinitely for each other to say something, and eventually
> > timeout.  Often they'll immediately both notice the other end is speaking
> > gibberish, and give up.
> > 
> > So I don't think there's much need to worry about this.  Do you have a specific
> > problem in mind?
> 
> Well, I've heard of people multiplexing HTTPS and SSH on port 443 to get
> around firewall restrictions, etc. And also, I think it would be good to
> be able to multiplex regular HTTP and hpss traffic. So that old clients
> can just connect to a plain HTTP port. But I don't have any major issues
> here. Just that a plain 'hello' seems a little too simple.

old clients can just connect to HTTP and new ones connect to HTTP and
get the smart server in the current plan: new clients will send a
request to the url of the smart server : something like a POST to
HTTP://host/path/of/branch/.bzr/smart

Apache will pass that back to the smart server which will respond to
that request.

If we get 404 back, then the hpss is not available for that bzr dir.


> > Transport.close/disconnect is a problematic issue Andrew and I are talking about
> > at the moment, probably we'll have a spec for 0.12 about this. Consider that we
> > want Transport to behave as a Proxy for a remote directory. For testability and
> > model-compatability with HTTP its easier if this is essentially stateless, and
> > close/disconnect are most definately not stateless. Consider what happens when
> > we have 3 or four transports sharing a single stream-based client - i.e. SFTP
> > transports, FTP transports, and HTTP with connection sharing, SmartTransport
> > with its client: does disconnect that ONE Transport is finished with, or that
> > they all should get disconnected, but an implicit reconnection may occur?
> > Consider the impact of that ambiguity with the requirement that we only prompt
> > for SSH passwords etc once during a single session with a single remote server.
> > (And that we dont ever get a copy of the SSH passphrase either!)
> 
> I'm not sure that Tranports are really meant to be stateless. Even with
> HTTP, if you have keep-alive, it is no longer 100% stateless. There is
> at least a connection that is being maintained.
> 
> I do understand the problem we have of multiple Transports all connected
> to the same socket, and we are done with one of them, but not all of them.

Our 'Transport is a directory' model is completely stateless. The HTTP
connection sharing is hidden in the layering : the HttpTransport should
never ever behave differently depending on whether there is an existing
tcp socket it can use or not. For instance, if theres a persistent
connection present, and it sends a request, and gets a zero size read
(i.e. the persistent connection has been timed out by the server, we
just encountered shutdown(SD_SEND) from the other end), the transport
must send the request again on a new connection.

A stateful model is one where we would get exposed to that error and
have to know its even possible to occur.

I do say essentially stateless - there is one piece of key state, and
thats the 'cwd' concept that Transport embodies.

For fun, imagine a transport that sent and received emails. Now thats
stateles ;).


> In my mind, it would be good to send the checksum as part of the header.
> So ok\1checksum: XXYYZZ\1bytes: 100\ndata...done\n. Or however you want
> to split it up. \1 or \n, or whatever.
> 
> I'm just thinking that doing it at the beginning is better than at the end.

So do we :). 

> ...
> 
> >> v-- I didn't see any tests with a LocalTransport as the backing server.
> >> It seems reasonable to have one that checks to make sure files show up
> >> on the filesystem.
> >>
> > [...]
> > 
> > There are already tests that check that files turn up on the backing transport.
> > I think the coverage of this aspect is already fine.
> 
> I didn't see any tests that did 'put('foo')'
> self.failUnlessExists('foo'). But that doesn't mean they weren't there.

Andrew, could you have a double-check for this? I'm heading out in a
little bit.

> 
> ...
> 
> > It's the thing that takes bytes in, dispatches to a handler method, and returns
> > bytes.  Calling it a "server" seems accurate to us.
> 
> Except that you have another "server" layered above it, right? (Whether
> it is TCP or SSH, etc). It seems weirder to me to have a Server which
> spawns a Server, which creates Server requests. Rather than having a
> Handler which hands off to another Handler.
> 
> But that is just my 2c.

Well, perhaps we can find better names when theres a few more eyes on
the code :).


> > Got a better name? FWIW squid uses http_port to mean a field that accepts
> > 'host:port', 'port' etc.
> > 
> > Also we should set a default port for the bzr:// url scheme.
> 
> Well, earlier I mentioned '--address'. I don't think it is critical to
> not require people to specify a host, since everyone knows about
> 'localhost'. But, whatever. I do agree that we want to come up with a
> relatively unused port for the 'bzr://' scheme. And I'm not super facist
> about the --port flag.

People not need to specify a host at the moment. --port 1234 works, as
does --port localhost:1234.

I think --address 1234 looks more weird than --port localhost:1234

-Rob
-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060916/da0998be/attachment.pgp