[MERGE][RFC] Proposed version 3 of smart server protocol

[Martin Pool]

+Conventional requests and responses
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By convention, most requests and responses have a simple "arguments plus
+optional body" structure, as in earlier protocol versions.  This section
+describes how such messages are encoded.  All requests and responses
+defined by earlier protocol versions must be encoded in this way.
+
+Conventional requests will send a sequence of:
+
+* Arguments (a STRUCTURE of a tuple)
+
+* (Optional) body
+
+  * Single body (BYTES), or
+
+  * Streamed body (multiple BYTES parts), followed by a status (ONE_BYTE)
+
+    * if status is "E", followed by an Error (STRUCTURE)
+

You should write this into EBNF too.  How about this:

STANDARD_V3_REQUEST = ARGUMENTS_V3 BODY? STATUS
BODY = BYTES+
STATUS = ONE_BYTE ( ERROR | SUCCESS )
ERROR_V3 = "E" structure-of (sequence (ERROR_NAME ERROR_ARGS))
SUCCESS = "S"

This does not quite correspond to your thing, because it has a status even
after a single body section.  I think you are ambiguous there.  It is ok
to me to either say it's always streamed and there's always a trailer, or
that there is a special case of a single chunk which cannot fail.

I have one reason to have it always streamed: if there is a special case
of "just one body block" people may tend to write code that expects it,
and will have trouble if we later change the other party to stream it.

Error indicates that the client encountered an internal error between
starting to send the request and completing it.

STANDARD_V3_RESPONSE = STATUS ARGUMENTS_V3 BODY? STATUS

The second status can be used to show an error occurred during the
streamed body.

+Conventional responses will send a sequence of:
+
+* Status (ONE_BYTE)
+
+* Arguments (a STRUCTURE of a tuple)
+
+* (Optional) body
+
+  * Single body (BYTES), or
+
+  * Streamed body (multiple BYTES parts), followed by a status (ONE_BYTE)
+
+    * if status is "E", followed by an Error (STRUCTURE)
+
+In all cases, the ONE_BYTE status is either "S" for Success or "E" for
+Error.  Note that the streamed body from version two is now just multiple
+BYTES parts.
+
+For new methods, these sequences are just a convention and may be varied
+if appropriate for a particular request or response.  However, each
+request should at least start with a STRUCTURE encoding the arguments
+tuple.  The first element of that tuple must be a string that names the
+request method.  (Note that arguments in this protocol version are
+bencoded.  As a result, unlike previous protocol versions, arguments in
+this version are 8-bit clean.)

We should make sure that any possible combination still allows us to
definitively tell when the response has finished; it's not clear from your
text that we can.  On the whole I think you need a structure like

   request = arguments (bytes|arguments)+ (success|failure arguments)
   response = (success (bytes|arguments)+ (success|failure arguments)
	       | failure arguments)

-- 
Martin