Schema for Juju RPC messages

[roger peppe]

On 28 July 2016 at 10:01, Mark Shuttleworth <mark at ubuntu.com> wrote:
> On 28/07/16 10:42, roger peppe wrote:
>> On 28 July 2016 at 01:07, Rick Harding <rick.harding at canonical.com> wrote:
>>> However, an API client in any language should never be auto generated.
>> This is a strong statement. I feel that, as with most things in
>> software engineering,
>> there's a trade-off. Personally I'm with Katherine "use the schema to
>> auto-generate the boilerplate, and then wrap that with a small shim
>> that is more crisp".
>
> Theres room for gray areas, and then there's times when it's better just
> to be crisp and clear. In this case, the crisp and clear house position
> is that we don't consider auto-generated code to be useful for anything
> other than auto-generated tests and auto-generated validation.
>
> In this example, you have the classic problem of how to deal with
> change. The auto-generated part will change in "brutal and blunt" ways,
> and your hand-crafted shim will need to know how to wrap it, which means
> you are constantly having to bridge the mental gap between what actually
> changed, how best to represent this idiomatically, and how the
> auto-generated blob represented the change.

I like this description. The auto-generated part
represents the reality of the network API. Having that auto-generated
part is really useful though, particularly in a statically typed language,
because the compiler will tell you where your nice hand-crafted shim
is now wrong.

And FWIW I do think that an auto-generated API can sometimes be very nearly as
nice as hand-generated code. For example, to take a concrete
example from github.com/juju/idmclient:

    userInfo, err := client.User(&params.UserRequest{User: "bob"})

is only a little less nice than the call one might manually craft:

    userInfo, err := client.User("bob")

It's a little more verbose, but just as clear to read, I think.

If a call isn't in a public API (true of 70% of entry points in the Juju API)
and there's only one or two invocations of that call in the entire code base
(true of many, perhaps most, of the calls), I'm wondering if it's
actually worth the extra effort (code + tests + additional ongoing maintenance)
to make those one or two lines of code that little bit prettier?

>> Generating the initial low level API client has some great advantages - it saves
>> a lot of developer time and it means that the higher level API has no chance of
>> getting trivial details like the spelling of field names wrong.
>
> This is auto-validation as Tim described it. By all means, before we
> pass anything over a REST call, a dumb blob of code can check that
> against a schema. On the other side, before any REST call is processed
> by human-written code, the same hunks can be checked against the same
> schema. That way nothing leaves the client and nothing enters the
> hand-crafted code domain on the server without having passed a schema
> validation. But this code is entirely invisible to the person crafting
> the API or using the API or writing the server-side responses.

In a statically-typed language like Go, auto-validation generally consists of
converting between a bunch of bytes and static types. If one is going
to do that, I think it's always going to have to be visible at some level,
to the person crafting the language-level API, even if not to the
caller of that.

>> If the statement "an API client in any language should never be auto generated",
>> was true, I don't think the likes of Amazon and Google would do that for their
>> APIs (see https://godoc.org/github.com/aws/aws-sdk-go/service/acm and
>> https://godoc.org/google.golang.org/api/analytics/v3 for example).
>>
>> I think it's nice to make nice idiomatic polished language-specific APIs for
>> well-used APIs, but in Juju's case the API is large (422 entry points
>> at my last count) and most API calls are used only by agents and
>> called only once
>> or twice in the code. The extra developer time hand-crafting code and
>> tests for these
>> calls seems to me like it could be better spent elsewhere.
>
> None of those 422 calls arrived magically. None of them showed up from
> an AI or as manna from heaven.
>
> We build them manually.
>
> A professional approach to that would be to consider, in the design
> stages, how these entry points should be *used* by people who want to
> use them to make magic happen with Juju. That would shape the design of
> the end points so that, instead of being a idiosyncratic collection of
> styles, they have a continuity and flow and follow conventions and
> patterns. At the same time, as part of that process of consideration and
> design, it would be normal to mock up the developer experience in JS, Go
> and Python.

Yes! Consistency and cleanliness of API is really important.

> 422 entry points maps to about 23 per member of the Juju core teams. I
> think that's tractable and worth doing.

Here's a quick stab at showing the public client-facing websocket
API provided by Juju:
http://rogpeppe-scratch.s3.amazonaws.com/juju-client-api-doc.html

134 entry points in total.

  cheers,
    rog.