Schema for Juju RPC messages

[Mark Shuttleworth]

On 28/07/16 10:42, roger peppe wrote:
> On 28 July 2016 at 01:07, Rick Harding <rick.harding at canonical.com> wrote:
>> However, an API client in any language should never be auto generated.
> This is a strong statement. I feel that, as with most things in
> software engineering,
> there's a trade-off. Personally I'm with Katherine "use the schema to
> auto-generate the boilerplate, and then wrap that with a small shim
> that is more crisp".

Theres room for gray areas, and then there's times when it's better just
to be crisp and clear. In this case, the crisp and clear house position
is that we don't consider auto-generated code to be useful for anything
other than auto-generated tests and auto-generated validation.

In this example, you have the classic problem of how to deal with
change. The auto-generated part will change in "brutal and blunt" ways,
and your hand-crafted shim will need to know how to wrap it, which means
you are constantly having to bridge the mental gap between what actually
changed, how best to represent this idiomatically, and how the
auto-generated blob represented the change.


> Generating the initial low level API client has some great advantages - it saves
> a lot of developer time and it means that the higher level API has no chance of
> getting trivial details like the spelling of field names wrong.

This is auto-validation as Tim described it. By all means, before we
pass anything over a REST call, a dumb blob of code can check that
against a schema. On the other side, before any REST call is processed
by human-written code, the same hunks can be checked against the same
schema. That way nothing leaves the client and nothing enters the
hand-crafted code domain on the server without having passed a schema
validation. But this code is entirely invisible to the person crafting
the API or using the API or writing the server-side responses.


> If the statement "an API client in any language should never be auto generated",
> was true, I don't think the likes of Amazon and Google would do that for their
> APIs (see https://godoc.org/github.com/aws/aws-sdk-go/service/acm and
> https://godoc.org/google.golang.org/api/analytics/v3 for example).
>
> I think it's nice to make nice idiomatic polished language-specific APIs for
> well-used APIs, but in Juju's case the API is large (422 entry points
> at my last count) and most API calls are used only by agents and
> called only once
> or twice in the code. The extra developer time hand-crafting code and
> tests for these
> calls seems to me like it could be better spent elsewhere.

None of those 422 calls arrived magically. None of them showed up from
an AI or as manna from heaven.

We build them manually.

A professional approach to that would be to consider, in the design
stages, how these entry points should be *used* by people who want to
use them to make magic happen with Juju. That would shape the design of
the end points so that, instead of being a idiosyncratic collection of
styles, they have a continuity and flow and follow conventions and
patterns. At the same time, as part of that process of consideration and
design, it would be normal to mock up the developer experience in JS, Go
and Python.

422 entry points maps to about 23 per member of the Juju core teams. I
think that's tractable and worth doing.

We have spent a considerable effort to *design* and curate the CLI in
2.0. People love the result. I think you'll find that if you make the
same commitment to the API, developers love the result.


> FWIW there is a proposal for strict field checking in the Go
> encoding/json package (https://github.com/golang/go/issues/15314)
> which would fix the specific issue raised at the start of this thread.
> It's trivial to add that feature (4 lines of code) and pending Go 1.8,
> we could potentially use a forked version of encoding/json for
> unmarshaling API calls to gain it - a quick win for marginal effort.
> That would be more-or-less equivalent to Tim's proposal, I think, but
> without the performance hit.

That sounds like a nice win, yes.

Mark