Schema for Juju RPC messages

[roger peppe]

On 28 July 2016 at 01:07, Rick Harding <rick.harding at canonical.com> wrote:
> However, an API client in any language should never be auto generated.

This is a strong statement. I feel that, as with most things in
software engineering,
there's a trade-off. Personally I'm with Katherine "use the schema to
auto-generate the boilerplate, and then wrap that with a small shim
that is more crisp".

Generating the initial low level API client has some great advantages - it saves
a lot of developer time and it means that the higher level API has no chance of
getting trivial details like the spelling of field names wrong.

If the statement "an API client in any language should never be auto generated",
was true, I don't think the likes of Amazon and Google would do that for their
APIs (see https://godoc.org/github.com/aws/aws-sdk-go/service/acm and
https://godoc.org/google.golang.org/api/analytics/v3 for example).

I think it's nice to make nice idiomatic polished language-specific APIs for
well-used APIs, but in Juju's case the API is large (422 entry points
at my last count) and most API calls are used only by agents and
called only once
or twice in the code. The extra developer time hand-crafting code and
tests for these
calls seems to me like it could be better spent elsewhere.

FWIW some of us are already auto-generating some client code for APIs.
For example, the Client methods in
https://godoc.org/github.com/juju/idmclient are auto-generated.
Indeed they're not the most idiomatic and beautiful API entry points ever, but
they're still easy to use, we save effort every time the API changes,
and we still have the freedom to hand-craft as many of the API methods
as we wish (the PermChecker API is an example of that
https://godoc.org/github.com/juju/idmclient#PermChecker).

With regard to the Juju API, some time ago I spent a little time
writing a tool to statically analyse it. The specific reason in that
case was to obtain this overall picture of the API
(http://rogpeppe-scratch.s3.amazonaws.com/juju-api-doc.html) but as a
precursor to the documentation, it generates a JSON representation of
the API (http://rogpeppe-scratch.s3.amazonaws.com/juju-api-doc.json).
This could be used to generate a "first cut" API client in various
languages with a relatively small amount of developer time - something
that could be layered upon to create more polished language-idiomatic
APIs without the risk of misspelling method and field names.

I think that using the server definitions of the methods as the
primary source of information for generating any schemas or clients is
a win - the Go types are already a kind of schema in themselves, and
having an independent description of the types (as it seems that we
see in https://github.com/jameinel/juju/blob/yaml-schema-rpc/rpc/jsoncodec/codec.go)
seems like just one more thing to maintain and potentially get out of
sync.

FWIW there is a proposal for strict field checking in the Go
encoding/json package (https://github.com/golang/go/issues/15314)
which would fix the specific issue raised at the start of this thread.
It's trivial to add that feature (4 lines of code) and pending Go 1.8,
we could potentially use a forked version of encoding/json for
unmarshaling API calls to gain it - a quick win for marginal effort.
That would be more-or-less equivalent to Tim's proposal, I think, but
without the performance hit.

  cheers,
    rog.