Reactive roadmap

[Simon Davy]

On Mon, Mar 14, 2016 at 2:28 PM, Cory Johns <cory.johns at canonical.com> wrote:
> On Tue, Mar 8, 2016 at 9:19 AM, Simon Davy <simon.davy at canonical.com> wrote:
>>
>> 1) Is there a roadmap for reactive? A target for a stable 1.0 release,
>> or similar? We'd ideally like a stable base to build from before
>> committing to use a new framework, having been (re)writing/maintaining
>> charms for 4+ years now :)
>
>
> The layered & reactive approach saw adoption more quickly than I expected,
> which is both great and unfortunate for working out the kinks.  That said,
> the core concepts of both aspects of this approach have been fairly stable,
> with additions rather than breakages.  There is a 2.0 release of charm-tools
> coming, to coincide with Juju 2.0, but again, the build process should be
> backwards compatible.  There are some issues with charms.reactive that may
> require incompatible changes to fix, but that would be for 2.0 and the
> semantic version range that the base layer uses gives us flexibility there,
> though that leads in to your next point.

That's helpful to know, thanks.

>> 2) Layer pinning. Right now, layers are evolving fast, and the lack of
>> pinning to layer versions has caused charm builds to break from day to
>> day. Is this a planned feature?
>
>
> There has been quite a bit of discussion about this and I don't think either
> side has swayed the other.  On the one hand, as you note, there is a valid
> argument for needing to avoid breakage and in these early days, layers are
> evolving quickly.
>
> On the other hand, we want to strongly encourage charm authors to always
> sync the layers and interfaces they use to take advantage of the
> improvements and fixes from upstream, lest charms end up stagnating.  And
> that means encouraging backward-compatibility in the layers.  To that end,
> it has been suggested that layers be handled like interface protocols in
> that, if you need to make an incompatible change, you can fork the layer
> with a different name and both can coexist until one supplants the other.

That is a reasonable policy, feels like a similar discussion in the
other thread about upgrading elasticsearch/kibana charms. But, as a
policy alone, it's not enough to trust for backwards compatibility. We
have that policy in Ubuntu, particularly for SRUs and security
updates. But it's backed up with significant testing, and thus we
trust it not to break stuff (or rather, not to break stuff with a high
enough frequency to matter). Layers/interfaces, as of yet don't have
that assurance.

So, regards the trade-off between the two approaches, we would always
come down on the former approach. This is because our experience
across the board with all dependency management tools we use is that
pinning to versions is essential for reliability.  We pin our
dependent code branches, we pin our python packages, we pin our npm
packages, we pin our charms, we pin/hold key apt packages. Because if
we don't an upstream change can break the CD pipeline and then we
can't deploy.

This 'vendor/pin all the things your app depends on' approach is part
of the use case for snappy, and docker. Also, a comparison to golang
tooling seems apt. Go only needs build time dependencies too, and yet
the standard approach is to vendor all the things.

Yes, there's a downside, it's painful to keep things in sync. If you
want to 'strongly encourage' folks to keep up to date, some good
tooling around vendoring/syncing would be my recommendation..

I suspect the reality is that if you went along the lines of the
second argument, then people would just work round it anyway (like we
are doing right now).

A thought that's been brewing for a while: what if layers/interfaces
were available as python wheels, and interfaces.juju.solutions
provided an additional PyPI index? Then you could leverage the
existing tooling of pip and requirements.txt to provide your
dependency management?


> Additionally, as Stuart pointed out with tongue-in-cheek, the local repo can
> serve that purpose, and with charm-tools 2.0 that will become easier with
> the pull-source command (https://github.com/juju/charm-tools/pull/125).

OK, so a standard tool as a part of charm-tools to vendor/sync layers
would work for us I think.

>> 3) Downloading from the internet. This issue has been common in
>> charmstore charms, and is discouraged, AIUI. But the same issue
>> applies for layers, and possibly with more effect, due to a layer's
>> composibility.  We simply can not utilise any layer that downloads
>> things from github or similar, and I'm sure others are in a similar
>> situation.
>
>
> Again, as Stuart mentioned, this is actually *better* with layers than it
> has been in the past, because layers encourage charm dependencies to be
> bundled in the charm's wheelhouse,

Right, but that only works for python dependencies...

> and layers can provide more consistent
> adoption of best practices (it only needs to be done right once, instead of
> every time).
>
> (Note, though, that's there's an open issue with regards to some few Python
> modules and network-restricted deployments:
> https://github.com/juju/charm-tools/issues/117)
>
>>
>> We're aware of resources, but not convinced this is a
>> scalable solution for layers, as it makes using a charm that has
>> layers that require resources much more complex. So, some clarity in
>> this area would be helpful.
>
>
> I can't say that I understand why you think resources and layers would make
> things more complicated.  They seem like they will work well together and
> would solve the other half of charms in network-restricted environments
> quite nicely.

They do, but the problem I see (maybe I've missed something about
resources) are these:

1) if my charm has 1 resource, and it uses a layer foo that requires
another resource, that's 2 resources I now need to manage, one of
which is not directly related to my charm. If it uses layer bar with
another resource, the problem increases. It means, rather than the
layer being plug-and-play, using it adds additional resources to be
managed by your deployment tooling.

2) If layer foo requires a resource, and I use that layer in 3
different charms, I now have three per-service resources to manage,
which are actually the same resource (AIUI, please correct me if I'm
wrong)

I think resources will work well when the composable unit is a
charm/service, but I think when composing layers, I suspect it will
not scale as well.

Hope I've managed to explain myself.

-- 
Thanks