Proposal: Charm testing for 2.0

Thu Mar 17 04:38:58 UTC 2016

Good evening,

I really like the notion of a bundle possessing functional tests as an
enhancement to test coverage.  I agree with almost all of those ideas.  :-)
  tldr;  I would suggest that we consider bundle tests 'in addition to' and
not 'as a replacement of' individual charm tests, because:

*# Coverage and relevance*
Any given charm may have many different modes of operation -- features
which are enabled in some bundles but not in others.  A bundle test will
likely only exercise that charm in the context of its configuration as it
pertains to that bundle.  However, those who propose changes to the
individual charm should know (via tests) if they've functionally broken the
wider set of its knobs, bells and levers, which may be irrelevant to, or
not testable in the bundle's amulet test due to its differing perspective.
This opens potential functional test coverage gaps if we lean solely on the
bundle for the test.

There are numerous cases where a charm can shift personalities and use
cases, but not always on-the-fly in an already-deployed model.  In those
cases, it may take a completely different and new deployment topology and
configuration (bundle) to be able to exercise the relevant functional
tests.  Without integrated amulet tests within the charm, one would have to
publish multiple bundles, each containing separate amulet tests.  For
low-dev-velocity charms, for simple charms, or for charms that aren't
likely to be involved in complex workloads, this may be manageable.  But I
don't think we should discourage or stop looking for individual charm
amulet tests even there.

A charm's integrated amulet test can be both more focused and more
expansive in what it exercises, as it can contain multiple deployment
topologies and configurations (equivalent to cycling multiple unique
bundles).  For example:  charm-xyz with and without SSL;  or in HA and
without HA;  or IPv4 vs. IPv6; or IPv4 HA vs. IPv6 HA, multicast vs.
unicast;  [IPv6 + HA + SSL] vs [IPv4 + HA + SSL]; or mysql deploying mysql
proper vs. mysql deploying a variant;   and you can see the gist of the
coverage explosion which translates to having a whole load of bundles to
produce and maintain.

*# Dev and test: cost, scale and velocity*
Individual charm amulet tests are an important piece in testing large or
complex models.  I'll share some bits of what we do for OpenStack charms as
an example.  No bias.  :-)

Each of the OpenStack charms contain amulet test definitions.  We lean
heavily on those tests to deploy fractions of a full OpenStack bundle as
the core of our CI development gate.  With [27 charms] x [stable + dev] x
[8 Ubuntu/OpenStack Release Combos], there are currently* ~432 *possible
variations of amulet tests (derived bundles of fractional OpenStacks).  A
subset of those are executed in gate, depending on relevance to the
developer's proposed change.  This allows us to endure a high velocity of
focused testing on development in these very active charms.  Because the
derived models are much smaller than the reference bundle, we can give
developers rapid and automated feedback, plus they can iterate on
development outside of our CI without having to be able to deploy a full
OpenStack.

That is not to say that we don't have acceptance and integration tests for
full OpenStack bundles.  We do that in the form of mojo specs which
dynamically deploy any number of full OpenStack bundle topologies and
configurations against multiple Ubuntu+OpenStack release combos, using
either the dev or the stable set of OpenStack charms.  It basically takes
what I've described above for amulet and allows us to pivot entire bundles
into different models automatically.  There are currently *84* such
OpenStack mojo specs with tests (bundle equivalents)

Fear not, this is mostly accomplished with bundle inheritance, yaml foo,
and shared test libraries.  We're not actually maintaining ~*516 bundles*.
But if we were to achieve the current level of coverage with bundles,
that's approximately how many there would need to be.  This includes the
upcoming Xenial and Mitaka releases.  Reduce by ~12% when Juno EOLs.  Add
12% when we hit Newton B1, and so on.

*# How I'd like to use the proposed ideas*
There are some OpenStack reference bundles in the charm store.  My
suggested approach would be to continue to leverage individual charm amulet
tests while adding functional tests to the existing charm store bundles.
That would increase test coverage, and provide a mechanism to validate
proposed changes to those specific bundles, such as to re-validate the
bundles when charm versions are revved within them.

To summarize, I am:

-1 to stopping or discouraging individual charm amulet tests

+1 for every charm containing amulet tests

+1 for every charm containing unit tests

+1 for every charm having amulet coverage in at least 1 bundle

+1 for every bundle possessing amulet tests

Also open to feedback, discussion, suggestions, kicks in the shin.

Thanks for all the great tooling and thought leadership.  We leverage the
everloving *stuff* out of Amulet.

Charm on!

-- 
Ryan Beisner
QA Engineer, Ubuntu OpenStack Engineering, Canonical, Ltd.
irc:beisner  gh/gerrit:ryan-beisner  lp:~1chb1n

On Wed, Mar 16, 2016 at 7:52 PM, Marco Ceppi <marco.ceppi at canonical.com>
wrote:

> Hello everyone!
>
> This is an email I've been meaning to write for a while, and have
> rewritten a few times now. With 2.0 on the horizon and the charm ecosystem
> rapidly growing, I couldn't keep the idea to myself any longer.
>
> # tl;dr:
>
> We should stop writing Amulet tests in charms and instead only write them
> Bundles and force charms to do unit-testing (when possible) and promote
> that all charms be included in bundles in the store.
>
> # Problem
>
> Without making this a novel, charm-testing and amulet started before
> bundles were even a construct in Juju with a spec written before Juju 1.0.
> Since then, many new comers to the ecosystem have remarked how odd it is to
> be writing deployment validations at the charm level. Indeed, as years have
> gone by and new tools have sprung up it's become clear that; having an
> author try to model all the permutations of a charms deployment and do the
> physical deploys at that charm level are tedious and incomplete at best.
>
> With the explosion of layers and improvements to uniting test in charms at
> that component level, I feel that continuing to create these bespoke
> "bundles" via amulet in a single charm will not be a robust solution going
> forward. As we sprint closer to Juju 2.0 we're seeing a higher demand for
> assurance of working scenarios, and a sharp focus on quality at every
> level. As such I'd like to propose the following policy changes:
>
> - All bundles must have tests before promulgation to the store
> - All charms need to have comprehensive tests (unit or amulet)
> - All charms should be included in a bundle
>
> I'll break down my reasoning and examples in the following sections:
>
> # All bundles must have tests before promulgation to the store
>
> Writing bundle tests with Amulet is actually a more compelling story today
> than writing an Amulet test case for a charm. As an example, there's a new
> ELK stack bundle being produced, here's what the test for that bundle looks
> like:
> https://github.com/juju-solutions/bundle-elk-stack/blob/master/tests/10-test-bundle
>
> This makes a lot of sense because it's asserting that the bundle is
> working as expected by the Author who put the bundle together. It's also
> loading the bundle.yaml as the deployment spec meaning as the bundle
> evolves the tests will make sure they continue to run as expected. Also,
> this could potentially be used in future smoke tests for charms being
> updated if a CI process swaps out, say elasticsearch, for a newer version
> of a charm being reviewed. We can assert that both the unittests in
> elasticsearch work and it operates properly in an existing real world
> solution a la the bundle.
>
> Additional examples:
> -
> https://github.com/juju-solutions/bundle-realtime-syslog-analytics/blob/master/tests/01-bundle.py
> -
> https://github.com/juju-solutions/bundle-apache-core-batch-processing/blob/master/tests/01-bundle.py
>
> # All charms need to have comprehensive tests (unit or amulet)
>
> This is just a clarification and more strongly typed policy change that
> require charms have (preferred) unit tests or, if not applicable, then an
> Amulet test. Bash doesn't really allow for unittesting, so in those
> scenarios, Amulet tests would function as a valid testing case.
>
> There are also some charms which will not make sense as a bundle. One
> example is the recently promulgated Fiche charm:
> http://bazaar.launchpad.net/~charmers/charms/trusty/fiche/trunk/view/head:/tests/10-deploy It's
> a standalone pastebin, but it's an awesome service that provides deployment
> validation with an Amulet test. The test stands up the charm, exercises
> configuration, and validates the service responds in an expected way. For
> scenarios where a charm does not have a bundle an Amulet test would be
> required.
>
> Any charm that currently includes an Amulet test is welcome to continue
> keeping such a test.
>
> # All charms should be included in a bundle
>
> This last one is to underscore that charms need to serve a purpose. This
> policy is written as not an absolute, but instead a strongly worded
> suggestion as there are always charms that are exceptions to the rules. One
> such example is the aforementioned Fiche charm which as a bundle would not
> make as much sense, but is still a purposeful charm.
>
> That being said, most users coming to consume Juju are looking to solve a
> problem. Bundles underscore solutions to problems that people can consume,
> and get started quicker.
>
> As such, when new applications are charmed a test of "is this application
> something that serves a clear purpose" having a bundle submitted alongside
> the charm validates that claim and provides users a way to immediately get
> started with a solution.
>
> # Conclusion
>
> These policy changes, once accepted, will be targeted at all charms and
> bundles in Xenial as well as any new charm submitted after policy
> acceptance date for trusty, and finally any charm currently under review
> will be encouraged to adhere to the new policy but won't be required.
>
> # Action items
>
> I'm seeking feedback on this concept and welcome suggestions for
> improvements, questions, dissenting opinions, and any other remarks as well
> as votes from ~charmers and feedback from the community at large.
>
> Thanks,
> Marco Ceppi
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160316/9e6cc0d2/attachment.html>