Proposal: Charm testing for 2.0

[Marco Ceppi]

Hello everyone!

This is an email I've been meaning to write for a while, and have rewritten
a few times now. With 2.0 on the horizon and the charm ecosystem rapidly
growing, I couldn't keep the idea to myself any longer.

# tl;dr:

We should stop writing Amulet tests in charms and instead only write them
Bundles and force charms to do unit-testing (when possible) and promote
that all charms be included in bundles in the store.

# Problem

Without making this a novel, charm-testing and amulet started before
bundles were even a construct in Juju with a spec written before Juju 1.0.
Since then, many new comers to the ecosystem have remarked how odd it is to
be writing deployment validations at the charm level. Indeed, as years have
gone by and new tools have sprung up it's become clear that; having an
author try to model all the permutations of a charms deployment and do the
physical deploys at that charm level are tedious and incomplete at best.

With the explosion of layers and improvements to uniting test in charms at
that component level, I feel that continuing to create these bespoke
"bundles" via amulet in a single charm will not be a robust solution going
forward. As we sprint closer to Juju 2.0 we're seeing a higher demand for
assurance of working scenarios, and a sharp focus on quality at every
level. As such I'd like to propose the following policy changes:

- All bundles must have tests before promulgation to the store
- All charms need to have comprehensive tests (unit or amulet)
- All charms should be included in a bundle

I'll break down my reasoning and examples in the following sections:

# All bundles must have tests before promulgation to the store

Writing bundle tests with Amulet is actually a more compelling story today
than writing an Amulet test case for a charm. As an example, there's a new
ELK stack bundle being produced, here's what the test for that bundle looks
like:
https://github.com/juju-solutions/bundle-elk-stack/blob/master/tests/10-test-bundle

This makes a lot of sense because it's asserting that the bundle is working
as expected by the Author who put the bundle together. It's also loading
the bundle.yaml as the deployment spec meaning as the bundle evolves the
tests will make sure they continue to run as expected. Also, this could
potentially be used in future smoke tests for charms being updated if a CI
process swaps out, say elasticsearch, for a newer version of a charm being
reviewed. We can assert that both the unittests in elasticsearch work and
it operates properly in an existing real world solution a la the bundle.

Additional examples:
-
https://github.com/juju-solutions/bundle-realtime-syslog-analytics/blob/master/tests/01-bundle.py
-
https://github.com/juju-solutions/bundle-apache-core-batch-processing/blob/master/tests/01-bundle.py

# All charms need to have comprehensive tests (unit or amulet)

This is just a clarification and more strongly typed policy change that
require charms have (preferred) unit tests or, if not applicable, then an
Amulet test. Bash doesn't really allow for unittesting, so in those
scenarios, Amulet tests would function as a valid testing case.

There are also some charms which will not make sense as a bundle. One
example is the recently promulgated Fiche charm:
http://bazaar.launchpad.net/~charmers/charms/trusty/fiche/trunk/view/head:/tests/10-deploy
It's
a standalone pastebin, but it's an awesome service that provides deployment
validation with an Amulet test. The test stands up the charm, exercises
configuration, and validates the service responds in an expected way. For
scenarios where a charm does not have a bundle an Amulet test would be
required.

Any charm that currently includes an Amulet test is welcome to continue
keeping such a test.

# All charms should be included in a bundle

This last one is to underscore that charms need to serve a purpose. This
policy is written as not an absolute, but instead a strongly worded
suggestion as there are always charms that are exceptions to the rules. One
such example is the aforementioned Fiche charm which as a bundle would not
make as much sense, but is still a purposeful charm.

That being said, most users coming to consume Juju are looking to solve a
problem. Bundles underscore solutions to problems that people can consume,
and get started quicker.

As such, when new applications are charmed a test of "is this application
something that serves a clear purpose" having a bundle submitted alongside
the charm validates that claim and provides users a way to immediately get
started with a solution.

# Conclusion

These policy changes, once accepted, will be targeted at all charms and
bundles in Xenial as well as any new charm submitted after policy
acceptance date for trusty, and finally any charm currently under review
will be encouraged to adhere to the new policy but won't be required.

# Action items

I'm seeking feedback on this concept and welcome suggestions for
improvements, questions, dissenting opinions, and any other remarks as well
as votes from ~charmers and feedback from the community at large.

Thanks,
Marco Ceppi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160317/c1a2ba72/attachment.html>