Proposal: Charm testing for 2.0

[Merlijn Sebrechts]

As an aside; is there a good write-up somewhere about charm unit testing.
I'd like to do this but I'm not sure how to do this. I am completely new to
unit testing so I'm having a hard time to see how a good unittest for a
Charm would look like and what exactly should be tested.

2016-03-17 1:52 GMT+01:00 Marco Ceppi <marco.ceppi at canonical.com>:

> Hello everyone!
>
> This is an email I've been meaning to write for a while, and have
> rewritten a few times now. With 2.0 on the horizon and the charm ecosystem
> rapidly growing, I couldn't keep the idea to myself any longer.
>
> # tl;dr:
>
> We should stop writing Amulet tests in charms and instead only write them
> Bundles and force charms to do unit-testing (when possible) and promote
> that all charms be included in bundles in the store.
>
> # Problem
>
> Without making this a novel, charm-testing and amulet started before
> bundles were even a construct in Juju with a spec written before Juju 1.0.
> Since then, many new comers to the ecosystem have remarked how odd it is to
> be writing deployment validations at the charm level. Indeed, as years have
> gone by and new tools have sprung up it's become clear that; having an
> author try to model all the permutations of a charms deployment and do the
> physical deploys at that charm level are tedious and incomplete at best.
>
> With the explosion of layers and improvements to uniting test in charms at
> that component level, I feel that continuing to create these bespoke
> "bundles" via amulet in a single charm will not be a robust solution going
> forward. As we sprint closer to Juju 2.0 we're seeing a higher demand for
> assurance of working scenarios, and a sharp focus on quality at every
> level. As such I'd like to propose the following policy changes:
>
> - All bundles must have tests before promulgation to the store
> - All charms need to have comprehensive tests (unit or amulet)
> - All charms should be included in a bundle
>
> I'll break down my reasoning and examples in the following sections:
>
> # All bundles must have tests before promulgation to the store
>
> Writing bundle tests with Amulet is actually a more compelling story today
> than writing an Amulet test case for a charm. As an example, there's a new
> ELK stack bundle being produced, here's what the test for that bundle looks
> like:
> https://github.com/juju-solutions/bundle-elk-stack/blob/master/tests/10-test-bundle
>
> This makes a lot of sense because it's asserting that the bundle is
> working as expected by the Author who put the bundle together. It's also
> loading the bundle.yaml as the deployment spec meaning as the bundle
> evolves the tests will make sure they continue to run as expected. Also,
> this could potentially be used in future smoke tests for charms being
> updated if a CI process swaps out, say elasticsearch, for a newer version
> of a charm being reviewed. We can assert that both the unittests in
> elasticsearch work and it operates properly in an existing real world
> solution a la the bundle.
>
> Additional examples:
> -
> https://github.com/juju-solutions/bundle-realtime-syslog-analytics/blob/master/tests/01-bundle.py
> -
> https://github.com/juju-solutions/bundle-apache-core-batch-processing/blob/master/tests/01-bundle.py
>
> # All charms need to have comprehensive tests (unit or amulet)
>
> This is just a clarification and more strongly typed policy change that
> require charms have (preferred) unit tests or, if not applicable, then an
> Amulet test. Bash doesn't really allow for unittesting, so in those
> scenarios, Amulet tests would function as a valid testing case.
>
> There are also some charms which will not make sense as a bundle. One
> example is the recently promulgated Fiche charm:
> http://bazaar.launchpad.net/~charmers/charms/trusty/fiche/trunk/view/head:/tests/10-deploy It's
> a standalone pastebin, but it's an awesome service that provides deployment
> validation with an Amulet test. The test stands up the charm, exercises
> configuration, and validates the service responds in an expected way. For
> scenarios where a charm does not have a bundle an Amulet test would be
> required.
>
> Any charm that currently includes an Amulet test is welcome to continue
> keeping such a test.
>
> # All charms should be included in a bundle
>
> This last one is to underscore that charms need to serve a purpose. This
> policy is written as not an absolute, but instead a strongly worded
> suggestion as there are always charms that are exceptions to the rules. One
> such example is the aforementioned Fiche charm which as a bundle would not
> make as much sense, but is still a purposeful charm.
>
> That being said, most users coming to consume Juju are looking to solve a
> problem. Bundles underscore solutions to problems that people can consume,
> and get started quicker.
>
> As such, when new applications are charmed a test of "is this application
> something that serves a clear purpose" having a bundle submitted alongside
> the charm validates that claim and provides users a way to immediately get
> started with a solution.
>
> # Conclusion
>
> These policy changes, once accepted, will be targeted at all charms and
> bundles in Xenial as well as any new charm submitted after policy
> acceptance date for trusty, and finally any charm currently under review
> will be encouraged to adhere to the new policy but won't be required.
>
> # Action items
>
> I'm seeking feedback on this concept and welcome suggestions for
> improvements, questions, dissenting opinions, and any other remarks as well
> as votes from ~charmers and feedback from the community at large.
>
> Thanks,
> Marco Ceppi
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160317/3fec1022/attachment.html>