Proposal: Charm testing for 2.0

Thu Mar 17 13:38:23 UTC 2016

On Thu, Mar 17, 2016 at 12:39 AM Ryan Beisner <ryan.beisner at canonical.com>
wrote:

> Good evening,
>
> I really like the notion of a bundle possessing functional tests as an
> enhancement to test coverage.  I agree with almost all of those ideas.  :-)
>   tldr;  I would suggest that we consider bundle tests 'in addition to' and
> not 'as a replacement of' individual charm tests, because:
>
>
> *# Coverage and relevance*
> Any given charm may have many different modes of operation -- features
> which are enabled in some bundles but not in others.  A bundle test will
> likely only exercise that charm in the context of its configuration as it
> pertains to that bundle.  However, those who propose changes to the
> individual charm should know (via tests) if they've functionally broken the
> wider set of its knobs, bells and levers, which may be irrelevant to, or
> not testable in the bundle's amulet test due to its differing perspective.
> This opens potential functional test coverage gaps if we lean solely on the
> bundle for the test.
>

In a world with layered charms, do we still need functional tests at the
charm level?

I'd also like to clarify that this would not deprecate functional tests in
charms but makes it so teh policy reads EITHER unit tests or functional
tests, with a preference on a layered approach using unittests. Development
is quicker and since ever level of the charm is now segmented into
repositories which (should) have their own testing we don't need to
validate that X interface works in Y permutation as the interface is a
library with both halves of the communication channel and with tests.

> There are numerous cases where a charm can shift personalities and use
> cases, but not always on-the-fly in an already-deployed model.  In those
> cases, it may take a completely different and new deployment topology and
> configuration (bundle) to be able to exercise the relevant functional
> tests.  Without integrated amulet tests within the charm, one would have to
> publish multiple bundles, each containing separate amulet tests.  For
> low-dev-velocity charms, for simple charms, or for charms that aren't
> likely to be involved in complex workloads, this may be manageable.  But I
> don't think we should discourage or stop looking for individual charm
> amulet tests even there.
>

We will always support charms with Amulet tests, ALWAYS, I think it'd even
be the hallmark of an exceptionally well written charm if it had the means
to do extensive functional testing.

I also have a separate email I'm authoring where we should be leveraging
health checks /inside/ a charm rather than externally poking it, so that
any deployment at anytime, in any mutation can be asked "hey, you ok?" and
get back a detailed report that assertions could be run against. I have a
follow up (and one of the reasons this email took me so long)

> A charm's integrated amulet test can be both more focused and more
> expansive in what it exercises, as it can contain multiple deployment
> topologies and configurations (equivalent to cycling multiple unique
> bundles).  For example:  charm-xyz with and without SSL;  or in HA and
> without HA;  or IPv4 vs. IPv6; or IPv4 HA vs. IPv6 HA, multicast vs.
> unicast;  [IPv6 + HA + SSL] vs [IPv4 + HA + SSL]; or mysql deploying mysql
> proper vs. mysql deploying a variant;   and you can see the gist of the
> coverage explosion which translates to having a whole load of bundles to
> produce and maintain.
>

Interesting, will consider this more.

> That is not to say that we don't have acceptance and integration tests for
> full OpenStack bundles.  We do that in the form of mojo specs which
> dynamically deploy any number of full OpenStack bundle topologies and
> configurations against multiple Ubuntu+OpenStack release combos, using
> either the dev or the stable set of OpenStack charms.  It basically takes
> what I've described above for amulet and allows us to pivot entire bundles
> into different models automatically.  There are currently *84* such
> OpenStack mojo specs with tests (bundle equivalents)
>
> Fear not, this is mostly accomplished with bundle inheritance, yaml foo,
> and shared test libraries.  We're not actually maintaining ~*516 bundles*.
> But if we were to achieve the current level of coverage with bundles,
> that's approximately how many there would need to be.  This includes the
> upcoming Xenial and Mitaka releases.  Reduce by ~12% when Juno EOLs.  Add
> 12% when we hit Newton B1, and so on.
>

This is most compelling, but OpenStack is one of those scenarios that is
the exception to the rule. First and foremost, the openstack-charmers team
gates these charms, meaning more or less circumvent the charmers review
process because of the unique nature of the solution space. As a result
these polices wouldn't directly apply to your existing processes which are
both excellent already and help continue to produce high quality artifacts

> *# How I'd like to use the proposed ideas*
> There are some OpenStack reference bundles in the charm store.  My
> suggested approach would be to continue to leverage individual charm amulet
> tests while adding functional tests to the existing charm store bundles.
> That would increase test coverage, and provide a mechanism to validate
> proposed changes to those specific bundles, such as to re-validate the
> bundles when charm versions are revved within them.
>

I agree, and I think this falls well into the "Bundles should have amulet
tests" we're investigating tooling and processes for next cycle if this
policy passes so that we can do reviews of configuration/charm revisions in
bundles and validate them.

> To summarize, I am:
>
> -1 to stopping or discouraging individual charm amulet tests
>
> +1 for every charm containing amulet tests
>

I disagree that this should be policy to have both an amulet and unittest
for the broader community and would instead like to say Testing is defined
as a unittest, or optionally an amulet test (or both)

>
> +1 for every charm containing unit tests
>
> +1 for every charm having amulet coverage in at least 1 bundle
>
> +1 for every bundle possessing amulet tests
>
>
> Also open to feedback, discussion, suggestions, kicks in the shin.
>
> Thanks for all the great tooling and thought leadership.  We leverage the
> everloving *stuff* out of Amulet.
>

We're starting an "Amulet for Juju 2.0" design conversation, feedback from
the OpenStack team on pieces missing and what we could move from
charmhelpers.contrib.amulet would be great:
https://github.com/juju/amulet/issues/116

>
> Charm on!
>
> --
> Ryan Beisner
> QA Engineer, Ubuntu OpenStack Engineering, Canonical, Ltd.
> irc:beisner  gh/gerrit:ryan-beisner  lp:~1chb1n
>
>
>
> On Wed, Mar 16, 2016 at 7:52 PM, Marco Ceppi <marco.ceppi at canonical.com>
> wrote:
>
>> Hello everyone!
>>
>> This is an email I've been meaning to write for a while, and have
>> rewritten a few times now. With 2.0 on the horizon and the charm ecosystem
>> rapidly growing, I couldn't keep the idea to myself any longer.
>>
>> # tl;dr:
>>
>> We should stop writing Amulet tests in charms and instead only write them
>> Bundles and force charms to do unit-testing (when possible) and promote
>> that all charms be included in bundles in the store.
>>
>> # Problem
>>
>> Without making this a novel, charm-testing and amulet started before
>> bundles were even a construct in Juju with a spec written before Juju 1.0.
>> Since then, many new comers to the ecosystem have remarked how odd it is to
>> be writing deployment validations at the charm level. Indeed, as years have
>> gone by and new tools have sprung up it's become clear that; having an
>> author try to model all the permutations of a charms deployment and do the
>> physical deploys at that charm level are tedious and incomplete at best.
>>
>> With the explosion of layers and improvements to uniting test in charms
>> at that component level, I feel that continuing to create these bespoke
>> "bundles" via amulet in a single charm will not be a robust solution going
>> forward. As we sprint closer to Juju 2.0 we're seeing a higher demand for
>> assurance of working scenarios, and a sharp focus on quality at every
>> level. As such I'd like to propose the following policy changes:
>>
>> - All bundles must have tests before promulgation to the store
>> - All charms need to have comprehensive tests (unit or amulet)
>> - All charms should be included in a bundle
>>
>> I'll break down my reasoning and examples in the following sections:
>>
>> # All bundles must have tests before promulgation to the store
>>
>> Writing bundle tests with Amulet is actually a more compelling story
>> today than writing an Amulet test case for a charm. As an example, there's
>> a new ELK stack bundle being produced, here's what the test for that bundle
>> looks like:
>> https://github.com/juju-solutions/bundle-elk-stack/blob/master/tests/10-test-bundle
>>
>> This makes a lot of sense because it's asserting that the bundle is
>> working as expected by the Author who put the bundle together. It's also
>> loading the bundle.yaml as the deployment spec meaning as the bundle
>> evolves the tests will make sure they continue to run as expected. Also,
>> this could potentially be used in future smoke tests for charms being
>> updated if a CI process swaps out, say elasticsearch, for a newer version
>> of a charm being reviewed. We can assert that both the unittests in
>> elasticsearch work and it operates properly in an existing real world
>> solution a la the bundle.
>>
>> Additional examples:
>> -
>> https://github.com/juju-solutions/bundle-realtime-syslog-analytics/blob/master/tests/01-bundle.py
>> -
>> https://github.com/juju-solutions/bundle-apache-core-batch-processing/blob/master/tests/01-bundle.py
>>
>> # All charms need to have comprehensive tests (unit or amulet)
>>
>> This is just a clarification and more strongly typed policy change that
>> require charms have (preferred) unit tests or, if not applicable, then an
>> Amulet test. Bash doesn't really allow for unittesting, so in those
>> scenarios, Amulet tests would function as a valid testing case.
>>
>> There are also some charms which will not make sense as a bundle. One
>> example is the recently promulgated Fiche charm:
>> http://bazaar.launchpad.net/~charmers/charms/trusty/fiche/trunk/view/head:/tests/10-deploy It's
>> a standalone pastebin, but it's an awesome service that provides deployment
>> validation with an Amulet test. The test stands up the charm, exercises
>> configuration, and validates the service responds in an expected way. For
>> scenarios where a charm does not have a bundle an Amulet test would be
>> required.
>>
>> Any charm that currently includes an Amulet test is welcome to continue
>> keeping such a test.
>>
>> # All charms should be included in a bundle
>>
>> This last one is to underscore that charms need to serve a purpose. This
>> policy is written as not an absolute, but instead a strongly worded
>> suggestion as there are always charms that are exceptions to the rules. One
>> such example is the aforementioned Fiche charm which as a bundle would not
>> make as much sense, but is still a purposeful charm.
>>
>> That being said, most users coming to consume Juju are looking to solve a
>> problem. Bundles underscore solutions to problems that people can consume,
>> and get started quicker.
>>
>> As such, when new applications are charmed a test of "is this application
>> something that serves a clear purpose" having a bundle submitted alongside
>> the charm validates that claim and provides users a way to immediately get
>> started with a solution.
>>
>> # Conclusion
>>
>> These policy changes, once accepted, will be targeted at all charms and
>> bundles in Xenial as well as any new charm submitted after policy
>> acceptance date for trusty, and finally any charm currently under review
>> will be encouraged to adhere to the new policy but won't be required.
>>
>> # Action items
>>
>> I'm seeking feedback on this concept and welcome suggestions for
>> improvements, questions, dissenting opinions, and any other remarks as well
>> as votes from ~charmers and feedback from the community at large.
>>
>> Thanks,
>> Marco Ceppi
>>
>> --
>> Juju mailing list
>> Juju at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160317/f69cd686/attachment.html>