Proposal: Charm testing for 2.0

Thu Mar 17 17:40:44 UTC 2016

The observation which might be too basic here is that for an Amulet test to
do something useful it needs to exercise the relations. This implies
(almost always) another charm. When your testing depends on more than one
charm (in what might be a synthetic situation) you are talking about
bundles. By depending on the tests living in a bundle we are saying this is
a known good, tested configuration intended for real world usage. Those are
important properties and not ones made by an Amulet test alone.

Even if it is the intention of an Amulet test to provide a guarantee of a
known good real world deployment that isn't surfaced to the consumer of the
charm. When consuming a bundle with tests the implications to the user are
both more transparent and practical.

-Ben

On Thu, Mar 17, 2016 at 9:12 AM Ryan Beisner <ryan.beisner at canonical.com>
wrote:

> On Thu, Mar 17, 2016 at 8:38 AM, Marco Ceppi <marco.ceppi at canonical.com>
> wrote:
>
>> On Thu, Mar 17, 2016 at 12:39 AM Ryan Beisner <ryan.beisner at canonical.com>
>> wrote:
>>
>>> Good evening,
>>>
>>> I really like the notion of a bundle possessing functional tests as an
>>> enhancement to test coverage.  I agree with almost all of those ideas.  :-)
>>>   tldr;  I would suggest that we consider bundle tests 'in addition to' and
>>> not 'as a replacement of' individual charm tests, because:
>>>
>>>
>>> *# Coverage and relevance*
>>> Any given charm may have many different modes of operation -- features
>>> which are enabled in some bundles but not in others.  A bundle test will
>>> likely only exercise that charm in the context of its configuration as it
>>> pertains to that bundle.  However, those who propose changes to the
>>> individual charm should know (via tests) if they've functionally broken the
>>> wider set of its knobs, bells and levers, which may be irrelevant to, or
>>> not testable in the bundle's amulet test due to its differing perspective.
>>> This opens potential functional test coverage gaps if we lean solely on the
>>> bundle for the test.
>>>
>>
>> In a world with layered charms, do we still need functional tests at the
>> charm level?
>>
>
> I believe the answer is not only yes, but that it becomes even more
> important with layered charms because of the multiple variables involved.
> Let's say you've rebuilt an existing layered charm that is composed of one
> of your own updated layers and 4 other published layers which may or may
> not have changed since the last charm build.  After rebuilding the charm,
> you can now re-run the charm Amulet test to verify that the rebuilt charm
> hasn't regressed in functionality.
>
> If there's no charm amulet test, you modify the bundle (because the
> bundle is not likely pointing to your newly-built charm), deploy the bundle
> and execute its tests.  This may only utilize a portion of the charm's
> capability, so the remaining capability of the re-built charm is at risk
> for breakage if that is the gate.
>
>
>
>>
>> I'd also like to clarify that this would not deprecate functional tests
>> in charms but makes it so teh policy reads EITHER unit tests or functional
>> tests, with a preference on a layered approach using unittests. Development
>> is quicker and since ever level of the charm is now segmented into
>> repositories which (should) have their own testing we don't need to
>> validate that X interface works in Y permutation as the interface is a
>> library with both halves of the communication channel and with tests.
>>
>>
>>> There are numerous cases where a charm can shift personalities and use
>>> cases, but not always on-the-fly in an already-deployed model.  In those
>>> cases, it may take a completely different and new deployment topology and
>>> configuration (bundle) to be able to exercise the relevant functional
>>> tests.  Without integrated amulet tests within the charm, one would have to
>>> publish multiple bundles, each containing separate amulet tests.  For
>>> low-dev-velocity charms, for simple charms, or for charms that aren't
>>> likely to be involved in complex workloads, this may be manageable.  But I
>>> don't think we should discourage or stop looking for individual charm
>>> amulet tests even there.
>>>
>>
>> We will always support charms with Amulet tests, ALWAYS, I think it'd
>> even be the hallmark of an exceptionally well written charm if it had the
>> means to do extensive functional testing.
>>
>
> *>>> # tl;dr:*
>
> *>>> We should stop writing Amulet tests in charms and instead only write
> them Bundles and force charms to do unit-testing (when possible) and
> promote that all charms be included in bundles in the store.*
>
> My main alarm here was the tl;dr.  I think we should not stop writing
> Amulet tests in charms.  Rather, encourage all fronts:  charm level Amulet
> tests, bundle Amulet tests, layer unit tests and charm unit tests.
>
>
>
>> I also have a separate email I'm authoring where we should be leveraging
>> health checks /inside/ a charm rather than externally poking it, so that
>> any deployment at anytime, in any mutation can be asked "hey, you ok?" and
>> get back a detailed report that assertions could be run against. I have a
>> follow up (and one of the reasons this email took me so long)
>>
>
> +1 to the "self-aware charm" as discussed (in Malta I think), as an added
> enhancement to quality and observability efforts.
>
>
>
>>
>>
>>> A charm's integrated amulet test can be both more focused and more
>>> expansive in what it exercises, as it can contain multiple deployment
>>> topologies and configurations (equivalent to cycling multiple unique
>>> bundles).  For example:  charm-xyz with and without SSL;  or in HA and
>>> without HA;  or IPv4 vs. IPv6; or IPv4 HA vs. IPv6 HA, multicast vs.
>>> unicast;  [IPv6 + HA + SSL] vs [IPv4 + HA + SSL]; or mysql deploying mysql
>>> proper vs. mysql deploying a variant;   and you can see the gist of the
>>> coverage explosion which translates to having a whole load of bundles to
>>> produce and maintain.
>>>
>>
>> Interesting, will consider this more.
>>
>>
>>> That is not to say that we don't have acceptance and integration tests
>>> for full OpenStack bundles.  We do that in the form of mojo specs which
>>> dynamically deploy any number of full OpenStack bundle topologies and
>>> configurations against multiple Ubuntu+OpenStack release combos, using
>>> either the dev or the stable set of OpenStack charms.  It basically takes
>>> what I've described above for amulet and allows us to pivot entire bundles
>>> into different models automatically.  There are currently *84* such
>>> OpenStack mojo specs with tests (bundle equivalents)
>>>
>>> Fear not, this is mostly accomplished with bundle inheritance, yaml foo,
>>> and shared test libraries.  We're not actually maintaining ~*516
>>> bundles*.  But if we were to achieve the current level of coverage with
>>> bundles, that's approximately how many there would need to be.  This
>>> includes the upcoming Xenial and Mitaka releases.  Reduce by ~12% when Juno
>>> EOLs.  Add 12% when we hit Newton B1, and so on.
>>>
>>
>> This is most compelling, but OpenStack is one of those scenarios that is
>> the exception to the rule. First and foremost, the openstack-charmers team
>> gates these charms, meaning more or less circumvent the charmers review
>> process because of the unique nature of the solution space. As a result
>> these polices wouldn't directly apply to your existing processes which are
>> both excellent already and help continue to produce high quality artifacts
>>
>
>>
>
>>
>>> *# How I'd like to use the proposed ideas*
>>> There are some OpenStack reference bundles in the charm store.  My
>>> suggested approach would be to continue to leverage individual charm amulet
>>> tests while adding functional tests to the existing charm store bundles.
>>> That would increase test coverage, and provide a mechanism to validate
>>> proposed changes to those specific bundles, such as to re-validate the
>>> bundles when charm versions are revved within them.
>>>
>>
>> I agree, and I think this falls well into the "Bundles should have amulet
>> tests" we're investigating tooling and processes for next cycle if this
>> policy passes so that we can do reviews of configuration/charm revisions in
>> bundles and validate them.
>>
>>
>>> To summarize, I am:
>>>
>>> -1 to stopping or discouraging individual charm amulet tests
>>>
>>> +1 for every charm containing amulet tests
>>>
>>
>> I disagree that this should be policy to have both an amulet and unittest
>> for the broader community and would instead like to say Testing is defined
>> as a unittest, or optionally an amulet test (or both)
>>
>
> Unit tests are super fast to run and can give an important level of
> confidence that methods and objects behave as intended, given the author's
> chosen sample or mocked data/objects.
>
> However, given their synthetic nature, unit tests aren't very indicative
> of how that charm will behave when it is deployed, has multiple peers, a
> load of relations and/or a wide range of differing configurations,
> substrates, network environments, latencies and such.  That's where an
> Amulet test really shines.  You can say:  using this charm's set of Amulet
> tests, deploy scenarios x, y, z, exercise each and make sure we've survived
> all of it.
>
> On that basis, I would encourage the idea that a charm should have both
> unit and Amulet tests.
>
>
>
>>
>>>
>>> +1 for every charm containing unit tests
>>>
>>> +1 for every charm having amulet coverage in at least 1 bundle
>>>
>>> +1 for every bundle possessing amulet tests
>>>
>>>
>>> Also open to feedback, discussion, suggestions, kicks in the shin.
>>>
>>> Thanks for all the great tooling and thought leadership.  We leverage
>>> the everloving *stuff* out of Amulet.
>>>
>>
>> We're starting an "Amulet for Juju 2.0" design conversation, feedback
>> from the OpenStack team on pieces missing and what we could move from
>> charmhelpers.contrib.amulet would be great:
>> https://github.com/juju/amulet/issues/116
>>
>
> Awesome.  Indeed, we should step through the various Amulet helpers and
> identify candidates for inclusion in Amulet proper.
>
>
>>
>>
>>
>>>
>>> Charm on!
>>>
>>> --
>>> Ryan Beisner
>>> QA Engineer, Ubuntu OpenStack Engineering, Canonical, Ltd.
>>> irc:beisner  gh/gerrit:ryan-beisner  lp:~1chb1n
>>>
>>>
>>>
>>> On Wed, Mar 16, 2016 at 7:52 PM, Marco Ceppi <marco.ceppi at canonical.com>
>>> wrote:
>>>
>>>> Hello everyone!
>>>>
>>>> This is an email I've been meaning to write for a while, and have
>>>> rewritten a few times now. With 2.0 on the horizon and the charm ecosystem
>>>> rapidly growing, I couldn't keep the idea to myself any longer.
>>>>
>>>> # tl;dr:
>>>>
>>>> We should stop writing Amulet tests in charms and instead only write
>>>> them Bundles and force charms to do unit-testing (when possible) and
>>>> promote that all charms be included in bundles in the store.
>>>>
>>>> # Problem
>>>>
>>>> Without making this a novel, charm-testing and amulet started before
>>>> bundles were even a construct in Juju with a spec written before Juju 1.0.
>>>> Since then, many new comers to the ecosystem have remarked how odd it is to
>>>> be writing deployment validations at the charm level. Indeed, as years have
>>>> gone by and new tools have sprung up it's become clear that; having an
>>>> author try to model all the permutations of a charms deployment and do the
>>>> physical deploys at that charm level are tedious and incomplete at best.
>>>>
>>>> With the explosion of layers and improvements to uniting test in charms
>>>> at that component level, I feel that continuing to create these bespoke
>>>> "bundles" via amulet in a single charm will not be a robust solution going
>>>> forward. As we sprint closer to Juju 2.0 we're seeing a higher demand for
>>>> assurance of working scenarios, and a sharp focus on quality at every
>>>> level. As such I'd like to propose the following policy changes:
>>>>
>>>> - All bundles must have tests before promulgation to the store
>>>> - All charms need to have comprehensive tests (unit or amulet)
>>>> - All charms should be included in a bundle
>>>>
>>>> I'll break down my reasoning and examples in the following sections:
>>>>
>>>> # All bundles must have tests before promulgation to the store
>>>>
>>>> Writing bundle tests with Amulet is actually a more compelling story
>>>> today than writing an Amulet test case for a charm. As an example, there's
>>>> a new ELK stack bundle being produced, here's what the test for that bundle
>>>> looks like:
>>>> https://github.com/juju-solutions/bundle-elk-stack/blob/master/tests/10-test-bundle
>>>>
>>>> This makes a lot of sense because it's asserting that the bundle is
>>>> working as expected by the Author who put the bundle together. It's also
>>>> loading the bundle.yaml as the deployment spec meaning as the bundle
>>>> evolves the tests will make sure they continue to run as expected. Also,
>>>> this could potentially be used in future smoke tests for charms being
>>>> updated if a CI process swaps out, say elasticsearch, for a newer version
>>>> of a charm being reviewed. We can assert that both the unittests in
>>>> elasticsearch work and it operates properly in an existing real world
>>>> solution a la the bundle.
>>>>
>>>> Additional examples:
>>>> -
>>>> https://github.com/juju-solutions/bundle-realtime-syslog-analytics/blob/master/tests/01-bundle.py
>>>> -
>>>> https://github.com/juju-solutions/bundle-apache-core-batch-processing/blob/master/tests/01-bundle.py
>>>>
>>>> # All charms need to have comprehensive tests (unit or amulet)
>>>>
>>>> This is just a clarification and more strongly typed policy change that
>>>> require charms have (preferred) unit tests or, if not applicable, then an
>>>> Amulet test. Bash doesn't really allow for unittesting, so in those
>>>> scenarios, Amulet tests would function as a valid testing case.
>>>>
>>>> There are also some charms which will not make sense as a bundle. One
>>>> example is the recently promulgated Fiche charm:
>>>> http://bazaar.launchpad.net/~charmers/charms/trusty/fiche/trunk/view/head:/tests/10-deploy It's
>>>> a standalone pastebin, but it's an awesome service that provides deployment
>>>> validation with an Amulet test. The test stands up the charm, exercises
>>>> configuration, and validates the service responds in an expected way. For
>>>> scenarios where a charm does not have a bundle an Amulet test would be
>>>> required.
>>>>
>>>> Any charm that currently includes an Amulet test is welcome to continue
>>>> keeping such a test.
>>>>
>>>> # All charms should be included in a bundle
>>>>
>>>> This last one is to underscore that charms need to serve a purpose.
>>>> This policy is written as not an absolute, but instead a strongly worded
>>>> suggestion as there are always charms that are exceptions to the rules. One
>>>> such example is the aforementioned Fiche charm which as a bundle would not
>>>> make as much sense, but is still a purposeful charm.
>>>>
>>>> That being said, most users coming to consume Juju are looking to solve
>>>> a problem. Bundles underscore solutions to problems that people can
>>>> consume, and get started quicker.
>>>>
>>>> As such, when new applications are charmed a test of "is this
>>>> application something that serves a clear purpose" having a bundle
>>>> submitted alongside the charm validates that claim and provides users a way
>>>> to immediately get started with a solution.
>>>>
>>>> # Conclusion
>>>>
>>>> These policy changes, once accepted, will be targeted at all charms and
>>>> bundles in Xenial as well as any new charm submitted after policy
>>>> acceptance date for trusty, and finally any charm currently under review
>>>> will be encouraged to adhere to the new policy but won't be required.
>>>>
>>>> # Action items
>>>>
>>>> I'm seeking feedback on this concept and welcome suggestions for
>>>> improvements, questions, dissenting opinions, and any other remarks as well
>>>> as votes from ~charmers and feedback from the community at large.
>>>>
>>>> Thanks,
>>>> Marco Ceppi
>>>>
>>>> --
>>>> Juju mailing list
>>>> Juju at lists.ubuntu.com
>>>> Modify settings or unsubscribe at:
>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>
>>>> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160317/3e0f322a/attachment.html>