Proposal: Charm testing for 2.0

Thu Mar 17 13:51:25 UTC 2016

On Thu, Mar 17, 2016 at 5:39 AM Tom Barber <tom at analytical-labs.com> wrote:

> I tend to agree with Ryan, I think the ideas are reasonably sound,
> although I'm not sure about the "every charm should be part of a bundle"
> policy, but I certainly don't think you should discourage testing at charm
> level the encapsulation can be useful, and you can never have too many
> tests!
>

Thanks for the feedback, I appreciate it. I want to clarify since I did a
terrible job explaining that I don't wish to stop Amulet tests in charm but
instead really focus on driving unittests in charms - and less charms but
more Layers. This will allow for quicker iterations and faster dev/test
cycles.

> I'm
> +0.5 for charms part of a bundle expectation
> -1 for discouraging charm tests
>

Want to caveat again, charm tests would be one or the two (or both)
[unittest, amulet]

> My 2 cents.
>

Thanks!

>
> Tom
>
> --------------
>
> Director Meteorite.bi - Saiku Analytics Founder
> Tel: +44(0)5603641316
>
> (Thanks to the Saiku community we reached our Kickstart
> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
> goal, but you can always help by sponsoring the project
> <http://www.meteorite.bi/products/saiku/sponsorship>)
>
> On 17 March 2016 at 04:38, Ryan Beisner <ryan.beisner at canonical.com>
> wrote:
>
>> Good evening,
>>
>> I really like the notion of a bundle possessing functional tests as an
>> enhancement to test coverage.  I agree with almost all of those ideas.  :-)
>>   tldr;  I would suggest that we consider bundle tests 'in addition to' and
>> not 'as a replacement of' individual charm tests, because:
>>
>>
>> *# Coverage and relevance*
>> Any given charm may have many different modes of operation -- features
>> which are enabled in some bundles but not in others.  A bundle test will
>> likely only exercise that charm in the context of its configuration as it
>> pertains to that bundle.  However, those who propose changes to the
>> individual charm should know (via tests) if they've functionally broken the
>> wider set of its knobs, bells and levers, which may be irrelevant to, or
>> not testable in the bundle's amulet test due to its differing perspective.
>> This opens potential functional test coverage gaps if we lean solely on the
>> bundle for the test.
>>
>> There are numerous cases where a charm can shift personalities and use
>> cases, but not always on-the-fly in an already-deployed model.  In those
>> cases, it may take a completely different and new deployment topology and
>> configuration (bundle) to be able to exercise the relevant functional
>> tests.  Without integrated amulet tests within the charm, one would have to
>> publish multiple bundles, each containing separate amulet tests.  For
>> low-dev-velocity charms, for simple charms, or for charms that aren't
>> likely to be involved in complex workloads, this may be manageable.  But I
>> don't think we should discourage or stop looking for individual charm
>> amulet tests even there.
>>
>> A charm's integrated amulet test can be both more focused and more
>> expansive in what it exercises, as it can contain multiple deployment
>> topologies and configurations (equivalent to cycling multiple unique
>> bundles).  For example:  charm-xyz with and without SSL;  or in HA and
>> without HA;  or IPv4 vs. IPv6; or IPv4 HA vs. IPv6 HA, multicast vs.
>> unicast;  [IPv6 + HA + SSL] vs [IPv4 + HA + SSL]; or mysql deploying mysql
>> proper vs. mysql deploying a variant;   and you can see the gist of the
>> coverage explosion which translates to having a whole load of bundles to
>> produce and maintain.
>>
>>
>> *# Dev and test: cost, scale and velocity*
>> Individual charm amulet tests are an important piece in testing large or
>> complex models.  I'll share some bits of what we do for OpenStack charms as
>> an example.  No bias.  :-)
>>
>> Each of the OpenStack charms contain amulet test definitions.  We lean
>> heavily on those tests to deploy fractions of a full OpenStack bundle as
>> the core of our CI development gate.  With [27 charms] x [stable + dev] x
>> [8 Ubuntu/OpenStack Release Combos], there are currently* ~432 *possible
>> variations of amulet tests (derived bundles of fractional OpenStacks).  A
>> subset of those are executed in gate, depending on relevance to the
>> developer's proposed change.  This allows us to endure a high velocity of
>> focused testing on development in these very active charms.  Because the
>> derived models are much smaller than the reference bundle, we can give
>> developers rapid and automated feedback, plus they can iterate on
>> development outside of our CI without having to be able to deploy a full
>> OpenStack.
>>
>> That is not to say that we don't have acceptance and integration tests
>> for full OpenStack bundles.  We do that in the form of mojo specs which
>> dynamically deploy any number of full OpenStack bundle topologies and
>> configurations against multiple Ubuntu+OpenStack release combos, using
>> either the dev or the stable set of OpenStack charms.  It basically takes
>> what I've described above for amulet and allows us to pivot entire bundles
>> into different models automatically.  There are currently *84* such
>> OpenStack mojo specs with tests (bundle equivalents)
>>
>> Fear not, this is mostly accomplished with bundle inheritance, yaml foo,
>> and shared test libraries.  We're not actually maintaining ~*516 bundles*.
>> But if we were to achieve the current level of coverage with bundles,
>> that's approximately how many there would need to be.  This includes the
>> upcoming Xenial and Mitaka releases.  Reduce by ~12% when Juno EOLs.  Add
>> 12% when we hit Newton B1, and so on.
>>
>>
>> *# How I'd like to use the proposed ideas*
>> There are some OpenStack reference bundles in the charm store.  My
>> suggested approach would be to continue to leverage individual charm amulet
>> tests while adding functional tests to the existing charm store bundles.
>> That would increase test coverage, and provide a mechanism to validate
>> proposed changes to those specific bundles, such as to re-validate the
>> bundles when charm versions are revved within them.
>>
>>
>> To summarize, I am:
>>
>> -1 to stopping or discouraging individual charm amulet tests
>>
>> +1 for every charm containing amulet tests
>>
>> +1 for every charm containing unit tests
>>
>> +1 for every charm having amulet coverage in at least 1 bundle
>>
>> +1 for every bundle possessing amulet tests
>>
>>
>> Also open to feedback, discussion, suggestions, kicks in the shin.
>>
>> Thanks for all the great tooling and thought leadership.  We leverage the
>> everloving *stuff* out of Amulet.
>>
>> Charm on!
>>
>> --
>> Ryan Beisner
>> QA Engineer, Ubuntu OpenStack Engineering, Canonical, Ltd.
>> irc:beisner  gh/gerrit:ryan-beisner  lp:~1chb1n
>>
>>
>>
>> On Wed, Mar 16, 2016 at 7:52 PM, Marco Ceppi <marco.ceppi at canonical.com>
>> wrote:
>>
>>> Hello everyone!
>>>
>>> This is an email I've been meaning to write for a while, and have
>>> rewritten a few times now. With 2.0 on the horizon and the charm ecosystem
>>> rapidly growing, I couldn't keep the idea to myself any longer.
>>>
>>> # tl;dr:
>>>
>>> We should stop writing Amulet tests in charms and instead only write
>>> them Bundles and force charms to do unit-testing (when possible) and
>>> promote that all charms be included in bundles in the store.
>>>
>>> # Problem
>>>
>>> Without making this a novel, charm-testing and amulet started before
>>> bundles were even a construct in Juju with a spec written before Juju 1.0.
>>> Since then, many new comers to the ecosystem have remarked how odd it is to
>>> be writing deployment validations at the charm level. Indeed, as years have
>>> gone by and new tools have sprung up it's become clear that; having an
>>> author try to model all the permutations of a charms deployment and do the
>>> physical deploys at that charm level are tedious and incomplete at best.
>>>
>>> With the explosion of layers and improvements to uniting test in charms
>>> at that component level, I feel that continuing to create these bespoke
>>> "bundles" via amulet in a single charm will not be a robust solution going
>>> forward. As we sprint closer to Juju 2.0 we're seeing a higher demand for
>>> assurance of working scenarios, and a sharp focus on quality at every
>>> level. As such I'd like to propose the following policy changes:
>>>
>>> - All bundles must have tests before promulgation to the store
>>> - All charms need to have comprehensive tests (unit or amulet)
>>> - All charms should be included in a bundle
>>>
>>> I'll break down my reasoning and examples in the following sections:
>>>
>>> # All bundles must have tests before promulgation to the store
>>>
>>> Writing bundle tests with Amulet is actually a more compelling story
>>> today than writing an Amulet test case for a charm. As an example, there's
>>> a new ELK stack bundle being produced, here's what the test for that bundle
>>> looks like:
>>> https://github.com/juju-solutions/bundle-elk-stack/blob/master/tests/10-test-bundle
>>>
>>> This makes a lot of sense because it's asserting that the bundle is
>>> working as expected by the Author who put the bundle together. It's also
>>> loading the bundle.yaml as the deployment spec meaning as the bundle
>>> evolves the tests will make sure they continue to run as expected. Also,
>>> this could potentially be used in future smoke tests for charms being
>>> updated if a CI process swaps out, say elasticsearch, for a newer version
>>> of a charm being reviewed. We can assert that both the unittests in
>>> elasticsearch work and it operates properly in an existing real world
>>> solution a la the bundle.
>>>
>>> Additional examples:
>>> -
>>> https://github.com/juju-solutions/bundle-realtime-syslog-analytics/blob/master/tests/01-bundle.py
>>> -
>>> https://github.com/juju-solutions/bundle-apache-core-batch-processing/blob/master/tests/01-bundle.py
>>>
>>> # All charms need to have comprehensive tests (unit or amulet)
>>>
>>> This is just a clarification and more strongly typed policy change that
>>> require charms have (preferred) unit tests or, if not applicable, then an
>>> Amulet test. Bash doesn't really allow for unittesting, so in those
>>> scenarios, Amulet tests would function as a valid testing case.
>>>
>>> There are also some charms which will not make sense as a bundle. One
>>> example is the recently promulgated Fiche charm:
>>> http://bazaar.launchpad.net/~charmers/charms/trusty/fiche/trunk/view/head:/tests/10-deploy It's
>>> a standalone pastebin, but it's an awesome service that provides deployment
>>> validation with an Amulet test. The test stands up the charm, exercises
>>> configuration, and validates the service responds in an expected way. For
>>> scenarios where a charm does not have a bundle an Amulet test would be
>>> required.
>>>
>>> Any charm that currently includes an Amulet test is welcome to continue
>>> keeping such a test.
>>>
>>> # All charms should be included in a bundle
>>>
>>> This last one is to underscore that charms need to serve a purpose. This
>>> policy is written as not an absolute, but instead a strongly worded
>>> suggestion as there are always charms that are exceptions to the rules. One
>>> such example is the aforementioned Fiche charm which as a bundle would not
>>> make as much sense, but is still a purposeful charm.
>>>
>>> That being said, most users coming to consume Juju are looking to solve
>>> a problem. Bundles underscore solutions to problems that people can
>>> consume, and get started quicker.
>>>
>>> As such, when new applications are charmed a test of "is this
>>> application something that serves a clear purpose" having a bundle
>>> submitted alongside the charm validates that claim and provides users a way
>>> to immediately get started with a solution.
>>>
>>> # Conclusion
>>>
>>> These policy changes, once accepted, will be targeted at all charms and
>>> bundles in Xenial as well as any new charm submitted after policy
>>> acceptance date for trusty, and finally any charm currently under review
>>> will be encouraged to adhere to the new policy but won't be required.
>>>
>>> # Action items
>>>
>>> I'm seeking feedback on this concept and welcome suggestions for
>>> improvements, questions, dissenting opinions, and any other remarks as well
>>> as votes from ~charmers and feedback from the community at large.
>>>
>>> Thanks,
>>> Marco Ceppi
>>>
>>> --
>>> Juju mailing list
>>> Juju at lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>
>>>
>>
>> --
>> Juju mailing list
>> Juju at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160317/c57c18e1/attachment.html>