<div dir="ltr"><div><div>Good evening,</div><div><br></div><div>I really like the notion of a bundle possessing functional tests as an enhancement to test coverage. I agree with almost all of those ideas. :-) tldr; I would suggest that we consider bundle tests 'in addition to' and not 'as a replacement of' individual charm tests, because:</div><div><br></div><div><br></div><div><b># Coverage and relevance</b></div></div><div><div>Any given charm may have many different modes of operation -- features which are enabled in some bundles but not in others. A bundle test will likely only exercise that charm in the context of its configuration as it pertains to that bundle. However, those who propose changes to the individual charm should know (via tests) if they've functionally broken the wider set of its knobs, bells and levers, which may be irrelevant to, or not testable in the bundle's amulet test due to its differing perspective. This opens potential functional test coverage gaps if we lean solely on the bundle for the test.</div></div><br>There are numerous cases where a charm can shift personalities and use cases, but not always on-the-fly in an already-deployed model. In those cases, it may take a completely different and new deployment topology and configuration (bundle) to be able to exercise the relevant functional tests. Without integrated amulet tests within the charm, one would have to publish multiple bundles, each containing separate amulet tests. For low-dev-velocity charms, for simple charms, or for charms that aren't likely to be involved in complex workloads, this may be manageable. But I don't think we should discourage or stop looking for individual charm amulet tests even there.<br><br>A charm's integrated amulet test can be both more focused and more expansive in what it exercises, as it can contain multiple deployment topologies and configurations (equivalent to cycling multiple unique bundles). For example: charm-xyz with and without SSL; or in HA and without HA; or IPv4 vs. IPv6; or IPv4 HA vs. IPv6 HA, multicast vs. unicast; [IPv6 + HA + SSL] vs [IPv4 + HA + SSL]; or mysql deploying mysql proper vs. mysql deploying a variant; and you can see the gist of the coverage explosion which translates to having a whole load of bundles to produce and maintain.<br><br><div><div></div></div><div><div><br></div></div><div><div><b># Dev and test: cost, scale and velocity</b></div></div>Individual charm amulet tests are an important piece in testing large or complex models. I'll share some bits of what we do for OpenStack charms as an example. No bias. :-)<br><br>Each of the OpenStack charms contain amulet test definitions. We lean heavily on those tests to deploy fractions of a full OpenStack bundle as the core of our CI development gate. With [27 charms] x [stable + dev] x [8 Ubuntu/OpenStack Release Combos], there are currently<b> ~432 </b>possible variations of amulet tests (derived bundles of fractional OpenStacks). A subset of those are executed in gate, depending on relevance to the developer's proposed change. This allows us to endure a high velocity of focused testing on development in these very active charms. Because the derived models are much smaller than the reference bundle, we can give developers rapid and automated feedback, plus they can iterate on development outside of our CI without having to be able to deploy a full OpenStack.<br><br>That is not to say that we don't have acceptance and integration tests for full OpenStack bundles. We do that in the form of mojo specs which dynamically deploy any number of full OpenStack bundle topologies and configurations against multiple Ubuntu+OpenStack release combos, using either the dev or the stable set of OpenStack charms. It basically takes what I've described above for amulet and allows us to pivot entire bundles into different models automatically. There are currently <b>84</b> such OpenStack mojo specs with tests (bundle equivalents)<br><br><div>Fear not, this is mostly accomplished with bundle inheritance, yaml foo, and shared test libraries. We're not actually maintaining ~<b>516 bundles</b>. But if we were to achieve the current level of coverage with bundles, that's approximately how many there would need to be. This includes the upcoming Xenial and Mitaka releases. Reduce by ~12% when Juno EOLs. Add 12% when we hit Newton B1, and so on.</div><div><br></div><div><div><br></div><div><b># How I'd like to use the proposed ideas</b></div></div><div><div>There are some OpenStack reference bundles in the charm store. My suggested approach would be to continue to leverage individual charm amulet tests while adding functional tests to the existing charm store bundles. That would increase test coverage, and provide a mechanism to validate proposed changes to those specific bundles, such as to re-validate the bundles when charm versions are revved within them.</div></div><div><div><br></div><div><br></div><div>To summarize, I am:</div><div><br></div></div>-1 to stopping or discouraging individual charm amulet tests<br><br>+1 for every charm containing amulet tests<br><br>+1 for every charm containing unit tests<br><br>+1 for every charm having amulet coverage in at least 1 bundle<br><br><div><div>+1 for every bundle possessing amulet tests</div></div><div><div><br></div><div><br></div><div>Also open to feedback, discussion, suggestions, kicks in the shin. </div><div><br></div><div>Thanks for all the great tooling and thought leadership. We leverage the everloving <i>stuff</i> out of Amulet.</div><div><br></div><div>Charm on!</div><div><br></div><div><div>-- </div><div>Ryan Beisner</div><div>QA Engineer, Ubuntu OpenStack Engineering, Canonical, Ltd.</div><div>irc:beisner gh/gerrit:ryan-beisner lp:~1chb1n</div></div></div><div><div><div><br></div></div></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 16, 2016 at 7:52 PM, Marco Ceppi <span dir="ltr"><<a href="mailto:marco.ceppi@canonical.com" target="_blank">marco.ceppi@canonical.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hello everyone!<div><br></div><div>This is an email I've been meaning to write for a while, and have rewritten a few times now. With 2.0 on the horizon and the charm ecosystem rapidly growing, I couldn't keep the idea to myself any longer.</div><div><br></div><div># tl;dr:</div><div><br></div><div>We should stop writing Amulet tests in charms and instead only write them Bundles and force charms to do unit-testing (when possible) and promote that all charms be included in bundles in the store.</div><div><br></div><div># Problem</div><div><br></div><div>Without making this a novel, charm-testing and amulet started before bundles were even a construct in Juju with a spec written before Juju 1.0. Since then, many new comers to the ecosystem have remarked how odd it is to be writing deployment validations at the charm level. Indeed, as years have gone by and new tools have sprung up it's become clear that; having an author try to model all the permutations of a charms deployment and do the physical deploys at that charm level are tedious and incomplete at best.</div><div><br></div><div>With the explosion of layers and improvements to uniting test in charms at that component level, I feel that continuing to create these bespoke "bundles" via amulet in a single charm will not be a robust solution going forward. As we sprint closer to Juju 2.0 we're seeing a higher demand for assurance of working scenarios, and a sharp focus on quality at every level. As such I'd like to propose the following policy changes:</div><div><br></div><div>- All bundles must have tests before promulgation to the store</div><div>- All charms need to have comprehensive tests (unit or amulet)</div><div>- All charms should be included in a bundle</div><div><br></div><div>I'll break down my reasoning and examples in the following sections:</div><div><br></div><div># All bundles must have tests before promulgation to the store</div><div><br></div><div>Writing bundle tests with Amulet is actually a more compelling story today than writing an Amulet test case for a charm. As an example, there's a new ELK stack bundle being produced, here's what the test for that bundle looks like: <a href="https://github.com/juju-solutions/bundle-elk-stack/blob/master/tests/10-test-bundle" target="_blank">https://github.com/juju-solutions/bundle-elk-stack/blob/master/tests/10-test-bundle</a></div><div><br></div><div>This makes a lot of sense because it's asserting that the bundle is working as expected by the Author who put the bundle together. It's also loading the bundle.yaml as the deployment spec meaning as the bundle evolves the tests will make sure they continue to run as expected. Also, this could potentially be used in future smoke tests for charms being updated if a CI process swaps out, say elasticsearch, for a newer version of a charm being reviewed. We can assert that both the unittests in elasticsearch work and it operates properly in an existing real world solution a la the bundle.</div><div><br></div><div>Additional examples:</div><div>- <a href="https://github.com/juju-solutions/bundle-realtime-syslog-analytics/blob/master/tests/01-bundle.py" target="_blank">https://github.com/juju-solutions/bundle-realtime-syslog-analytics/blob/master/tests/01-bundle.py</a></div><div>- <a href="https://github.com/juju-solutions/bundle-apache-core-batch-processing/blob/master/tests/01-bundle.py" target="_blank">https://github.com/juju-solutions/bundle-apache-core-batch-processing/blob/master/tests/01-bundle.py</a></div><div><br></div><div># All charms need to have comprehensive tests (unit or amulet)</div><div><br></div><div>This is just a clarification and more strongly typed policy change that require charms have (preferred) unit tests or, if not applicable, then an Amulet test. Bash doesn't really allow for unittesting, so in those scenarios, Amulet tests would function as a valid testing case.</div><div><br></div><div>There are also some charms which will not make sense as a bundle. One example is the recently promulgated Fiche charm: <a href="http://bazaar.launchpad.net/~charmers/charms/trusty/fiche/trunk/view/head:/tests/10-deploy" target="_blank">http://bazaar.launchpad.net/~charmers/charms/trusty/fiche/trunk/view/head:/tests/10-deploy</a> It's a standalone pastebin, but it's an awesome service that provides deployment validation with an Amulet test. The test stands up the charm, exercises configuration, and validates the service responds in an expected way. For scenarios where a charm does not have a bundle an Amulet test would be required.</div><div><br></div><div>Any charm that currently includes an Amulet test is welcome to continue keeping such a test.</div><div><br></div><div># All charms should be included in a bundle</div><div><br></div><div>This last one is to underscore that charms need to serve a purpose. This policy is written as not an absolute, but instead a strongly worded suggestion as there are always charms that are exceptions to the rules. One such example is the aforementioned Fiche charm which as a bundle would not make as much sense, but is still a purposeful charm.</div><div><br></div><div>That being said, most users coming to consume Juju are looking to solve a problem. Bundles underscore solutions to problems that people can consume, and get started quicker.</div><div><br></div><div>As such, when new applications are charmed a test of "is this application something that serves a clear purpose" having a bundle submitted alongside the charm validates that claim and provides users a way to immediately get started with a solution.</div><div><br></div><div># Conclusion</div><div><br></div><div>These policy changes, once accepted, will be targeted at all charms and bundles in Xenial as well as any new charm submitted after policy acceptance date for trusty, and finally any charm currently under review will be encouraged to adhere to the new policy but won't be required.</div><div><br></div><div># Action items</div><div><br></div><div>I'm seeking feedback on this concept and welcome suggestions for improvements, questions, dissenting opinions, and any other remarks as well as votes from ~charmers and feedback from the community at large.</div><div><br></div><div>Thanks,</div><div>Marco Ceppi</div></div>
<br>--<br>
Juju mailing list<br>
<a href="mailto:Juju@lists.ubuntu.com" target="_blank">Juju@lists.ubuntu.com</a><br>
Modify settings or unsubscribe at: <a href="https://lists.ubuntu.com/mailman/listinfo/juju" rel="noreferrer" target="_blank">https://lists.ubuntu.com/mailman/listinfo/juju</a><br>
<br></blockquote></div><br></div></div>