Matrix: An Update

[Cory Johns]

Greetings,

We wanted to give an update on the progress and direction of the Matrix
project, which Ben announced previously[1].  To recap, Matrix is a new
bundle testing tool that augments the existing testing tools, such as
bundletester and mojo, by differing in the following core philosophies:


   -

   Juju provides a set of operational primitives that are generally
   expected to apply to any bundle and charms.  Given that, if we test those
   primitives against a bundle, we can gain a level of confidence in the
   fitness of that bundle, and provide feedback to the bundle and charm
   authors, while both shifting the burden of testing away from the charm
   author and toward the stewards of the testing tool as well as ensuring that
   the tests cover cases which might not be considered by the charm author.



   -

   Unpredictability and failure are inherent properties of running big
   software, especially in the cloud; thus, the testing should include
   somewhat chaotic and potentially destructive mutations to the deployment.
   This will both uncover bugs (in Juju, charms, and the software the charms
   manage) as well as provide insight into areas for improving reliability of
   the bundle in production.


Ben, Pete, and I have been working to get the first version finished and
polished up, and have begun working with the Juju QA team to transfer
ownership over to Torsten, Seman, and Aaron for maintainership of the
default set of test suite.  Our goal to finish this transition is December
16th.

The current default test suite can be run against a bundle and verify that
it deploys and stabilizes (within the somewhat limited ability currently
available for charms to report health, i.e., the agent and workload status
at the unit and application level), and then runs a random set of mutations
(a.k.a. “glitch”) to the deployment and watches the health as that
progresses.  (This is similar to Netflix’s idea of a “chaos monkey.”)  The
glitch plan that is generated is saved and can be replayed to reproduce a
failure.

The bundle under test can additionally provide a custom set of suites in a
tests/matrix.yaml file, as well as custom tasks to test things that are
non-generic and specific to the bundle being tested or in more depth than
can be tested by the generic framework.  The custom suites can also
override the tests built into the Matrix, if for some reason they need to
be tweaked, or to test new functionality before it is moved into the
generic test suites.

We will continue to work with the QA team to support them improving the
built-in suites, and to add features to improve the tests that are done. We
are confident that the QA team will provide great insights moving forward
as they take ownership of the Matrix project.

Once again, the code for Matrix lives at
https://github.com/juju-solutions/matrix and reviews, comments, issues, and
contributions are welcome.

[1] https://lists.ubuntu.com/archives/juju/2016-November/008108.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20161201/8ab4987c/attachment.html>