Charm Testing == Complex Deployment Testing Automation
james.page at ubuntu.com
Mon Oct 3 14:25:53 UTC 2011
-----BEGIN PGP SIGNED MESSAGE-----
I've spent the last couple of days working on a rough approach and
prototype to testing deployed sets of charms/services.
The specific objective of this work was to be able to automated the
testing of a Juju deployed OpenStack environment.
This is still very much first draft and I would potentially expect
some of the features I'm discussing in this email to evolve into
something a little more concrete in the long term.
The approach introduces a new charm - the 'charm-tester':
This is a simple charm which has a minimal interface:
Any charm which wants to be tested by the charm-tester needs to
implement an optionally required interface, e.g. :
When a service relates to the charm-tester service, the charm-tester
publishes a SSH public key on the relation - this is used during the
actual test execution later and should be installed in the ubuntu
account on related service:
charm-tester --- pubkey ---> related-service
The service then needs to publish its SSH identify back to the
charm-tester service - this also acts as the signal that the
charm-tester should then test the related service.
charm-tester <--- hostid --- related-service
At this point in time the charm-tester then executes the tests on the
related service via SSH and collects the results of testing using scp.
What the related service does in terms of how it tests itself if
fairly flexible so long as it fulfills the following requirements:
1) The tests are run over ssh using the following command:
cd /opt/formula-tester; sudo ./run_test.sh
2) Results are expected to be stored in:
I felt it was important that a charm should encapsulate the tests that
validate it is operating correctly for a given deployment scenario.
In the OpenStack test cases I've implemented a python unittest and
used subunit to produce JUnit XML output - this helps with integration
into Jenkins as this is supported directly by Jenkins core for test
Links to examples for OpenStack testing:
At the moment the tests are fairly basic - they just check for running
daemons - but they could easily be extended to complete more complex
This lays out a general framework for executing tests using juju.
I've also lashed together a python script that glues this all together:
Usage: execute_formula_test.py [options] test
-h, --help show this help message and exit
-d, --debug enable debugging
i) juju bootstraps and environment
ii) executes a 'deploy' hook to deploy a set of charms
iii) executes a 'test' hook to setup the charm-tester with the set of
deployed charms (I don't like this and it will be refactored)
iv) Collects the test results from the charm-tester
v) Collects general log and configuration data from all machines
vi) Destroys the environment
This can then be combined with a test-charm test:
Which defines charm configuration, deploy and test hooks for openstack
This can then be executed and reported on by Jenkins:
OpenStack is just one test scenario - this approach could be applied
to test any type of charm and I think goes some way towards fulfilling
the goal of automating complex deployment testing.
1) It would have been nice to be able to validate that the relations a
service is currently maintaining are functional - however I could not
find a way to identify current relations and configuration data.
2) Service Relation state; you will observe that there are quite a few
sleep's in the deploy and test hooks - this is because as soon as a
relation is built it goes to state up; it may still be firing under
the hood so the sleeps let everything settle before testing happens.
It would be nice to have something that allows the activity state of
juju to be assessed.
I think that just about covers it.
Feedback much appreciated - hope this provokes some good discussion!
P.S. Yes - I know there are a few references to formula - I will fix
Ubuntu Server Developer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
-----END PGP SIGNATURE-----
More information about the Juju