Charm Testing == Complex Deployment Testing Automation

James Page james.page at ubuntu.com
Mon Oct 3 14:25:53 UTC 2011


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi All

I've spent the last couple of days working on a rough approach and
prototype to testing deployed sets of charms/services.

The specific objective of this work was to be able to automated the
testing of a Juju deployed OpenStack environment.

This is still very much first draft and I would potentially expect
some of the features I'm discussing in this email to evolve into
something a little more concrete in the long term.

Overview
- --------

The approach introduces a new charm - the 'charm-tester':

 lp:~james-page/charm/oneiric/charm-tester/trunk/

This is a simple charm which has a minimal interface:

provides:
  tester:
    interface: testing

Any charm which wants to be tested by the charm-tester needs to
implement an optionally required interface, e.g. :

requires:
  ...
  tester:
    interface: testing
    optional: true

When a service relates to the charm-tester service, the charm-tester
publishes a SSH public key on the relation - this is used during the
actual test execution later and should be installed in the ubuntu
account on related service:

                  tester-relation-joined:
          charm-tester --- pubkey ---> related-service

The service then needs to publish its SSH identify back to the
charm-tester service - this also acts as the signal that the
charm-tester should then test the related service.

                  tester-relation-changed:
          charm-tester <--- hostid --- related-service

At this point in time the charm-tester then executes the tests on the
related service via SSH and collects the results of testing using scp.

What the related service does in terms of how it tests itself if
fairly flexible so long as it fulfills the following requirements:

1) The tests are run over ssh using the following command:

  cd /opt/formula-tester; sudo ./run_test.sh

2) Results are expected to be stored in:

  /opt/formula-tester/results/

I felt it was important that a charm should encapsulate the tests that
validate it is operating correctly for a given deployment scenario.

In the OpenStack test cases I've implemented a python unittest and
used subunit to produce JUnit XML output - this helps with integration
into Jenkins as this is supported directly by Jenkins core for test
reporting.

Links to examples for OpenStack testing:

  lp:~james-page/charm/oneiric/nova-compute/charm-tester
  lp:~james-page/charm/oneiric/nova-cloud-controller/charm-tester
  lp:~james-page/charm/oneiric/mysql/charm-tester
  lp:~james-page/charm/oneiric/rabbitmq-server/charm-tester
  lp:~james-page/charm/oneiric/glance/charm-tester

At the moment the tests are fairly basic - they just check for running
daemons - but they could easily be extended to complete more complex
testing.

This lays out a general framework for executing tests using juju.

I've also lashed together a python script that glues this all together:

 lp:~james-page/+junk/test-charm

Usage: execute_formula_test.py [options] test

Options:
  -h, --help   show this help message and exit
  -d, --debug  enable debugging

This script:

i) juju bootstraps and environment
ii) executes a 'deploy' hook to deploy a set of charms
iii) executes a 'test' hook to setup the charm-tester with the set of
deployed charms (I don't like this and it will be refactored)
iv) Collects the test results from the charm-tester
v) Collects general log and configuration data from all machines
vi) Destroys the environment

This can then be combined with a test-charm test:

  lp:~james-page/+junk/openstack-charm-test

Which defines charm configuration, deploy and test hooks for openstack
testing.

This can then be executed and reported on by Jenkins:


http://ec2-79-125-48-136.eu-west-1.compute.amazonaws.com:8080/job/juju-openstack-test/

OpenStack is just one test scenario - this approach could be applied
to test any type of charm and I think goes some way towards fulfilling
the goal of automating complex deployment testing.

Gotchas
- -------

1) It would have been nice to be able to validate that the relations a
service is currently maintaining are functional - however I could not
find a way to identify current relations and configuration data.

2) Service Relation state; you will observe that there are quite a few
sleep's in the deploy and test hooks - this is because as soon as a
relation is built it goes to state up; it may still be firing under
the hood so the sleeps let everything settle before testing happens.
It would be nice to have something that allows the activity state of
juju to be assessed.

I think that just about covers it.

Feedback much appreciated - hope this provokes some good discussion!

Cheers

James

P.S. Yes - I know there are a few references to formula - I will fix
those...
- -- 
James Page
Ubuntu Server Developer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJOicXtAAoJEL/srsug59jDbcUP/iqkSKZuybgyiWREMXJFd6G3
iXxlxiz18+Yw6gemCuxsYFvLoF7OcHFJGL1JWI7Aw3/3EKDsVNsClIDIFgIJP3RJ
YYtUOhWNQkkX0ykgt3gF4e9jyctllp0C9ISEl/HjxvIs9pyEq43gt9k3FDeeBMK5
0kA/E3OtmwDFsM36YLpT3QUgit4ltaBRwnP7TSpKnLkWGrh5FoW26IqzVzKpE6p4
yPiN9G7Jb1CrHf6rIt2ZMGZNqYwfk/8WLX0Bsz5iKmsb/XF/qXcKO+ydoGwQi2ga
X4NQ41RdVbGlUAkN1GSWNJcbpLZpY3VwN6PYZ1u1vkePwW2s22Jlcdc/gLJVhh4Q
LI0PR6v2sPuMVrjKCVTfw1ock9WHEocz44byHylPOXOkSfkEiiUh5psTIklFBDNU
mTRZ+7X9+MMlwlxWLe45jUq3K71HOF4YYDKJu93NJSEow+JdjfCOLxSPiyOoSe17
IUfBQjR7GMa9kA3DmZFmapTHq6TvIhFsyzoD94zIV+lr4Juvpxv5RZrWr9Pg8zRo
bezcGwrO8whLNNa2yjeGmCHUeoEEn98/SKfjVvn0vWAUlZ/u3NxN234MN+rWfzHC
HfEf1aUhK2KtKBX+/xx4DQwhAqFBy7uc80QQHnWMU8wwOufcFRJThVBNX5Ph5yeF
52q41/sLK6u5V7/1EdOA
=UgYm
-----END PGP SIGNATURE-----



More information about the Juju mailing list