Testing of ansible-based charms

[Michael Nelson]

Hi people,

I saw a ping from last Friday:

12:21 < jcastro> bloodearnest, noodles775: evan is asking (We're at a
sprint) on how we test the ansible bits in a charm?
12:47 < bloodearnest> jcastro: not easily, other than actually running
it, which is a downside to using ansible currently
12:47 < bloodearnest> jcastro: the 2 things I do do are:
12:48 < bloodearnest> 1) use ansible-playbook --syntax-check to check for typos
12:48 < bloodearnest> 2) unit test any templates standalone
12:48 < jcastro> ack, thanks, exactly what we needed to know, thanks.
12:49 < bloodearnest> jcastro: we've talked about spending some time
seeing if we can write something that mocks out ansible at some level
(it's all py
thon, after all)

and thought it was worth a followup discussion.

IMO unit-testing of an ansible charm only makes sense in the context
of unit-testing any logic in your templates/playbooks and or the
*syntax* of your playbook (as Simon mentioned).

Unit-testing any actual tasks that modify the system is going to
require lots of mocking, just like it does when unit-testing straight
python hooks. It'd be better to have the ansible modules unit-tested
themselves (if they are custom), and focus on integration tests for
the charm (which can be included in your actual playbook - see the
google-groups thread below).

All the arguments both ways are exchanged between Michael deHaan
(ansible) and a few others on this thread:

https://groups.google.com/forum/#!topic/ansible-project/7VhqDDtf6Js

summarised by one of the people asking about testing here:

http://sharknet.us/2013/12/12/test-driven-development-with-ansible/

I've also read about people gating their ansible changes by running
these tests on travis-ci (or anywhere where you can test against a
known vm state):

http://renemoser.net/blog/2014/01/01/test-ansible-roles-with-travis-ci/

We'd like to do something similar for the shareable ansible roles that
we're using for our charms.

If anyone has other ideas or issues, please contribute :)

Cheers,
Michael