The spec looks good in general. What follows are inline comments on
the copy & pasted context.
To facilitate tests attached to charms, a simple structure will be
utilized to attach tests to charms. Under the charm root directory,
"To facilitate tests attached to charms .. to attach tests to charms."
Would be nice to reword.
* the default environment is bootstrapped
What's the default environment?
* The CWD is the charm root directory
It would be more comfortable to have the tests directory as such.
* the bare name of any charm in arguments to juju will be resolved to a
charm url and/or repository arguments of the test runner's choice.
The "runner's choice" isn't clear or helpful. We have to define what that means.
* a special sub-command of juju, ``deploy-previous``, will deploy the
last successfully tested charm instead of the one from the current
delta. This will allow testing upgrade-charm.
This command doesn't exist. Let's please remove that from the
specification for the moment. We can define how to do that in a future
version once the first version is in.
* ``~/.juju`` will not be accessible to the tests
If present, tests/requirements.yaml will be read to determine packages
that need to be installed on the host running tests in order to facilitate
the tests.
Can we please name that as "tests/tests.yaml", so that this may be
used for other needs too?
If a tool is needed to perform a test and not available in the Ubuntu
s/and not/and is not/
The purpose of these tests is to assert that the charm works well on the
intended platform and performs the expected configuration steps. Examples
of things to test in each charm beyond install/start is:
Section looks good.
There is a general convention which output should follow, though it will
not be interpreted by machine. On stdout, a message indicating the reason
Why the convention? The runner should interpret the exit code and take
appropriate action. The test output should contain arbitrary details
about what the test is doing, and should be made available in case
something fails so that one can better understand what is being done.
The shell script should also be run with -e, which means it may fail
arbitrarily and give no chance to print anything.
teardown() {
juju destroy-service memcached
juju destroy-service mysql
juju destroy-service mediawiki
Destroying the service manually should not be necessary. The harness
should be able to put the environment back onto a clean state without
any actions from the test itself.
if [ -n "$datadir" ] ; then
if [ -f $datadir/passed ]; then
rm -r $datadir
Ditto.
if [ $try -eq 600 ] ; then
echo FAIL: Timed out waiting.
exit 1
fi
Not only the prefix is unnecessary here, but the whole test won't be
reached since wget will fail with a non-zero status and break down the
test early, with proper information that this was the case.
touch $datadir/passed
What's that doing?
trap - EXIT
teardown
echo INFO: PASS
exit 0
All of that looks like unnecessary boilerplate we can save every test
from having.
test results in such a way where a developer can repeat the exact set
s/where/that/?
All of the charms will be scanned for tests in lexical order by
series, charm name, branch name.
Sounds like an implementation detail that is unnecessary to specify.