Follow-up on unit testing layered charms

[Pete Vander Giessen]

Hi All,

Some of you may recall, during a hangout a month or so ago, that I spoke
enthusiastically about a testing harness for Python-based layered charms
that we were working on in the Big Data team.

Unfortunately, that project ran into some technical blockers (details below
my sig, for the curious). As a result, we've moved away from the idea of
doing a testing harness, and instead put together a short list of "best
practices" for unit testing layered charms.

We currently have a PR open against the Juju docs here:
https://github.com/juju/docs/pull/1317, and we welcome feedback! If
anything needs clarification, if you disagree with anything in particular
actually being a "best practice", or you have anything to else to add,
please drop a note on that PR.

~ PeteVG

The details follow ...

The problem to be addressed: we'd like to run unit tests before building a
layered charm, as unit tests should execute in seconds (or less), and
`charm build` can take multiple seconds to run. Unfortunately, if you run
tests before Python libraries from other layers have been added by `charm
build`, you may wind up with ImportErrors in your test code.

The harness: I wrote a test harness that temporarily patches sys.modules
with mock objects in place of the missing libraries. This fixes the
ImportErrors, but it creates other problems, some of which proved to be
intractable ...

The problems with the harness: patching sys.modules leads to a catch-22: if
we don't leave the mocks in place, we still get import errors when using
the mock library's mock.patch method, but if we do leave them in place,
tests that set properties on them can end up interfering with each other.
There are workarounds, but they're not intuitive, and they don't generate
friendly error messages when they fail. We felt it best to leave the
harness behind, and provide some advice on more straightforward things that
you can do to work around the import errors. Thus the PR referenced above.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160830/587bed0c/attachment.html>