A simpler test framework? (was: Re: The 'easy' tag)

Martin Pool mbp at sourcefrog.net
Tue Jul 14 08:47:14 BST 2009

I thought of one specific thing we could do, which is to support
doctest-like black box tests, so you could write literally:

  % echo foo > foo
  % bzr add
  added foo

This would be pretty approachable for new people adding tests and also
makes it very clear what's happening.  It could potentially be
copy-and-pasted from a bug report with just a little cleanup.

We've talked before about doing this on top of python's doctest, which
can be used to test Python APIs in a similar style.  Doctest has a bit
of a bad reputation at least around Canonical for causing some testing
antipatterns but it's not necessarily bad if it's used in the right
place - testable documentation or maybe smoke tests.  It tends to run
on through the rest of the file if one test fails which causes a big
mess, and the parameterization and cleanup features we've developed in
the rest of the bzrlib test suite aren't available, and it only
compares output strings and can give unclear failure messages.

There's also the drawback that the command line interface is both
slower and less precise for setting up test cases and reporting errors
than using the programming API.  On the other hand there is a real
value in it being easy for new contributors to write tests, and to
read over the tests and quickly see what's done.

We could alternatively support this kind of thing not based on doctest
but our own code, maybe providing an extended blackbox routine which
takes a triplequoted string containing commands and output and
evaluates it.  This could fit within regular tests, could be
parameterized and could be automatically cleaned up.

I do think, looking at for example Aaron's 'merge -i' proposal, it
would have been nice to see some screenshot examples of it in use.
Even as testable documentation it would be good in doing the review.

So I guess it comes down to this: if you could write tests in this
style, would you or other people actually write new tests?

Martin <http://launchpad.net/~mbp/>

More information about the bazaar mailing list