RFC: blackbox tests are not integration tests

[Martin Pool]

2009/9/17 Robert Collins <robertc at robertcollins.net>:
> Thus, I propose 3 new test types (and following our current conventions,
> that means 3 new subdirs):
> bzrlib.tests.acceptance
> bzrlib.tests.integration
> bzrlib.tests.cli
>
> The goal would be to end up with no blackbox tests, all tests being
> tuned to be more aligned with their actual intent and placed into one of
> those 3 containers, *or* moved to be a much more precise test.
>
> The acceptance test suites should be the smallest of our suites: they
> are intrinsically resistant to optimisations.
>
> The integration suite should be relatively cheap, though I suspect that
> many such tests would be written in the style of acceptance tests - so
> we don't want them added willynilly.
>
> The cli tests *today* have no better way of being written; but if we
> decide that a given test is testing the cli, it allows someone to make
> them radically faster [e.g. by introducing test fakes or a more concrete
> UI layer] without concern about dropping some intended aspect of the
> test.

+0.9, being clearer about the intention of each test will make it
easier to understand what they really need to test.  Also, this will
give us a chance to audit and remove some old misplace blackbox tests.

After discussion (or silent acquiescence :-) you should add this to
the developer guide.

But I think it would help if you can more clearly explain the
difference in practice between acceptance and integration tests.  It
looks like at the moment they'll look very similar.  That's not
necessarily a problem, but do you have a science fiction of how they
should look?

-- 
Martin <http://launchpad.net/~mbp/>