Questions after testtools merge

[Vincent Ladeuil]

>>>>> "jml" == Jonathan Lange <jml at mumak.net> writes:

<snip/>

    >>> However, if I read the above correctly, teardown is clearly called
    >>> *before* cleanups. Which doesn't quite fit with the fact that setUp is
    >>> called before addCleanup, etc.

I rely on that behavior a lot and had always considered:

try:
   test specific setup
   exercise SUT
   assertions
finally:
   test specific tear down

as equivalent to:

test setup
add cleanup
exercise SUT
assertions

And as long as cleanups are called *before* tearDown, I feel
safe. And I did the above change in many existing bzr tests.

This saved me from defining a specific class with its associated
setUp/tearDown for one test.

If this is no longer true, I agree with John and Martin
that we should stop mixing tearDown and addCleanup and use only
the later.

<snip/>

    jml> Actually, other than the mistaken docstring, it seems to
    jml> make the behaviour pretty clear. I'd be happy to make
    jml> some improvements if you can think of any.

Thinking more about it, I don't think it's clear at all, I first
thought that cleanups should only be called before tearDown but I
realized that what I really want is a clean unwinding of setups
that can't be implemented properly by separating cleanups and
tear downs as they are today.

On the surface, calling cleanups first addresses my point above but
more involved configurations can require that cleanups and
teardowns calls are intermixed and that would certainly makes
things harder to understand for little gain.

Still, as Martin showed, it's quite easy to fall in subtle traps
when cleanups are called *after* tearDown and the only safe way
to use cleanups it to never mix them with setups... quite hard to
get right.

Apart from the environment variables example, it means that you
can't use tearDown to delete a test directory and use cleanups to
delete files in that directory: 1) they are already gone, 2) You
can't check in tearDown that test aren't leaking anymore, 3)
tearDown should be ready to delete a non-empty directory.

    jml> The other thing I forgot to mention is that we used to
    jml> run the cleanups before tearDown, but we changed to
    jml> match what Python standard library does.

May be they should be fixed then :D

You said in a previous mail that calling before or after was
rather arbitrary, I disagree. The semantic is clearly different
and lead to different usages... but calling them after tear downs
require more care and can make the cleanups more complex... or
will just force us to abandon tear downs completely (which in
itself means you need to warn potential users anyway).

Should we continue the discussion and provide more examples or
even (gasp, tests :) or is it already a firm decision ?

If you want to keep cleanups called after tear downs, can we
discuss *another* feature where code can be called *before*
tearDown (or at the beginning of tearDown ) ?

    Vincent

P.S.: bzrlib.test.TestCase still defines a 'cleanups' attribute
that we want to delete as some future... cleanup ? :D