test suite debugging

[Robert Collins]

On Thu, 2006-09-28 at 20:24 -0400, Aaron Bentley wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1


> > Ideally most tests will test one and only one thing. Support
> > infrastructure to test that thing should be unobtrusive and fast. For
> > instance, a test that WorkingTree.commit() uses commitbuilder in the
> > expected way does not need disk resources - so we can use a MemoryTree. 
> 
> It comforts me to know that tests are testing more than one thing,
> because I know how hard it is to cover all cases.  Frequently when I
> make a mistake, the test case that catches it is one that was never
> intended to detect that problem.
> 
> Anyhow, that's why my gut reaction is discomfort.  I wouldn't call it my
> considered opinion.
> 
> And I find that tests which test only one thing lead to repeated set-up
> and tear-down, which also makes the test suite slower.

Theres a couple of mispatterns that can occur. Tests that need a lot of
setup and teardown usually are testing something high up the dependency
chart, making the setup and tear down expensive. There are a couple of
routes to address that. managed resources such as 'testresources' offers
is one route, and stub implementations that provide the needed context
for the test are another. Globbing the tests together leads to tests
where the thing you want to test is stuck deep in a long series of
events, so a bug that affects a couple of areas will often prevent the
thing you want to test being tested - the test fails early. This makes
it harder to debug, as you need to wade through less helpful layers to
figure out whats wrong.

> > John - dumping to disk - I dont plan on that, we have a very good
> > introspection tool in pdb. I'm happy to defer converting tests where it
> > seems like that is important until there is a 'dump to disk after
> > failure' option, but honestly, in the last year I've used --keep-output
> > once.
> 
> I use pdb a lot, but I also use keep-output frequently.  Maybe a tenth
> as often as pdb, but it still adds up to plenty.

Interesting. What do you use keep-output for? (What precisely do you
look at when you are doing that?)

-Rob
-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060929/141bb523/attachment.pgp