[RFC] small updates to test running

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Aaron Bentley wrote:
> John Arbash Meinel wrote:
>>> Second, I think it would be nice if the non-verbose mode would print out
>>> a single line for failed/error tests. It goes back to Aaron's earlier
>>> complaint, that now you have to run the whole test suite before it
>>> prints out anything about what has failed. I think doing something like:
>>>
>>> test.foo.bar.baz: FAIL
>>> test.xxx.yy.zzz: ERROR
> 
> I like this, but I'm inclined to take it further: we can print out the
> entire test failure when it actually fails.  I don't think there's a
> real benefit in delaying that output until all the tests have been run.
>  Is there?
> 

Well, I want a nice short list of the failed tests. Because frequently I
don't use the traceback information, and it just clutters the screen.
Martin is doing some work to help that, but it isn't here yet.

> I guess for -v, there is, but I rarely do -v.
> 
> -            self.pb.update(self._ellipsise_unimportant_words('ERROR',
> 13), self.testsRun, None)
> +            self.pb.note(self._ellipsise_unimportant_words(
> +                            test.id() + ': ERROR',
> +                            osutils.terminal_width(), keep_start=True))
> 
> I'm not sure we need to elipsise this.
> 
> In terms of dealing with slow tests, I'd love to take an approach of
> minimizing the time-to-failure.  For each test, record:
> 
> 1. total number of failures (f)
> 2. total duration (d)
> 
> Then sort the test cases by d/f.
> 
> Aaron

Actually, my favorite thing to do would be to do real dependency
checking. I think our code base is a little intermixed, so I'm not sure
how well it would work...
But for my projects I've started using Scons, and hooking it into the
test suite (a little). So it doesn't run tests that haven't changed
unless the code they depend on has changed.
I didn't figure out how to get scons to auto generate dependencies from
the 'import' lines. But for now, it means that when I'm just my
'external' test suite, my 'internal' test suite doesn't run.
It would be really nice to have a system where it could figure out more
fine-grained dependencies, and then the default would be to only run
tests that are affected by the current changes. Even better if it does
direct dependent tests before far away dependent tests.

In the short-term, though. What you say would be pretty good. I think it
would involve customizing the test runner a little bit more. It seems to
just run the tests in the order that they appear in the TestSuite.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEtS49JdeBCYSNAAMRAswkAKDOf/mDiSK+BYpcZETf4FgsXX2DIACfTWsR
E8uq+yVzf747xMdsjR44Pj8=
=x03E
-----END PGP SIGNATURE-----