[RFC] small updates to test running

[Aaron Bentley]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

John Arbash Meinel wrote:
> Aaron Bentley wrote:
> Well, I want a nice short list of the failed tests. Because frequently I
> don't use the traceback information, and it just clutters the screen.
> Martin is doing some work to help that, but it isn't here yet.

The actual traceback is almost always useful to me, but the log stuff
almost never is.  Is it the traceback per se that you don't want to see,
or the rest of the output?

> I didn't figure out how to get scons to auto generate dependencies from
> the 'import' lines. But for now, it means that when I'm just my
> 'external' test suite, my 'internal' test suite doesn't run.
> It would be really nice to have a system where it could figure out more
> fine-grained dependencies, and then the default would be to only run
> tests that are affected by the current changes. Even better if it does
> direct dependent tests before far away dependent tests.

Real dependencies are not something a machine could determine.  We could
do Aegis-style probability testing, though.

If you're willing to store the data, you can actually record failures
according to which files were modified when the test failed.  Then your
predictions get much better, and they don't require the test framework
to understand how your project is structured.

> In the short-term, though. What you say would be pretty good. I think it
> would involve customizing the test runner a little bit more. It seems to
> just run the tests in the order that they appear in the TestSuite.

Yeah.  Anyway, I'm pretty pro your changes, just the elipsis thing needs
discussion.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFEtT1F0F+nu1YWqI0RAqL2AJ4gCLxnOpAfzcfL89oqSLUmIOvnewCfeLWh
zlftIjCgVPbGZRjhff3POuw=
=FGIt
-----END PGP SIGNATURE-----