[RFC] small updates to test running

John Arbash Meinel john at arbash-meinel.com
Wed Jul 12 19:43:48 BST 2006


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Aaron Bentley wrote:
> John Arbash Meinel wrote:
>>> Aaron Bentley wrote:
>>> Well, I want a nice short list of the failed tests. Because frequently I
>>> don't use the traceback information, and it just clutters the screen.
>>> Martin is doing some work to help that, but it isn't here yet.
> 
> The actual traceback is almost always useful to me, but the log stuff
> almost never is.  Is it the traceback per se that you don't want to see,
> or the rest of the output?

I generally agree with you. Generally the times the other stuff is
useful is when I put it in there with my own mutter() calls.
Sometimes seeing the arguments and output of a run_bzr() call, and a few
other cases are useful.

So I would be okay with seeing the traceback. Except it still is nice to
have a short summary to see what tests are failing to pick out what you
want to work on.

> 
>>> I didn't figure out how to get scons to auto generate dependencies from
>>> the 'import' lines. But for now, it means that when I'm just my
>>> 'external' test suite, my 'internal' test suite doesn't run.
>>> It would be really nice to have a system where it could figure out more
>>> fine-grained dependencies, and then the default would be to only run
>>> tests that are affected by the current changes. Even better if it does
>>> direct dependent tests before far away dependent tests.
> 
> Real dependencies are not something a machine could determine.  We could
> do Aegis-style probability testing, though.
> 
> If you're willing to store the data, you can actually record failures
> according to which files were modified when the test failed.  Then your
> predictions get much better, and they don't require the test framework
> to understand how your project is structured.

Well, I was more thinking of looking at the import lines, and having
file A depend on file B if there is an 'import B' at the top.

Certainly in python duck-typing allows the dependencies to be inverted.
(I import you and pass myself into your code, etc)

But it is generally a good first approximation. (It *is* what gcc -MM
and scons use).

> 
>>> In the short-term, though. What you say would be pretty good. I think it
>>> would involve customizing the test runner a little bit more. It seems to
>>> just run the tests in the order that they appear in the TestSuite.
> 
> Yeah.  Anyway, I'm pretty pro your changes, just the elipsis thing needs
> discussion.
> 
> Aaron

I put in the _ellipsis stuff because it was already there for other
things. And it is kind of nice to only see 1 line, regardless of how
long the test name is. (We have some *really* long test names, like with
'workingtree_implementations', which is a set of adapted tests).

It also isn't high priority for me right now. I don't think it needs to
land before 0.9. Just something I was thinking about.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEtULkJdeBCYSNAAMRAm1MAKCWjhkx3Vvo86fIbCHEdjM4g70JhgCgltpU
UzycbfKJCanVmms9ayHwuoo=
=dg2t
-----END PGP SIGNATURE-----




More information about the bazaar mailing list