[RFC] parsable test suite log

[Martin Pool]

On Tue, Mar 25, 2008 at 2:21 PM, Robert Collins
<robertc at robertcollins.net> wrote:
> I'd like to do some machine processing of test suite runs.
>
>  In the first instance, I'd like to get a list of failed tests from the
>  previous run. The workflow I'm anticipating is:
>  a) write code
>  run some tests
>  loop a until code completed
>  b) run all tests with --record

What would --record be?

How about just --subunit telling it to put the output in that style?

>  c) take list of failures, fix one
>  run just the failures
>  loop c until no failures
>  loop b until no failures
>
>  Now, I have a serialisation format I prepared earlier: subunit. I could
>  plug this in trivially. subunits wire protocol records test runs and
>  could allow some nice introspection stuff in future.
>
>  Alternatively, I can do a custom serialisation format like vila proposed
>  for his fast-loading stuff; I'd rather not though.
>
>  Note that this isn't changing loading facilities at all - no shelf or
>  anything; its just about being able to get info about the prior test run
>  via python, rather than copy n paste.

That would be good.

(The following is based on last time I read about subunit; it might be wrong.)

There is some structure within our test output that isn't contemplated
by subunit; specifically we have a traceback and some log output.
Maybe it doesn't matter, we should just dump it as one text result and
let the subunit client figure it out.

I would love to have a gui where i can click a test to (re)run it and
click a line of output to go there in my editor.

-- 
Martin <http://launchpad.net/~mbp/>