A simpler test framework?

[Martin Pool]

2009/7/15 Aaron Bentley <aaron at aaronbentley.com>:
>> This would be pretty approachable for new people adding tests and also
>> makes it very clear what's happening.  It could potentially be
>> copy-and-pasted from a bug report with just a little cleanup.
>
> I think we would want to avoid repeated invocation of bzr, so "% bzr
> add" would need to do run_bzr under the hood, and there's a lot of magic
> that way.  We'd need to implement a pseudo-shell, actually, so that it
> would work on Windows.

Yes, that's quite true.  And we might want to fake up the existence of
a universe in which you can do the testing, so that

 bzr branch http://example.com/bzr/foo

will actually go to a local http server run by the test suite, and
maybe there are what look like branches on the local disk with some
history.  But one step at a time.

(Did you know example.com is the officially sanctioned example domain
name? <http://en.wikipedia.org/wiki/example.com>)

>
>> We could alternatively support this kind of thing not based on doctest
>> but our own code, maybe providing an extended blackbox routine which
>> takes a triplequoted string containing commands and output and
>> evaluates it.  This could fit within regular tests, could be
>> parameterized and could be automatically cleaned up.
>
> There is also a new doctest-like thing: "manuel"
> http://packages.python.org/manuel/
>
> It seems to focus on extensibility.
>
>> I do think, looking at for example Aaron's 'merge -i' proposal, it
>> would have been nice to see some screenshot examples of it in use.
>
> Fully agreed.  Testing this interactive stuff is pretty hard with our
> current framework, but I'm not sure this would help that case.  Shelver
> hides its prompts after they're answered, so its output is something like:
>
> "Apply change? [yNfq?]y\r                     Apply 1 change(s)y\r
>                "
>
> etc.

I think in Shelver this should be abstracted a bit into a method
called on the UIFactory that asks for a multiple-choice response.
Then the UIFactory can arrange that when run in this context, it has
just

  Apply change? [yNfq] y

I think that's clearly the most desirable thing as far as testable
documentation or easy approachability, though it is starting to get
the doctest quality of heuristic rather than strict matching of the
output.

We'd need it, as it works through the script, to match the output of
the prompt from the first part of the line, and then take the rest of
the line as the input the user gave, in this case 'y', which would
then be passed back to the Shelver.

-- 
Martin <http://launchpad.net/~mbp/>