[RFC] testable tutorial / doctests for ui

[Martin Pool]

On 29 Jan 2006, Robert Collins <robertc at robertcollins.net> wrote:
> Hi,
> 	I dont recall if this has been discussed on list or not, but it has
> certainly been discussed a little here n there.
> 
> There are two related concepts:
>  * Making the example commands in the tutorial be tested during test
> suite runs, so that its always current and accurate.
>  * Making it easier to write CLI tests by providing an extension of the
> doctest parser that understands bzr commands.

Yes, I'm definitely in favor of the concept.  

I think the emphasis should definitely be on testing the documentation,
not on using the documentation to test the UI.  (Testing the UI is
definitely important, but the type and number of tests to give good
coverage don't really belong in user documentation.)  In other words the
point is mainly to check that the tutorial is up to date with the code.

> So to do this, someone needs to write up a doctest parser, and a
> supporting emulation of cwd etc.
> 
> Seems to me that if we agree on what the syntax to be used in such tests
> is, someone can write the parser etc. :)..
> 
> So, I propose the following:
> 
> I'd like to allow the use of python in the tests, for when we need to
> get down and dirty - and it would be useful in some parts of the
> tutorial too I suspect. (I.e. if we have a tutorial on using the
> library).

I think being able to run Python code may be useful in making the tests
work, but it should not appear in the processed output, unless of course
it is a tutorial about how to use the Python API.  I don't think we need
to disable the doctest thing that extracts python interactions but we
should try to make sure they're not needed.

We need to think about which tests need to be grouped together into a
common context (i.e. the same test directory).  Doctest by default runs
them in small groups but for a tutorial we may want the whole thing to
build up one example.

We'll need to use the doctest feature of ignoring some varying parts of
output, e.g. date stamps.  Or perhaps we could have some hidden
sideeffects that cause the clock to be set predictably.

As much as possible I would like the commands to be written to run on
either Unix or Windows.  Possibly variations can be marked as such:

.. bzrdoctest: unix
    $ ln -s foo bar

We may need to say "run your editor here".

-- 
Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060130/ac46c14b/attachment.pgp