[RFC] testable tutorial / doctests for ui

[Robert Collins]

On Mon, 2006-01-30 at 21:50 +1100, Martin Pool wrote:
> On 29 Jan 2006, Robert Collins <robertc at robertcollins.net> wrote:
> > Hi,
> > 	I dont recall if this has been discussed on list or not, but it has
> > certainly been discussed a little here n there.
> > 
> > There are two related concepts:
> >  * Making the example commands in the tutorial be tested during test
> > suite runs, so that its always current and accurate.
> >  * Making it easier to write CLI tests by providing an extension of the
> > doctest parser that understands bzr commands.
> 
> Yes, I'm definitely in favor of the concept.  
> 
> I think the emphasis should definitely be on testing the documentation,
> not on using the documentation to test the UI.  (Testing the UI is
> definitely important, but the type and number of tests to give good
> coverage don't really belong in user documentation.)  In other words the
> point is mainly to check that the tutorial is up to date with the code.

I see two applications for the technique:
a) making sure the tutorial isn't stale, and ditto for the api
documentation [Testable documentation]
b) in bzrlib/tests/blackbox/ we have a raft of tests that are primarily
a sequence of ui commands being checked for return code and output. I
thin it may make sense for them to be written using this more pithy
form.... [doctests for the ui]


> We need to think about which tests need to be grouped together into a
> common context (i.e. the same test directory).  Doctest by default runs
> them in small groups but for a tutorial we may want the whole thing to
> build up one example.

Sure, we can do that. The way we do it in launchpad is to have stories.
A story is a subdirectory with a sequence [ordered by filename] of
doctest test scripts, and global state they accrue is preserved across
the scripts. I will look into changing this to preserve the local state
too. We can then choose whether the tutorial should have multiple
stories, or be one big story. We can for example have a tutorial, a user
manual, and a HOWTO guide all tested separately using this.

> We'll need to use the doctest feature of ignoring some varying parts of
> output, e.g. date stamps.  Or perhaps we could have some hidden
> sideeffects that cause the clock to be set predictably.
> 
> As much as possible I would like the commands to be written to run on
> either Unix or Windows.  Possibly variations can be marked as such:
> 
> .. bzrdoctest: unix
>     $ ln -s foo bar
> 
> We may need to say "run your editor here".

I was thinking that like with Mock objects, we can handle that..
something like

...
    bzreditorresponse:
        foo
        bar
    $ bzr commit
    ...

Rob


-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060131/6930b9f4/attachment.pgp