[MERGE] Move the notes on writing tests out of HACKING into a new file, and improve them.

[Andrew Bennetts]

Adrian Wilkins wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> John Arbash Meinel wrote:
> 
> > But it isn't going to handle specific win32 cases (like having
> > FindFilesEx, etc.)
> 
> To do proper full continuous integration including testing win32 cases, you'd
> need a machine (virtual or otherwise) that could be remotely called to run the
> tests and return results. This might need the test suite to output more
> structured output (can it?).
> 
> You might need more than one win32 machine if you start supporting Vista
> features like symlinks.

Yep.  Automated testing is great!  It's been discussed a bit on the list
before... IIRC, the two obvious options are:

 - arrange for PQM to trigger 'make check' on multiple machines, and require
   them all to pass, rather than just the single local 'make check' it does
   currently.

 - set up a BuildBot master, and set up some slaves on multiple machines to run
   tests after each change to a PQM-managed branch.

The first option really requires that all tests already pass for a platform, so
it would be relatively hard to do for Windows as it is today (although thanks to
Mark Hammond it's rapidly improving).  There's also some hand-waving in how to
make PQM kick off make check on multiple platforms... (possibly write a script
to that runs 'bzr co $target foo && cd foo && bzr merge $proposed && make check'
via SSH?).

The second option can be done without requiring all tests to be passing already,
but would still make test results for multiple platforms more automatic and
visible to developers.

Either one would be great, IMO.  The biggest hassle is probably just finding and
managing the machines.  Does anyone have a spare win32 machine (virtual or real)
with net access they want to turn into a bzr test slave?

-Andrew.