[MERGE] Move the notes on writing tests out of HACKING into a new file, and improve them.

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew Bennetts wrote:
> Adrian Wilkins wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> John Arbash Meinel wrote:
>>
>>> But it isn't going to handle specific win32 cases (like having
>>> FindFilesEx, etc.)
>> To do proper full continuous integration including testing win32 cases, you'd
>> need a machine (virtual or otherwise) that could be remotely called to run the
>> tests and return results. This might need the test suite to output more
>> structured output (can it?).
>>
>> You might need more than one win32 machine if you start supporting Vista
>> features like symlinks.
> 
> Yep.  Automated testing is great!  It's been discussed a bit on the list
> before... IIRC, the two obvious options are:
> 
>  - arrange for PQM to trigger 'make check' on multiple machines, and require
>    them all to pass, rather than just the single local 'make check' it does
>    currently.
> 
>  - set up a BuildBot master, and set up some slaves on multiple machines to run
>    tests after each change to a PQM-managed branch.
> 
> The first option really requires that all tests already pass for a platform, so
> it would be relatively hard to do for Windows as it is today (although thanks to
> Mark Hammond it's rapidly improving).  There's also some hand-waving in how to
> make PQM kick off make check on multiple platforms... (possibly write a script
> to that runs 'bzr co $target foo && cd foo && bzr merge $proposed && make check'
> via SSH?).
> 
> The second option can be done without requiring all tests to be passing already,
> but would still make test results for multiple platforms more automatic and
> visible to developers.
> 
> Either one would be great, IMO.  The biggest hassle is probably just finding and
> managing the machines.  Does anyone have a spare win32 machine (virtual or real)
> with net access they want to turn into a bzr test slave?
> 
> -Andrew.
> 
> 
> 

Well, the other problem with that as a hard-dependency, is that the developer
who wrote the patch may not have access to a win32 machine to be able to debug
and actually fix it.

So the first part is that we want a way to *know* when things break. (A *long*
time ago, I actually had a clean 'bzr selftest' on win32. Maybe back around
0.8 or 0.9.

This was in the 0.9 release notes:

    * Lots of win32 fixes (the test suite passes again).
      (John Arbash Meinel, #50155)

Anyway, we need to get back there, so that we know when incremental changes
break something, and then the author (or a win32 shepherd) knows that
something needs to happen.

I would like to say that all tests should pass on all platforms for every
commit to mainline. But I get the feeling that is a bit onerous.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIo50IJdeBCYSNAAMRAjivAJ94YhGtto3992Pu22t/cYAfx263JgCgjrE/
Q189hXXSIL5bJmGsPvQF/4s=
=w0tp
-----END PGP SIGNATURE-----