[MERGE] Automatic discovery of tests

[Martin Pool]

On Nov 14, 2007 8:44 AM, John Arbash Meinel <john at arbash-meinel.com> wrote:
> How did this break the test suite.
>
> And I certainly don't feel a consensus was reached.
>
> The last thing from Alexander was that it was fine, as I believe the same from
> Vincent.
>
> It doesn't break the test suite for normal usage (installing bzr directly). It
> is only broken for the standalone installer.
>
> Robert feels iffy about having the tests automatically found, because of tests
> not getting run if they aren't found. However, we already are auto-loading the
> tests from files that we list. So it isn't introducing any new problems. In
> fact, I think it is closing a hole where we add the tests, but forget to update
> the lists.
>
> Double recording means that if you forget to 'bzr add' it will complain when it
> gets to the pqm. However, if you fail to add, we won't see any tests in review.
> Which again is safer than seeing the tests in review, and not having them run
> by the PQM. (Because you forgot to add a single line to the test list.)
>
> If the only blocker is because of the standalone installer, I can write the
> code to list tests inside a zip file.
>
> But I do think you rejected this before an actual consensus was reached.

I agree with your analysis.  I agree it's trading holes, but the
tradeoff is reasonable, and rather than spin on this anymore let's put
it in.  We can always revert if people turn out to often forget to add
their tests to the tree.

(Assuming the code actually works and any tradeoffs about testing exes
etc are acceptable.)

-- 
Martin