selftest performance: landing our code faster and developing quicker.

[Vincent Ladeuil]

>>>>> "jam" == John Arbash Meinel <john at arbash-meinel.com> writes:

    jam> ...

    martin> We should probably make sure if any compiled
    martin> extensions are missing that gives a clear warning in
    martin> the test run; it's ok to test without them but you
    martin> should know.
    >> 
    >> +1

    jam> Speaking of which, we should probably have a buildbot that runs the test
    jam> suite *without* extensions built. Given that we claim to support that,
    jam> and we've found a couple of bugs accidentally under those circumstances.

<cough>

The OS X test suites don't build their extensions so this is
actually the case.

Of course I should fix that and then add another build.

But on the other hand, we shouldn't just multiply the selftest
runs semi-blindly just to ensure we cover edge cases.

We could do so in the short term, but long term, we'd better
handle these edge cases explicitly in the test suite if only
because everybody will then check them at a lower cost (neither
the locale nor these extensions tests requires the *whole* test
suite).

I think we did a pretty good job lately to ensure that python and
pyrex versions receive the same unit tests, I don't think we
encountered bugs due to lack of coverage there but even if we
did, that's bumps on the road but this road is the good one.

We still can have such a specific run without building the
extensions, just to reassure ourselves that we're doing good, but
once a week say, should be enough.

     Vincent