deprecation warnings and the test suite

[Andrew Bennetts]

On Tue, Feb 21, 2006 at 08:13:25AM -0500, Aaron Bentley wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Robert Collins wrote:
> | I think that we should not merge code that leaves deprecation warnings
> | being triggered by the test suite - right now todays merges have
> | introduced a handful of such warnings.
> 
> Ideally yes, but it's hard to notice new deprecation warning when there
> are so many already being produced by the test suite.

One way to notice them is to make the test suite fail if there are new
deprecation warnings.  A way to make that happen would be to teach the test
runner to count warnings that occur while running the tests, then check a file
in the tree, say "ratchets.txt", to see if they exceed a threshold.

The file could look like:

    DeprecationWarning: 42
    FooWarning: 88

You could even have lines like:

    Modules with no docstring: 99

(with an accompanying test that imports all the modules in bzrlib, and counts
those without __doc__, and tests that it doesn't exceed the value from
ratchets.txt).

When someone fixes some deprecations, they can also ratchet the tolerance of
deprecation warnings down by adjusting ratchets.txt...

Eventually, when the value hits zero, you just change the test to
unconditionally fail if the value exceeds zero, and delete the line from
ratchets.txt :)

[This is an idea I've been meaning to implement elsewhere, but haven't yet
gotten around to]

-Andrew.