[RFC][Bug #111914]selftest options to fail if there are missing dependencies, known-failures, etc

[Martin Pool]

On 7/29/07, Daniel Watkins <D.M.Watkins at warwick.ac.uk> wrote:
> Daniel Watkins wrote:
> > Attached is a preliminary fix for bug #111914.
> The above is actually true of this post.

That looks ok.  You should put the string '[merge]' in the mail
subject so that Bundle Buggy catches it and your patch isn't lost.

Does everything pass in bzr.dev with this merged?

You should probably add a note in the testing section of HACKING
explaining this.

It looks like this needs the finer-grained test results I posted about
a while ago to be most useful: we want --strict to fail if there are
known failures or missing dependencies, but not because of OS
limitations.  But if it passes at present I don't mind putting it in
as preparation.

> @@ -2511,7 +2513,8 @@
>                                matching_tests_first=first,
>                                list_only=list_only,
>                                random_seed=randomize,
> -                              exclude_pattern=exclude
> +                              exclude_pattern=exclude,
> +                              strict=strict
>                                )
>          finally:
>              if benchfile is not None:
>

To make later updates easier it's good to include a trailing comma here.

> @@ -2206,7 +2207,12 @@
>              suite = filter_suite_by_re(suite, pattern, exclude_pattern,
>                  random_order)
>      result = runner.run(suite)
> -    return result.wasSuccessful()
> +    success = result.wasSuccessful()
> +
> +    if strict and (result.known_failure_count or result.unsupported):
> +        success = False
> +
> +    return success

I wonder if evaluation of this should be moved into the result through
e.g. result.wasStrictlySuccessful()?  Then as we add more results we
shouldn't need to change this method.

>  def selftest(verbose=False, pattern=".*", stop_on_failure=True,
> @@ -2217,7 +2223,8 @@
>               matching_tests_first=None,
>               list_only=False,
>               random_seed=None,
> -             exclude_pattern=None):
> +             exclude_pattern=None,
> +             strict=False,):
>      """Run the whole test suite under the enhanced runner"""
>      # XXX: Very ugly way to do this...
>      # Disable warning about old formats because we don't want it to disturb

And if you have a comma the paren should be on the following line.

-- 
Martin