[MERGE] Hidden bug in `bzr info` when dirstate tree is write-locked

[Martin Pool]

On 4/25/07, John Arbash Meinel <john at arbash-meinel.com> wrote:
> Well, I updated the comment as we discussed on IRC.
>
> I personally feel that you should never raise "knownFailure()" without
> actually checking that it fails.
>
> If only because if the underlying failure is ever fixed, you have no
> idea what tests need to be updated (because they are all "passing").

Yes I know what you mean.

So maybe the api should actually be:

  def test_something():
    # this isn't fully implemented yet
    self.expectFailure()
    self.assertEquals(1,2)

  def test_broken_on_windows():
    if sys.platform == 'win32':
      self.expectFailure()
    ....

And this would set a flag on the test meaning that if an assertion
fails it's considered a known failure, and if the test does fail it's
considered an unexpected pass (which is a problem).  This would catch
things being accidentally fixed.  It is a bit imprecise for long tests
where we might expect one thing to fail but actually another does.

> I suppose you could set a flag that makes "knownFailure" actually mark
> the test as failed. But if we start using "knownFailure" for more things
> (and there seems to be a trend in that direction) you will end up with
> 100 failing tests at that point. Which again makes it unclear what you
> should actually be updating.

I agree it would be a problem if there were too many or if people were
regularly merging with some of their tests still known to fail.  It is
useful in some cases like known portability problems, or wanting to
write a test for a bug that's not yet fixed.  I think we need some
guidelines as to what can be merged and what can be released.  Maybe
we should require known failures to include a bug number.

>
> John
> =:->
>
>


-- 
Martin