Subunit stream considers unexpected success as a success?

[Jonathan Lange]

On Sat, Sep 25, 2010 at 7:48 PM, John Arbash Meinel
<john at arbash-meinel.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I've been working on the log filtering code, and came across this artifact.
>
> Specifically, I have a test run that results in "UnexpectedSuccess" with:
>
>        def test_unexpected_success(self):
>            mutter('test with unexpected success')
>            self.expectFailure('should_fail', lambda: None)
>
> When I run this locally, I get
>
>  result.unexpectedSuccesses = [test]
>
> However if I run it through the SubUnit stream, I end up with:
> (Pdb) print content
> progress: 1
> time: 2010-09-25 18:42:02.322000Z
> test: bzrlib.tests.test_selftest.ExampleTests.test_unexpected_success
> time: 2010-09-25 18:42:02.322000Z
> successful:
> bzrlib.tests.test_selftest.ExampleTests.test_unexpected_success [ multipart
> Content-Type: text/plain;charset=utf8
> log
> 24
> 3.213  test with unexpected success
> 0
> Content-Type: text/plain
> reason
> B
> should_fail0
> ]
> time: 2010-09-25 18:42:02.322000Z
>
>
> Which has the 'reason' entry, indicating that the test failed, but it is
> considered a "successful: " test.
>
> For now, I'm just going to skip testing this, since it seems broken, but
> I was hoping to verify.
>


Testtools has code that maps addUnexpectedSuccess to addSuccess for
older TestResults.

Subunit's TestProtocolClient (the bit that serializes test outcomes)
has this line:
      addUnexpectedSuccess = addSuccess

Either of these could be causing your problem, but my money is on the second.

I'm not sure the subunit protocol even has a way of encoding unexpected success.

jml