[MERGE] timing sensitive test can cause spurious failures

[John Arbash Meinel]

Andrew Bennetts wrote:
> I just got a spurious failure:
> 
> ======================================================================
> FAIL: test_time_creates_benchmark_in_result (bzrlib.tests.test_selftest.TestTestCase)
> 
> vvvv[log from bzrlib.tests.test_selftest.TestTestCase.test_time_creates_benchmark_in_result]
> 
> ^^^^[log from bzrlib.tests.test_selftest.TestTestCase.test_time_creates_benchmark_in_result]
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File ".../bzrlib/tests/test_selftest.py", line 795, in test_time_creates_benchmark_in_result
>     "[1-9][0-9]ms/   [1-9][0-9]ms\n$")
>   File ".../bzrlib/tests/__init__.py", line 540, in assertContainsRe
>     raise AssertionError('pattern "%s" not found in "%s"'
> AssertionError: pattern "[1-9][0-9]ms/   [1-9][0-9]ms
> $" not found in "test_selftest.TestTestCase.method_that_times_a_bit_twice                                                               OK    14ms/  711ms
> "
> ======================================================================
> 
> Timing-sensitive tests suck.  They will always break for someone somewhere,
> because there's always a system that's faster or slower than you ever expected.
> 
> Also, it's probably a good habit to think of time.sleep as a low-resolution
> hint, rather than something to be trusted.
> 

Well the test under consideration should actually be taking ~1ms. It is
something *really* simple. We've already bumped it from being <10ms to
being <100ms. We discussed it, and we don't really want to go above
that. Robert explicitly didn't want it to be too loose, and miss a real bug.

You machine is taking 100 times too long to respond to a simple
time.sleep() call. It might be better to use a loop, or something like
that, which should take a little while, but maybe be less sensitive than
time.sleep().

John
=:->

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060830/e8ccbd28/attachment.pgp