Another timing dependent test

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Pool wrote:
> On 4/14/07, John Arbash Meinel <john at arbash-meinel.com> wrote:
>> I'm trying to ferret out some of the random-failures for PQM tests. Most
>> of them are because of dependency on time.time() in one way or another.
>> I sent in a patch for 2 of them earlier. But I have another one I'm not
>> sure what to do with:
>>
>>
>> class SFTPLatencyKnob(TestCaseWithSFTPServer):
>>     """Test that the testing SFTPServer's latency knob works."""
>> ...
>>
>>     def test_default(self):
>>         # This test is potentially brittle: under extremely high machine
>> load
>>         # it could fail, but that is quite unlikely
>>
>> I found it... interesting that the test itself understood it could fail
>> but was "quite unlikely".
> 
> "Side effects include headaches, nausea... sudden death"
> 
> At the moment this is testing that you can get a transport in less
> than 500ms, which while desirable is not really a functional
> requirement or something we can insist on.  It might be a reasonable
> thing to have while adding the latency knob but it's not helping any
> more.
> 
> So I would ask, what kind of regression is this test trying to catch?
> Maybe that the server has a latency delay by default?  In that case I
> would just assert that the test's server has add_latency == 0.
> 
>> I know this really wants to test that the latency knob is making things
>> slower, and it doesn't make much sense to test that it is slow with the
>> knob turned up without testing that it isn't slow without the knob.
> 
> I don't think that necessarily follows...
> 

Well, there are 2 paired tests (atm).

test_default
test_latency_knob_slows_transport

The former tests that it takes <500ms, and the latter tests that when
latency is added, it takes longer than 400ms.

Without the former, you have no baseline for the latter test. The latter
test could succeed just because we are always very slow at opening a
transport object.

I can understand not wanting to test a timing side-effect.  It just
isn't clear to me how to test this specifically. I realize that
"test_latency_knob_slows_transport" is a functional (as opposed to unit)
test. But it seems like a general meaningful test to have...

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGJAgCJdeBCYSNAAMRAmeAAJwPOPXCAG3DGb/G7WQ4KqzXOgEM9wCdGv9K
0mw9ZgeN9bIqf+RsL+M+TKw=
=wGyr
-----END PGP SIGNATURE-----