Should selftest success depend on verbosity?

[Vincent Ladeuil]

>>>>> "Maritza" == Maritza Mendez <martitzam at gmail.com> writes:

    Maritza> Ok.  I don't get this.  Please someone help me understand this.
    Maritza> System is jaunty jackelope x86_64 up to date and with PPA 1.15-1

    Maritza> *Observation #1*
    Maritza> If I say

    Maritza>     bzr --no-plugins selftest test_http

Try:

  bzr --no-plugins selftest -s bt.test_http

That will be faster.

In the following, I'll use either 'bzr' or '/usr/bin/bzr'. As
mentioned in the other thread, 'usr/bin/bzr', aka the PPA jaunty
version is exhibiting the bugs while running from the 1.15
sources doesn't. I still haven't understood why, I'm using it to
reproduce the problem.


    Maritza> testing stalls at

    Maritza>     test_http.TestAuth.test_wrong_pass(pycurl,HTTP/1.1,basicdigest)

    Maritza> *Observation #2*
    Maritza> But if I say

    Maritza>     bzr --no-plugins selftest test_http -v

    Maritza> testing stalls at

    Maritza>     test_http.TestActivity.test_get(urllib,HTTP/1.0,https)


    Maritza> *Observation #3*

    Maritza> What's even more remarkable is that the
    Maritza> test_wrong_pass which stalled the non-verbose run
    Maritza> actually passes in verbose mode:

    Maritza> test_http.TestAuth.test_wrong_pass(pycurl,HTTP/1.1,basicdigest)
    Maritza> OK                  49ms

    Maritza> Each of these observations was 100% reproducible in
    Maritza> ten consecutive trials.

    Maritza> I suppose this could be explained by test order
    Maritza> (shouldn't order by the same for the commands quoted
    Maritza> above?) or by sensitivity to timing differences
    Maritza> between verbose and non-verbose modes.  But I have
    Maritza> no evidence to support this pure speculation.

You can try --randomize if you want to play with test order, but
I doubt that's the root cause here.

But more importantly:

  bzr --no-plugins selftest -s bt.test_http --list  | wc -l 
406

-s (short for --start-with) will speed up your repeated attempts,
 especially when you want to run a small subset of the test
 sure.

--list will give you only the IDs of the test without running
  them which helps understanding which tests are run under
  various options combinations or simply explore the test suite.

Each test has a unique ID in the test suite (that's not strictly
enforced, but is true 99.99999% of the time, if you find an
exception, tells us ;-).

A test ID tells you where the code is in a python way:

  test_http.TestAuth.test_wrong_pass(pycurl,HTTP/1.1,basicdigest)

An implicit bzrlib.tests doesn't appear here, so the full ID is really:

  bzrlib.tests.test_http.TestAuth.test_wrong_pass(pycurl,HTTP/1.1,basicdigest)

Here, we're talking about a test method 'test_wrong_pass' in the
TestAuth class in the bzrlib/tests/test_http.py file.

'pycurl' , 'HTTP/1.1' and 'basicdigest' are parameters for the
test, you can generally find which test object attributes receive
these values by looking for a load_tests() method in the test
file or at the test class definition.

Knowing that, to isolate the failing test, you can try things like:

  bzr --no-plugins selftest -s bt.test_http Activity.*urllib --list | wc -l
16
  bzr selftest -s bt.test_http Activity.*urllib --list | wc -l
16

16 out of the 406: faster and more precise (we exclude pycurl)
and no need to type --no-plugins anymore (knowing that no plugins
can influence *these* tests is an art, I'm pretty sure no
plugins can influence *these* tests, but who knows ;-)

Note that bt is a shortcut for bzrlib.tests, there is also bb for
bzrlib.tests.blackbox and bp for bzrlib.plugins.

But wait

  bzr --no-plugins selftest -s bt.test_http Activity.*pycurl --list | wc -l
0

Huh ? Where are the pycurl tests ? Looks like a bug ! Damn, I
fixed that one, I'm sure... Anyway, unrelated to the problem at
hand.

Back to diagnosis:

  /usr/bin/bzr selftest -s bzrlib.tests.test_http.TestActivity -x https -v
tests passed

  /usr/bin/bzr selftest -s bzrlib.tests.test_http.TestActivity https -v
hangs

/usr/bin/bzr selftest -s bzrlib.tests.test_http.TestActivity.test_get https --list
bzrlib.tests.test_http.TestActivity.test_get(urllib,HTTP/1.0,https)
bzrlib.tests.test_http.TestActivity.test_get(urllib,HTTP/1.1,https)

Right, only two tests left (note that other TestActivity with
https hangs too, but most probably that's the same cause).

I'll dig that bug and the one about pycurl test missing.

Using:

  bzr selftest -s bt.test_http.TestActivity -x https -v

passes here, so that should get you unblocked on that one.

I hope these explanations are helpful, I realize there is a lot
to learn about using selftest as a diagnosis tool (as opposed to
using it as a validation tool when running the whole test suite)
:-/

   Vincent