Should selftest success depend on verbosity?

[Maritza Mendez]

On Sat, Jun 20, 2009 at 2:49 AM, Vincent Ladeuil
<v.ladeuil+lp at free.fr<v.ladeuil%2Blp at free.fr>
> wrote:

> Try:
>
>  bzr --no-plugins selftest -s bt.test_http
>
> That will be faster.
>

Thanks.  I've been doing that (and using --list to find my way around) as
you suggest.



> In the following, I'll use either 'bzr' or '/usr/bin/bzr'. As
> mentioned in the other thread, 'usr/bin/bzr', aka the PPA jaunty
> version is exhibiting the bugs while running from the 1.15
> sources doesn't. I still haven't understood why, I'm using it to
> reproduce the problem.
>

I will see if I can reproduce your result... runs from source but not from
packaged build.


You can try --randomize if you want to play with test order, but
> I doubt that's the root cause here.
>


Confirmed: order seems to make no difference.


> Each test has a unique ID in the test suite (that's not strictly
> enforced, but is true 99.99999% of the time, if you find an
> exception, tells us ;-).
>

Ouch.  I am assuming 100% uniqueness.  I will watch for that now.


  test_http.TestAuth.test_wrong_pass(pycurl,HTTP/1.1,basicdigest)

An implicit bzrlib.tests doesn't appear here, so the full ID is really:
>
>  bzrlib.tests.test_http.TestAuth.test_wrong_pass(pycurl,HTTP/1.1,basicdigest)
>
>

Yes.  I thought that was pretty clear.  Dropping the 'bzrlib.tests.' prefix
makes it easier to see what's going on and also works great with --exclude.


> Here, we're talking about a test method 'test_wrong_pass' in the
> TestAuth class in the bzrlib/tests/test_http.py file.
>

Yep.  I've been poking around in that exact file looking for clues.  No joy
yet.  (See other posts.)


> 'pycurl' , 'HTTP/1.1' and 'basicdigest' are parameters for the
> test, you can generally find which test object attributes receive
> these values by looking for a load_tests() method in the test
> file or at the test class definition.
>

See above.


> Knowing that, to isolate the failing test, you can try things like:
>
>  bzr --no-plugins selftest -s bt.test_http Activity.*urllib --list | wc -l
> 16
>  bzr selftest -s bt.test_http Activity.*urllib --list | wc -l
> 16
>

> 16 out of the 406: faster and more precise (we exclude pycurl)
> and no need to type --no-plugins anymore (knowing that no plugins
> can influence *these* tests is an art, I'm pretty sure no
> plugins can influence *these* tests, but who knows ;-)
>

My result is different.

 bzr --no-plugins selftest -s bt.test_http Activity.*urllib --list | wc -l

gives

16est 0/16] Starting

which I believe is the result of 'wc' and the progressbar fighting for
stdout.

So I suggest using '-v' to keep everything neat:

bzr --no-plugins selftest -s bt.test_http Activity.*urllib --list -v | wc -l

gives

19

Yes: 19, not 16.
Maybe you are using an environment variable to suppress progressbar.  And
maybe it is test 16 for your build and 19 for mine.  Anyway, the key thing
as John pointed out is to be wary of output buffered by progressbar.


Note that bt is a shortcut for bzrlib.tests, there is also bb for
> bzrlib.tests.blackbox and bp for bzrlib.plugins.
>

Nice.  Thanks!


>
> But wait
>
>  bzr --no-plugins selftest -s bt.test_http Activity.*pycurl --list | wc -l
> 0
>


Not here.  If I run the same command I get

16est 0/16] Starting

Again, the progress bar (in 1.15-1) is messing with us.

So if I run verbose

bzr --no-plugins selftest test_http --list -v | wc -l

I get

19

which is close but wrong because of the three leading lines printed by bzr
selftest.

So what we really want is

bzr --no-plugins selftest test_http --list -v | grep 'bzrlib.tests' | wc -l

which gives

16

as expected.


>
> Huh ? Where are the pycurl tests ? Looks like a bug ! Damn, I
> fixed that one, I'm sure... Anyway, unrelated to the problem at
> hand.
>
> Back to diagnosis:
>
>  /usr/bin/bzr selftest -s bzrlib.tests.test_http.TestActivity -x https -v
> tests passed
>

Same here.


>
>  /usr/bin/bzr selftest -s bzrlib.tests.test_http.TestActivity https -v
> hangs
>

 Same here.  As in my original report.


> /usr/bin/bzr selftest -s bzrlib.tests.test_http.TestActivity.test_get https
> --list
> bzrlib.tests.test_http.TestActivity.test_get(urllib,HTTP/1.0,https)
> bzrlib.tests.test_http.TestActivity.test_get(urllib,HTTP/1.1,https)
>

> Right, only two tests left (note that other TestActivity with
> https hangs too, but most probably that's the same cause).
>

Could be.  I don't know the code well enough yet to speculate about that.


>
> I'll dig that bug and the one about pycurl test missing.
>
> Using:
>
>  bzr selftest -s bt.test_http.TestActivity -x https -v
>
> passes here, so that should get you unblocked on that one.
>

That's what I've been doing, just a little less surgically.  I pick which
suites to run or exclude, but don't usually bother with -s.  Either way,
we're both doing divide and conquer I think.


> I hope these explanations are helpful, I realize there is a lot
> to learn about using selftest as a diagnosis tool (as opposed to
> using it as a validation tool when running the whole test suite)
>

Yes.  And the most helpful thing is conformation that we are both seeing
some of the same things using the PPA.

I have not repeated my testing on the source distribution.  That should be
my next step.

Thanks!  It's good to know that I'm not the only one seeing this.

-M
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/archives/bazaar/attachments/20090620/90252401/attachment.htm