Generic selftest questions

[Vincent Ladeuil]

>>>>> "Maritza" == Maritza Mendez <martitzam at gmail.com> writes:

    Maritza> Thanks Martin.  My plan for windows is to do clean
    Maritza> production installs on three or four different
    Maritza> machines and cross-correlate selftest logs to see if
    Maritza> anything stands out over multiple runs and focus on
    Maritza> those.

    Maritza> Not as ambitious as Vincent's plan but someplace to
    Maritza> start.

Scratch ambitious :) I think we have roughly the same intent:
ensures full selftest is passing, one platform at a time.

On the other hand, if you're willing to dedicate a small amount
of daily CPU time, I'll be happy to work with you on setting up a
builbot slave, contact me privately if you're interested.

    Maritza> Fyi I spent some time reading http.test and I do not
    Maritza> see anything that shouts concern.  

http hanging bugs (or any test server hanging bugs) are the
hardest ones I had to track, at least here they are reproducible
:-/

But that's certainly not the easiest task to begin with (I don't
intend to frighten you here, *I* like hard bugs, just not
everyday ;-).

    Maritza> But I'm just getting started.

Thank you for that.

    >>> 
    >>> Since I am seeing issues (which I log at launchpad when I
    >>> have enough info) which I do not understand, I am running
    >>> 'bzr --no-plugins selftest' on clean installs of the
    >>> latest packaged builds (PPA for Ubuntu and Windows
    >>> default installer for WinXP).

That's a good idea. It will give us a starting point.

    >>>   And I am finding failures so I've decided I need to
    >>> start from clean installs.  The issues I am seeing are
    >>> too many to dump on the mailing list and too poorly
    >>> understood by me to file good bug reports.

You can just file a bug and attach the log files.

    >>> Some of the issues may be silly things like not having a
    >>> port open for bzr server.  Both januty and WinXP running
    >>> bzr 1.15-1 (both with python 2.6) will finish 'bzr
    >>> --no-plugins selftest'.



    >>>   On linux, I get the hang in test_http reported here a
    >>> couple days ago. 

I have yet to look at that one more deeply :-/

    >>> On WinXP, I get "80 err, 21 fail, 6 missing" and
    >>> ultimately a hang on a different part of test_http
    >>> (TestBadProtocol).
    >>> 
    >>> Is there a log of known failures by platform?

No. But I wonder if having the ability to provide such a list to
selftest may help here... That will at least gives a way to
monitor progress / regression better than no failures /
christmas-tree failures.

    >> There is not a separate log.  The goal is that on every
    >> platform, every test will either pass or at least give a
    >> known failure.  So if you identify that a particular test
    >> always fails on Windows, the best option is probably to
    >> file a bug for it and submit a patch that turns it into a
    >> KnownFailure pointing to that bug.


Or if you can establish that a set of tests share some common
failure mode, that will be even more valuable ;-D

<snip/>

        Vincent

[1]: OS locks being the exception rather than the rule, but they
must die anyway ;-D

[2]: And I'm not throwing stones to anybody here, it takes times
to run the test suite regularly on any platform and doing so at
release time or near release time is not enough, and once you
lose the full passing state, it's hard to get it back.