[MERGE] Speedup selftest by loading a previously saved test list

[Vincent Ladeuil]

>>>>> "robert" == Robert Collins <robertc at robertcollins.net> writes:

    robert> On Fri, 2008-01-11 at 00:16 +0100, Vincent Ladeuil wrote:
    >> This patch add two options to the selftest command:
    >> 
    >> --save-list <file> 
    >> --load-list <file>
    >> 
    >> You use the first one to define a set of tests you want to run
    >> repeatedly and use the second one to avoid loading the whole test
    >> suite:
    robert> ...
    >> - test inheriting from a class defined inside a function (or
    >> using a lambda as an attribute) can't be pickled either (but
    >> see the patch for some obvious fixes in the bzr core tests),
    >> 
    >> But overall I think more than 99% of tests can be used with this
    >> mechanism.

    robert> I feel pretty uncomfortable about this.

What I've tested so far: 99.76% of our core test suite are
serialized reliably (10530 out of 10561 tests) see below.

    robert> You have to assume a great deal to be sure that tests
    robert> will execute the correct test when you are pickling
    robert> their state.

Serialized before setUp is run. I assume that each test is
initialized by setUp and setUp only regarding its state.

    robert> I'd rather not see this come in, it feels basically
    robert> wrong. You should be able to get nearly the same
    robert> performance by saving just the names and asking for
    robert> just those tests.

Been there, thought about that, can't be used for parametrized
tests (~6000 out of the ~10000 existing tests) which requires
some serialization for parameters anyway.

,----
| ./bzr selftest --no-plugins --save-list all_tests
| <...>
| Ran 10561 tests in 482.516s
| 
| OK (known_failures=15)
| 666 tests skipped
`----

31 tests can't be serialized reliably and are excluded automatically.

,----
| ./bzr selftest --no-plugins --load-list all_tests
| | <...>
| Ran 10530 tests in 487.059s
| 
| FAILED (failures=2, known_failure_count=15)
| 666 tests skipped
`----

Both failures are in
tests/bzrdir_implementations.TestBzrDir.test_format_initialize_find_open

Skipping the details, this can be fixed by:

=== modified file 'bzrlib/bzrdir.py'
--- bzrlib/bzrdir.py	2008-01-09 00:51:01 +0000
+++ bzrlib/bzrdir.py	2008-01-16 12:00:41 +0000
@@ -1670,6 +1670,13 @@
 
     _lock_class = lockable_files.TransportLock
 
+    def __eq__(self, other):
+        if other.__class__ is not self.__class__:
+            return False
+        if other.repository_format != self.repository_format:
+            return False
+        return True
+
     def get_format_string(self):
         """See BzrDirFormat.get_format_string()."""
         return "Bazaar-NG branch, format 5\n"
@@ -1729,6 +1736,13 @@
 
     _lock_class = lockable_files.TransportLock
 
+    def __eq__(self, other):
+        if other.__class__ is not self.__class__:
+            return False
+        if other.repository_format != self.repository_format:
+            return False
+        return True
+
     def get_format_string(self):
         """See BzrDirFormat.get_format_string()."""
         return "Bazaar-NG branch, format 6\n"

So, with the above patch applied:
,----
| ./bzr selftest  --load-list all_tests --no-plugins
| Ran 10530 tests in 487.052s
| 
| OK (known_failures=15)
| 666 tests skipped
`----

So 99.76% of our core test suite are serialized reliably (10530
out of 10561 tests).

,----
| time -p ./bzr selftest rocks --save-list  rocks
| <...>
| Ran 1 test in 0.008s
| 
| OK
| tests passed
| real 3.09
| user 0.69
| sys 1.75
`----

,----
| time -p ./bzr selftest rocks --load-list  rocks
| <...>
| Ran 1 test in 0.122s
| 
| OK
| tests passed
| real 0.43
| user 0.02
| sys 0.25
`----

2.5 seconds out of 3 shaved on a pretty fast machine.

,----
| time -p ./bzr selftest rocks --save-list  rocks
| <...>
| Ran 1 test in 0.006s
| 
| OK
| tests passed
| real 8.50
| user 7.73
| sys 0.43
`----

,----
| time -p ./bzr selftest rocks --load-list  rocks
| <...>
| Ran 1 test in 0.210s
| 
| OK
| tests passed
| real 0.88
| user 0.46
| sys 0.19
`----

7 seconds out of 8.5 shaved on a less fast one.

    robert> bb:reject

Still ???

      Vincent