Rev 5368: (spiv) Better (and simpler) algorithm for partition_tests. (Andrew Bennetts) in file:///home/pqm/archives/thelove/bzr/%2Btrunk/

[Canonical.com Patch Queue Manager]

At file:///home/pqm/archives/thelove/bzr/%2Btrunk/

------------------------------------------------------------
revno: 5368 [merge]
revision-id: pqm at pqm.ubuntu.com-20100805083007-b14pqhgoyerngrh6
parent: pqm at pqm.ubuntu.com-20100805070232-ezo69a4k078j1xmp
parent: andrew.bennetts at canonical.com-20100805054134-9jrikl1jkcs0xk5n
committer: Canonical.com Patch Queue Manager <pqm at pqm.ubuntu.com>
branch nick: +trunk
timestamp: Thu 2010-08-05 09:30:07 +0100
message:
  (spiv) Better (and simpler) algorithm for partition_tests. (Andrew Bennetts)
modified:
  NEWS                           NEWS-20050323055033-4e00b5db738777ff
  bzrlib/tests/__init__.py       selftest.py-20050531073622-8d0e3c8845c97a64
=== modified file 'NEWS'
--- a/NEWS	2010-08-03 13:31:06 +0000
+++ b/NEWS	2010-08-05 08:30:07 +0000
@@ -100,6 +100,13 @@
 Testing
 *******
 
+* The way ``bzr selftest --parallel`` generates N partitions of tests to
+  run in parallel has changed.  Instead of splitting the list of tests at
+  N-1 points, it distributes the tests one-by-one into the partitions in a
+  round robin fashion.  This reduces the total time to run the tests in
+  parallel because a series of slow tests in the test suite will be
+  distributed evenly among the parallel test suites, rather than slowing
+  down just one suite.  (Andrew Bennetts)
 
 bzr 2.2rc1
 ##########

=== modified file 'bzrlib/tests/__init__.py'
--- a/bzrlib/tests/__init__.py	2010-07-29 11:17:57 +0000
+++ b/bzrlib/tests/__init__.py	2010-08-05 05:41:34 +0000
@@ -34,6 +34,7 @@
 import difflib
 import doctest
 import errno
+import itertools
 import logging
 import math
 import os
@@ -3196,15 +3197,16 @@
 
 def partition_tests(suite, count):
     """Partition suite into count lists of tests."""
-    result = []
-    tests = list(iter_suite_tests(suite))
-    tests_per_process = int(math.ceil(float(len(tests)) / count))
-    for block in range(count):
-        low_test = block * tests_per_process
-        high_test = low_test + tests_per_process
-        process_tests = tests[low_test:high_test]
-        result.append(process_tests)
-    return result
+    # This just assigns tests in a round-robin fashion.  On one hand this
+    # splits up blocks of related tests that might run faster if they shared
+    # resources, but on the other it avoids assigning blocks of slow tests to
+    # just one partition.  So the slowest partition shouldn't be much slower
+    # than the fastest.
+    partitions = [list() for i in range(count)]
+    tests = iter_suite_tests(suite)
+    for partition, test in itertools.izip(itertools.cycle(partitions), tests):
+        partition.append(test)
+    return partitions
 
 
 def workaround_zealous_crypto_random():