selftest performance: landing our code faster and developing quicker.

Robert Collins robert.collins at canonical.com
Thu Aug 27 09:28:03 BST 2009


So, while we've been frantically focused ferreting out defects in 2a,
we've let the test suite incrementally increase, to the point where it
takes substantially more than half an hour to run on my laptop.

vila can run it in 4 minutes, I believe, using fast disks and 8
processes - so say 20-30 minutes there. This affects landings in PQM
(its the driver for turning off the C check on PQM, for instance),
releasing (RM's are meant to do a test run), and general landing of
patches (if you miss something you do a PQM roundtrip, but checking the
full thing is painful [unless you have a 8-way machine]).

We have a number of things contributing to this:
 - some of our test parameterisation runs many permutations we expect to
   fail. we could fairly cheaply just not create those.
 - we have some tests that need to do expensive things
 - some of our test support code is slower than it needs to be.
 - We run our core code several hundred thousand times, and our UI code 
   only once or twice for some tests.
 - IO is slow.

Now, we can't fix all this at once, and for some tests we should be
willing to have them slow. For instance, knowing that we call 'setup.py'
with the right arguments won't tell us that the .so files actually build
in the right place or are loadable afterwards. And we can't actually
test 3 level btrees without having a 3 level btree, so its reasonable
that tests needing to be sure of that behaviour take a little longer.

Is this a problem, you may be asking? I think it is, because of what it
affects - and the heart of it is cycle time: the time to make a change
and be confident its correct.

We have a lot of test cases: 22646 in bzr.rev (+- some for local changes
I have):
Ran 22646 tests in 3136.932s

If we could get down to (say) 1ms each, we'd still be looking at 22
seconds: but that would be tolerable, for a full test run. We're
currently at 140ms (more or less). So 1ms is a very high bar to set.
However, the median is 74ms, so actually we're half way there for
half of our tests :).

I've thought of some things we can do - I'm sure there are more:
 - split out tests that we really only want to run rarely. E.g.
   bzrlib.tests.test_setup.TestSetup.test_build_and_install might be
   appropriate to run less often. OTOH its very nice to know that bzr
   doesn't build on a given platform.
 - run only tests that execute changed code. This requires a cache call
   graph that needs to be pretty fast to answer this question. I have
   a sketch on my laptop from some months back. I hope to finish this
   after 2.0
 - think hard about what you want to test when you write a test. If you
   want to test how an error looks: test that specifically, not how
   its raised or when its raised. (E.g. add a test to test_errors, not
   to bzrlib.blackbox.test_CMD).
 - profile your tests. Run bzr selftest -v <test_id>  and
   see if your test is taking a reasonable amount of time. if its more
   than 70ms, its higher than the median - and well worth spending a 
   little time considering how to bring it down. One good way to see
   what a test is doing is selftest --lsprof-tests <test_id>.
 - use test doubles. I don't mean 'Mocks' or 'Stubs' specifically (and
   they are different :P). I mean though, that you can make use of such
   tools and similar things (like MemoryTree and MemoryTransport) to
   reduce the amount of unrelated work done by your test. TreeTransform,
   today, doesn't count as a test double, because it performs IO. This
   is well worth fixing though, because as Aaron has commented
   MemoryTree is a small subset of WorkingTree and may not be
   trustworthy as a test double. (This doesn't mean that it isn't, just
   that its not known-great).
 - Do less IO. IO is slow. We go to great lengths with our lockdir and
   disk data structures to be fast, but atomic - and doing atomicish
   operations tends to make the filesystem batch up a chunk of
   operations. In ext4, I hear that many fewer temporary files will
   ever hit disk - but we'll still be serialising and deserialising all
   that data. Even so, all that data being pushed out to kernel space
   and back has an overhead, and we'll be paying it. On my laptop the
   disk light stays locked on pretty much the entire test run - once the
   tests get 5-10 seconds in.
 - Be thoughtful with locks. Bzr caches during a lock, so if your test
   doesn't need to drop and reacquire locks, you will have your test
   run faster.

I'm sure more things will come to mind as we discuss this, and I intend
to put a polished document together in the docs based on the outcome of
our discussions.

I'd like to get a ratchet going, where we draw a line in the sand after
2.0 and make sure that the test time starts to come down. 10 seconds a
day would make a dramatic difference in a pretty short time.

Attached to this are a few pages of the slowest tests, for your
entertainment. You can see how sharp the curve is.

-Rob

-------------- next part --------------
bzrlib.tests.per_repository.test_refresh_data.TestRefreshData.test_refresh_data_after_fetch_new_data_visible(RepositoryFormat5) 1.333
bzrlib.tests.test_bundle.V4_2aBundleTester.test_whitespace_bundle 1.337
bzrlib.tests.test_merge.TestMergerEntriesLCAOnDisk.test_modified_symlink 1.358
bzrlib.tests.per_transport.TransportTests.test_copy_to(HttpServer_PyCurl) 1.368
bzrlib.tests.test_merge.TestMergerEntriesLCAOnDisk.test_symlink_all_wt 1.368
bzrlib.tests.test_info.TestInfo.test_describe_tree_format 1.374
bzrlib.tests.test_missing.TestMissing.test_find_unmerged 1.386
bzrlib.tests.per_transport.TransportTests.test_copy_to(HttpServer_urllib) 1.392
bzrlib.tests.per_workingtree.test_commit.TestCommit.test_no_autodelete_alternate_renamed(WorkingTreeFormat2) 1.400
bzrlib.tests.per_repository.test_repository.TestRepository.test_attribute_text_store_basics(RepositoryFormatKnitPack6RichRoot) 1.404
bzrlib.tests.test_bundle.V4_2aBundleTester.test_binary_bundle 1.434
bzrlib.tests.blackbox.test_non_ascii.TestNonAscii.test_push(iso-8859-2) 1.444
bzrlib.tests.blackbox.test_non_ascii.TestNonAscii.test_push(cp1251) 1.452
bzrlib.tests.test_merge_core.MergeTest.test_file_moves 1.465
bzrlib.tests.per_repository.test_repository.TestRepository.test_attribute_text_store_basics(RepositoryFormatKnit1) 1.481
bzrlib.tests.test_merge.TestMergerEntriesLCAOnDisk.test_other_reverted_content_to_base 1.491
bzrlib.tests.per_repository.test_repository.TestRepository.test_attribute_revision_store_basics(RepositoryFormat2a) 1.495
bzrlib.tests.per_transport.TransportTests.test_copy_to(HTTPSServer_urllib) 1.508
bzrlib.tests.blackbox.test_cat.TestCat.test_cat_different_id 1.513
bzrlib.tests.per_repository.test_repository.TestRepository.test_attribute_text_store_basics(RepositoryFormatPackDevelopment2Subtree) 1.526
bzrlib.tests.test_merge.TestMergerEntriesLCA.test_not_in_base 1.527
bzrlib.tests.blackbox.test_non_ascii.TestNonAscii.test_push(iso-8859-1) 1.543
bzrlib.tests.per_transport.TransportTests.test_copy_to(HTTPSServer_PyCurl) 1.570
bzrlib.tests.per_transport.TransportTests.test_clone(HTTPSServer_PyCurl) 1.575
bzrlib.tests.test_merge.TestMergerEntriesLCAOnDisk.test_nested_tree_subtree_renamed_and_modified 1.597
bzrlib.tests.blackbox.test_non_ascii.TestNonAscii.test_push(utf-8,2) 1.599
bzrlib.tests.test_upgrade_stacked.TestStackUpgrade.test_stack_upgrade(1.6, 1.6.1-rich-root) 0.808
bzrlib.tests.test_upgrade_stacked.TestStackUpgrade.test_stack_upgrade(knit, 1.6) 1.643
bzrlib.tests.test_bundle.V09BundleKnit2Tester.test_bundle_empty_property_alt 1.614
bzrlib.tests.test_merge.TestMergerEntriesLCAOnDisk.test_other_deletes_lca_renames 1.645
bzrlib.tests.test_smart.TestSmartServerPackRepositoryAutopack.test_autopack_needed 1.655
bzrlib.tests.test_revisionspec.TestRevisionSpec_revno.test_as_revision_id 1.692
bzrlib.tests.test_merge_core.MergeTest.test_no_passive_add 1.715
bzrlib.tests.per_repository.test_repository.TestRepository.test_attribute_text_store_basics(RepositoryFormatKnitPack6) 1.794
bzrlib.tests.blackbox.test_push.TestPush.test_push_new_branch_stacked_uses_parent_public 1.803
bzrlib.tests.per_repository.test_fileid_involved.TestFileIdInvolved.test_fileids_altered_by_revision_ids(RepositoryFormatKnitPack5) 1.825
bzrlib.tests.per_repository.test_repository.TestRepository.test_attribute_text_store_basics(RepositoryFormat7) 1.833
bzrlib.tests.per_repository.test_repository.TestRepository.test_find_branches_using(RepositoryFormatKnitPack4) 1.869
bzrlib.tests.per_branch.test_stacking.TestStacking.test_fetch_copies_from_stacked_on(BzrBranchFormat5) 1.873
bzrlib.tests.blackbox.test_commit.TestCommit.test_unsupported_encoding_commit_message 1.889
bzrlib.tests.per_workingtree.test_commit.TestCommit.test_commit_exclude_pending_merge_fails(WorkingTreeFormat6) 1.895
bzrlib.tests.test_merge.TestMergerEntriesLCAOnDisk.test_executable_changes 1.899
bzrlib.tests.per_transport.TransportTests.test_has(HTTPSServer_urllib) 1.921
bzrlib.tests.test_sftp_transport.SFTPLatencyKnob.test_latency_knob_slows_transport 1.973
bzrlib.tests.test_merge_core.MergeTest.test_contents_merge 2.076
bzrlib.tests.test_merge_core.MergeTest.test_contents_merge3 2.100
bzrlib.tests.test_merge.TestMergerEntriesLCAOnDisk.test_all_wt 2.110
bzrlib.tests.test_merge_directive.TestMergeDirective2Branch.test_generate_patch 2.118
bzrlib.tests.blackbox.test_non_ascii.TestNonAscii.test_push(utf-8,1) 2.129
bzrlib.tests.blackbox.test_diff.TestCheckoutDiff.test_diff_p1 2.150
bzrlib.tests.test_merge_core.MergeTest.test_change_name 2.175
bzrlib.tests.test_repository.Test2a.test_get_stream_for_missing_keys_includes_all_chk_refs 2.331
bzrlib.tests.test_info.TestInfo.test_describe_checkout_format 2.405
bzrlib.tests.blackbox.test_cat.TestCat.test_cat 2.513
bzrlib.tests.test_merge_core.MergeTest.test_contents_merge2 2.587
bzrlib.tests.test__groupcompress.TestMakeAndApplyDelta.test_make_delta_with_large_copies(python) 2.594
bzrlib.tests.per_transport.TransportTests.test_has(HTTPSServer_PyCurl) 2.685
bzrlib.tests.test_merge_core.FunctionalMergeTest.test_merge_swapping_renames 2.744
bzrlib.tests.test_transform.TestBuildTree.test_directory_conflict_handling 2.878
bzrlib.tests.test_merge_core.FunctionalMergeTest.test_merge_unrelated 2.902
bzrlib.tests.per_tree.test_path_content_summary.TestPathContentSummary.test_file_content_summary_non_exec(WorkingTreeFormat3) 2.903
bzrlib.tests.per_workingtree.test_executable.TestExecutable.test_05_removed_and_committed(WorkingTreeFormat3) 3.004
bzrlib.tests.per_repository.test_repository.TestRepository.test_attribute_text_store_basics(RepositoryFormat5) 3.304
bzrlib.tests.per_workingtree.test_executable.TestExecutable.test_05_removed_and_committed(WorkingTreeFormat2) 3.457
bzrlib.tests.test_bundle.V4_2aBundleTester.test_bundle 3.697
bzrlib.tests.per_tree.test_test_trees.TestTreeShapes.test_abc_tree_content_4_no_parents(DirStateRevisionTree,WT5) 3.784
bzrlib.tests.per_tree.test_inv.TestInventory.test_canonical_path_root(WorkingTreeFormat3) 3.897
bzrlib.tests.test_bundle.V4BundleTester.test_bundle 3.909
bzrlib.tests.per_branch.test_stacking.TestStacking.test_clone_stacking_policy_handling(BzrBranchFormat6) 4.025
bzrlib.tests.per_repository.test_iter_reverse_revision_history.TestIterReverseRevisionHistory.test_ghost(RemoteRepositoryFormat-v2) 4.089
bzrlib.tests.test_shelf.TestPrepareShelf.test_shelve_rename 4.106
bzrlib.tests.test_workingtree_4.TestWorkingTreeFormat4.test_observed_sha1_cachable 4.142
bzrlib.tests.test_workingtree_4.TestWorkingTreeFormat4.test_commit_updates_hash_cache 4.177
bzrlib.tests.test_bundle.V09BundleKnit1Tester.test_bundle 4.300
bzrlib.tests.test_btree_index.TestBTreeIndex.test_iter_all_entries_reads 4.301
bzrlib.tests.test_hashcache.TestHashCache.test_hashcache_load 5.051
bzrlib.tests.per_tree.test_get_root_id.TestGetRootID.test_get_root_id_fixed(WorkingTreeFormat4) 5.067
bzrlib.tests.test_bzrdir.TestRepositoryAcquisitionPolicy.test_sprout_obeys_stacking_policy 5.077
bzrlib.tests.per_repository.test_iter_reverse_revision_history.TestIterReverseRevisionHistory.test_ghost(RepositoryFormat6) 5.122
bzrlib.tests.test_read_bundle.TestReadBundleFromURL.test_read_mergeable_respects_possible_transports(FakeNFSServer) 5.338
bzrlib.tests.test_inv.TestDeltaApplication.test_mismatched_new_path_None_entry(WorkingTreeFormat4.update_basis_by_delta) 5.423
bzrlib.tests.per_transport.TransportTests.test_readv_with_adjust_for_latency(HTTPSServer_PyCurl) 5.513
bzrlib.tests.test_bundle.V09BundleKnit2Tester.test_bundle 5.842
bzrlib.tests.test_repository.Test2a.test_autopack_unchanged_chk_nodes 5.958
bzrlib.tests.test_btree_index.TestBTreeBuilder.test_three_level_tree_details 5.964
bzrlib.tests.test_selftest.TestActuallyStartBzrSubprocess.test_start_and_stop_bzr_subprocess_send_signal 6.064
bzrlib.tests.test_bundle.V4WeaveBundleTester.test_bundle 6.195
bzrlib.tests.test_version_info.TestVersionInfo.test_rio_version 6.279
bzrlib.tests.test_read_bundle.TestReadBundleFromURL.test_read_mergeable_respects_possible_transports(SFTPAbsoluteServer) 6.454
bzrlib.tests.test_smart.TestSmartServerRepositoryGetRevIdForRevno.test_revno_found 6.567
bzrlib.tests.test_upgrade.TestUpgrade.test_upgrade_makes_dir_weaves 7.552
bzrlib.tests.test_bundle.V4WeaveBundleTester.test_symlink_bundle 7.989
bzrlib.tests.per_repository.test_repository.TestRepository.test_attribute_revision_store_basics(RepositoryFormat5) 8.614
bzrlib.tests.blackbox.test_breakin.TestBreakin.test_breakin 12.661
bzrlib.tests.test_sftp_transport.SSHVendorBadConnection.test_bad_connection_ssh 12.725
bzrlib.tests.test_bundle.V08BundleTester.test_bundle 14.400
bzrlib.tests.test_hashcache.TestHashCache.test_hammer_hashcache 27.061
bzrlib.tests.test_source.TestSource.test_coding_style 37.319
bzrlib.tests.test_source.TestSource.test_no_asserts 53.900
bzrlib.tests.test_setup.TestSetup.test_build_and_install 199.637
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20090827/59cbfdd1/attachment-0002.pgp 


More information about the bazaar mailing list