Rev 4319: (robertc) Many less round trips on bzr push to a smart server. in file:///home/pqm/archives/thelove/bzr/%2Btrunk/
Canonical.com Patch Queue Manager
pqm at pqm.ubuntu.com
Fri May 1 08:33:14 BST 2009
At file:///home/pqm/archives/thelove/bzr/%2Btrunk/
------------------------------------------------------------
revno: 4319
revision-id: pqm at pqm.ubuntu.com-20090501073309-ysq34enbsw9avhyw
parent: pqm at pqm.ubuntu.com-20090501053220-6zjnfoxj3t7lme0o
parent: robertc at robertcollins.net-20090501063612-1898qo17eimybm7w
committer: Canonical.com Patch Queue Manager <pqm at pqm.ubuntu.com>
branch nick: +trunk
timestamp: Fri 2009-05-01 08:33:09 +0100
message:
(robertc) Many less round trips on bzr push to a smart server.
(Robert Collins)
modified:
bzrlib/branch.py branch.py-20050309040759-e4baf4e0d046576e
bzrlib/bzrdir.py bzrdir.py-20060131065624-156dfea39c4387cb
bzrlib/fetch.py fetch.py-20050818234941-26fea6105696365d
bzrlib/remote.py remote.py-20060720103555-yeeg2x51vn0rbtdp-1
bzrlib/repository.py rev_storage.py-20051111201905-119e9401e46257e3
bzrlib/smart/bzrdir.py bzrdir.py-20061122024551-ol0l0o0oofsu9b3t-1
bzrlib/tests/blackbox/test_push.py test_push.py-20060329002750-929af230d5d22663
bzrlib/tests/bzrdir_implementations/test_bzrdir.py test_bzrdir.py-20060131065642-0ebeca5e30e30866
bzrlib/tests/test_fetch.py testfetch.py-20050825090644-f73e07e7dfb1765a
bzrlib/tests/test_smart.py test_smart.py-20061122024551-ol0l0o0oofsu9b3t-2
------------------------------------------------------------
revno: 4307.2.6
revision-id: robertc at robertcollins.net-20090501063612-1898qo17eimybm7w
parent: robertc at robertcollins.net-20090429041426-91s1jby1uq6mac1l
committer: Robert Collins <robertc at robertcollins.net>
branch nick: push.roundtrips
timestamp: Fri 2009-05-01 16:36:12 +1000
message:
Handle repositories that mutex on writes (rather than transactions).
modified:
bzrlib/bzrdir.py bzrdir.py-20060131065624-156dfea39c4387cb
------------------------------------------------------------
revno: 4307.2.5
revision-id: robertc at robertcollins.net-20090429041426-91s1jby1uq6mac1l
parent: robertc at robertcollins.net-20090428065329-vx84wza4w6l979ai
committer: Robert Collins <robertc at robertcollins.net>
branch nick: push.roundtrips
timestamp: Wed 2009-04-29 14:14:26 +1000
message:
Remove too-early checks for revisions adding unnecessary round trips, at the cost of actually reading revision data when pulling (because we currently don't have a hint as about whats local for fetch).
modified:
bzrlib/fetch.py fetch.py-20050818234941-26fea6105696365d
bzrlib/repository.py rev_storage.py-20051111201905-119e9401e46257e3
bzrlib/tests/blackbox/test_push.py test_push.py-20060329002750-929af230d5d22663
bzrlib/tests/test_fetch.py testfetch.py-20050825090644-f73e07e7dfb1765a
------------------------------------------------------------
revno: 4307.2.4
revision-id: robertc at robertcollins.net-20090428065329-vx84wza4w6l979ai
parent: robertc at robertcollins.net-20090428052904-sttkmasfldpxlq2m
committer: Robert Collins <robertc at robertcollins.net>
branch nick: push.roundtrips
timestamp: Tue 2009-04-28 16:53:29 +1000
message:
Enable caching of negative revision lookups in RemoteRepository write locks when no _real_repository has been constructed.
modified:
bzrlib/remote.py remote.py-20060720103555-yeeg2x51vn0rbtdp-1
bzrlib/tests/blackbox/test_push.py test_push.py-20060329002750-929af230d5d22663
------------------------------------------------------------
revno: 4307.2.3
revision-id: robertc at robertcollins.net-20090428052904-sttkmasfldpxlq2m
parent: robertc at robertcollins.net-20090428035556-z6ucu9363itcb9qx
committer: Robert Collins <robertc at robertcollins.net>
branch nick: push.roundtrips
timestamp: Tue 2009-04-28 15:29:04 +1000
message:
Change RemoteRepository.has_revision to use get_parent_map to leverage the caching.
modified:
bzrlib/remote.py remote.py-20060720103555-yeeg2x51vn0rbtdp-1
------------------------------------------------------------
revno: 4307.2.2
revision-id: robertc at robertcollins.net-20090428035556-z6ucu9363itcb9qx
parent: robertc at robertcollins.net-20090428015132-a9n7dqmbm03w5nlh
committer: Robert Collins <robertc at robertcollins.net>
branch nick: push.roundtrips
timestamp: Tue 2009-04-28 13:55:56 +1000
message:
Lock repositories created by BzrDirFormat.initialize_on_transport_ex.
modified:
bzrlib/branch.py branch.py-20050309040759-e4baf4e0d046576e
bzrlib/bzrdir.py bzrdir.py-20060131065624-156dfea39c4387cb
bzrlib/smart/bzrdir.py bzrdir.py-20061122024551-ol0l0o0oofsu9b3t-1
bzrlib/tests/blackbox/test_push.py test_push.py-20060329002750-929af230d5d22663
bzrlib/tests/bzrdir_implementations/test_bzrdir.py test_bzrdir.py-20060131065642-0ebeca5e30e30866
bzrlib/tests/test_smart.py test_smart.py-20061122024551-ol0l0o0oofsu9b3t-2
------------------------------------------------------------
revno: 4307.2.1
revision-id: robertc at robertcollins.net-20090428015132-a9n7dqmbm03w5nlh
parent: pqm at pqm.ubuntu.com-20090428004234-6j7ndsmvsx3hsrqo
committer: Robert Collins <robertc at robertcollins.net>
branch nick: push.roundtrips
timestamp: Tue 2009-04-28 11:51:32 +1000
message:
Don't probe for bzrdir objects we just created via the smart server.
modified:
bzrlib/bzrdir.py bzrdir.py-20060131065624-156dfea39c4387cb
bzrlib/tests/blackbox/test_push.py test_push.py-20060329002750-929af230d5d22663
=== modified file 'bzrlib/branch.py'
--- a/bzrlib/branch.py 2009-04-27 03:47:55 +0000
+++ b/bzrlib/branch.py 2009-04-28 03:55:56 +0000
@@ -1397,7 +1397,14 @@
control_files = lockable_files.LockableFiles(branch_transport,
lock_name, lock_class)
control_files.create_lock()
- control_files.lock_write()
+ try:
+ control_files.lock_write()
+ except errors.LockContention:
+ if lock_type != 'branch4':
+ raise
+ lock_taken = False
+ else:
+ lock_taken = True
if set_format:
utf8_files += [('format', self.get_format_string())]
try:
@@ -1406,7 +1413,8 @@
filename, content,
mode=a_bzrdir._get_file_mode())
finally:
- control_files.unlock()
+ if lock_taken:
+ control_files.unlock()
return self.open(a_bzrdir, _found=True)
def initialize(self, a_bzrdir):
=== modified file 'bzrlib/bzrdir.py'
--- a/bzrlib/bzrdir.py 2009-04-27 05:56:16 +0000
+++ b/bzrlib/bzrdir.py 2009-05-01 06:36:12 +0000
@@ -246,22 +246,26 @@
repo_format_name=repo_format_name,
make_working_trees=make_working_trees, shared_repo=want_shared)
if repo_format_name:
- # If the result repository is in the same place as the resulting
- # bzr dir, it will have no content, further if the result is not stacked
- # then we know all content should be copied, and finally if we are
- # copying up to a specific revision_id then we can use the
- # pending-ancestry-result which does not require traversing all of
- # history to describe it.
- if (result_repo.bzrdir.root_transport.base ==
- result.root_transport.base and not require_stacking and
- revision_id is not None):
- fetch_spec = graph.PendingAncestryResult(
- [revision_id], local_repo)
- result_repo.fetch(local_repo, fetch_spec=fetch_spec)
- else:
- result_repo.fetch(local_repo, revision_id=revision_id)
+ try:
+ # If the result repository is in the same place as the
+ # resulting bzr dir, it will have no content, further if the
+ # result is not stacked then we know all content should be
+ # copied, and finally if we are copying up to a specific
+ # revision_id then we can use the pending-ancestry-result which
+ # does not require traversing all of history to describe it.
+ if (result_repo.bzrdir.root_transport.base ==
+ result.root_transport.base and not require_stacking and
+ revision_id is not None):
+ fetch_spec = graph.PendingAncestryResult(
+ [revision_id], local_repo)
+ result_repo.fetch(local_repo, fetch_spec=fetch_spec)
+ else:
+ result_repo.fetch(local_repo, revision_id=revision_id)
+ finally:
+ result_repo.unlock()
else:
- result_repo = None
+ if result_repo is not None:
+ raise AssertionError('result_repo not None(%r)' % result_repo)
# 1 if there is a branch present
# make sure its content is available in the target repository
# clone it.
@@ -1946,6 +1950,7 @@
if not require_stacking and repository_policy._require_stacking:
require_stacking = True
result._format.require_stacking()
+ result_repo.lock_write()
else:
result_repo = None
repository_policy = None
@@ -3149,7 +3154,7 @@
format = RemoteBzrDirFormat()
format._network_name = bzrdir_name
self._supply_sub_formats_to(format)
- bzrdir = remote.RemoteBzrDir(transport, format)
+ bzrdir = remote.RemoteBzrDir(transport, format, _client=client)
if repo_path:
repo_format = remote.response_tuple_to_repo_format(response[1:])
if repo_path == '.':
@@ -3164,6 +3169,14 @@
final_stack = response[8] or None
final_stack_pwd = response[9] or None
remote_repo = remote.RemoteRepository(repo_bzr, repo_format)
+ if len(response) > 10:
+ # Updated server verb that locks remotely.
+ repo_lock_token = response[10] or None
+ remote_repo.lock_write(repo_lock_token, _skip_rpc=True)
+ if repo_lock_token:
+ remote_repo.dont_leave_lock_in_place()
+ else:
+ remote_repo.lock_write()
policy = UseExistingRepository(remote_repo, final_stack,
final_stack_pwd, require_stacking)
policy.acquire_repository()
=== modified file 'bzrlib/fetch.py'
--- a/bzrlib/fetch.py 2009-04-27 23:14:00 +0000
+++ b/bzrlib/fetch.py 2009-04-29 04:14:26 +0000
@@ -170,9 +170,6 @@
if self._last_revision is NULL_REVISION:
# explicit limit of no revisions needed
return None
- if (self._last_revision is not None and
- self.to_repository.has_revision(self._last_revision)):
- return None
try:
return self.to_repository.search_missing_revision_ids(
self.from_repository, self._last_revision,
=== modified file 'bzrlib/remote.py'
--- a/bzrlib/remote.py 2009-04-27 23:14:00 +0000
+++ b/bzrlib/remote.py 2009-04-28 06:53:29 +0000
@@ -30,6 +30,7 @@
pack,
repository,
revision,
+ revision as _mod_revision,
symbol_versioning,
urlutils,
)
@@ -604,6 +605,8 @@
self._lock_token = None
self._lock_count = 0
self._leave_lock = False
+ # Cache of revision parents; misses are cached during read locks, and
+ # write locks when no _real_repository has been set.
self._unstacked_provider = graph.CachingParentsProvider(
get_parent_map=self._get_parent_map_rpc)
self._unstacked_provider.disable_cache()
@@ -685,6 +688,7 @@
invocation. If in doubt chat to the bzr network team.
"""
if self._real_repository is None:
+ self._unstacked_provider.missing_keys.clear()
self.bzrdir._ensure_real()
self._set_real_repository(
self.bzrdir._real_bzrdir.open_repository())
@@ -750,30 +754,24 @@
"""Return a source for streaming from this repository."""
return RemoteStreamSource(self, to_format)
+ @needs_read_lock
def has_revision(self, revision_id):
- """See Repository.has_revision()."""
- if revision_id == NULL_REVISION:
- # The null revision is always present.
- return True
- path = self.bzrdir._path_for_remote_call(self._client)
- response = self._call('Repository.has_revision', path, revision_id)
- if response[0] not in ('yes', 'no'):
- raise errors.UnexpectedSmartServerResponse(response)
- if response[0] == 'yes':
- return True
- for fallback_repo in self._fallback_repositories:
- if fallback_repo.has_revision(revision_id):
- return True
- return False
+ """True if this repository has a copy of the revision."""
+ # Copy of bzrlib.repository.Repository.has_revision
+ return revision_id in self.has_revisions((revision_id,))
+ @needs_read_lock
def has_revisions(self, revision_ids):
- """See Repository.has_revisions()."""
- # FIXME: This does many roundtrips, particularly when there are
- # fallback repositories. -- mbp 20080905
- result = set()
- for revision_id in revision_ids:
- if self.has_revision(revision_id):
- result.add(revision_id)
+ """Probe to find out the presence of multiple revisions.
+
+ :param revision_ids: An iterable of revision_ids.
+ :return: A set of the revision_ids that were present.
+ """
+ # Copy of bzrlib.repository.Repository.has_revisions
+ parent_map = self.get_parent_map(revision_ids)
+ result = set(parent_map)
+ if _mod_revision.NULL_REVISION in revision_ids:
+ result.add(_mod_revision.NULL_REVISION)
return result
def has_same_location(self, other):
@@ -897,7 +895,8 @@
self._leave_lock = False
self._lock_mode = 'w'
self._lock_count = 1
- self._unstacked_provider.enable_cache(cache_misses=False)
+ cache_misses = self._real_repository is None
+ self._unstacked_provider.enable_cache(cache_misses=cache_misses)
elif self._lock_mode == 'r':
raise errors.ReadOnlyError(self)
else:
@@ -1610,6 +1609,7 @@
def insert_stream(self, stream, src_format, resume_tokens):
target = self.target_repo
+ target._unstacked_provider.missing_keys.clear()
if target._lock_token:
verb = 'Repository.insert_stream_locked'
extra_args = (target._lock_token or '',)
=== modified file 'bzrlib/repository.py'
--- a/bzrlib/repository.py 2009-04-27 23:14:00 +0000
+++ b/bzrlib/repository.py 2009-04-29 04:14:26 +0000
@@ -3092,13 +3092,6 @@
"""
target_graph = self.target.get_graph()
revision_ids = frozenset(revision_ids)
- # Fast path for the case where all the revisions are already in the
- # target repo.
- # (Although this does incur an extra round trip for the
- # fairly common case where the target doesn't already have the revision
- # we're pushing.)
- if set(target_graph.get_parent_map(revision_ids)) == revision_ids:
- return graph.SearchResult(revision_ids, set(), 0, set())
missing_revs = set()
source_graph = self.source.get_graph()
# ensure we don't pay silly lookup costs.
=== modified file 'bzrlib/smart/bzrdir.py'
--- a/bzrlib/smart/bzrdir.py 2009-04-24 05:08:51 +0000
+++ b/bzrlib/smart/bzrdir.py 2009-04-28 03:55:56 +0000
@@ -356,7 +356,11 @@
make_working_trees, shared_repo):
"""Initialize a bzrdir at path as per BzrDirFormat.initialize_ex
- :return: SmartServerResponse()
+ :return: return SuccessfulSmartServerResponse((repo_path, rich_root,
+ tree_ref, external_lookup, repo_network_name,
+ repo_bzrdir_network_name, bzrdir_format_network_name,
+ NoneTrueFalse(stacking), final_stack, final_stack_pwd,
+ repo_lock_token))
"""
target_transport = self.transport_from_client_path(path)
format = network_format_registry.get(bzrdir_network_name)
@@ -383,6 +387,7 @@
repo_bzrdir_name = ''
final_stack = None
final_stack_pwd = None
+ repo_lock_token = ''
else:
repo_path = self._repo_relpath(bzrdir.root_transport, repo)
if repo_path == '':
@@ -393,13 +398,20 @@
repo_bzrdir_name = repo.bzrdir._format.network_name()
final_stack = repository_policy._stack_on
final_stack_pwd = repository_policy._stack_on_pwd
+ # It is returned locked, but we need to do the lock to get the lock
+ # token.
+ repo.unlock()
+ repo_lock_token = repo.lock_write() or ''
+ if repo_lock_token:
+ repo.leave_lock_in_place()
+ repo.unlock()
final_stack = final_stack or ''
final_stack_pwd = final_stack_pwd or ''
return SuccessfulSmartServerResponse((repo_path, rich_root, tree_ref,
external_lookup, repo_name, repo_bzrdir_name,
bzrdir._format.network_name(),
self._serialize_NoneTrueFalse(stacking), final_stack,
- final_stack_pwd))
+ final_stack_pwd, repo_lock_token))
class SmartServerRequestOpenBranch(SmartServerRequestBzrDir):
=== modified file 'bzrlib/tests/blackbox/test_push.py'
--- a/bzrlib/tests/blackbox/test_push.py 2009-04-27 03:47:55 +0000
+++ b/bzrlib/tests/blackbox/test_push.py 2009-04-29 04:14:26 +0000
@@ -201,7 +201,7 @@
# being too low. If rpc_count increases, more network roundtrips have
# become necessary for this use case. Please do not adjust this number
# upwards without agreement from bzr's network support maintainers.
- self.assertLength(11, self.hpss_calls)
+ self.assertLength(9, self.hpss_calls)
def test_push_smart_stacked_streaming_acceptance(self):
self.setup_smart_server_with_call_log()
@@ -217,7 +217,7 @@
# being too low. If rpc_count increases, more network roundtrips have
# become necessary for this use case. Please do not adjust this number
# upwards without agreement from bzr's network support maintainers.
- self.assertLength(18, self.hpss_calls)
+ self.assertLength(14, self.hpss_calls)
remote = Branch.open('public')
self.assertEndsWith(remote.get_stacked_on_url(), '/parent')
=== modified file 'bzrlib/tests/bzrdir_implementations/test_bzrdir.py'
--- a/bzrlib/tests/bzrdir_implementations/test_bzrdir.py 2009-04-27 03:34:12 +0000
+++ b/bzrlib/tests/bzrdir_implementations/test_bzrdir.py 2009-04-28 03:55:56 +0000
@@ -1207,6 +1207,9 @@
return
self.assertNotEqual(repo.bzrdir.root_transport.base,
made_repo.bzrdir.root_transport.base)
+ # New repositories are write locked.
+ self.assertTrue(made_repo.is_write_locked())
+ made_repo.unlock()
def test_format_initialize_on_transport_ex_force_new_repo_False(self):
t = self.get_transport('repo')
@@ -1239,6 +1242,9 @@
# uninitialisable format
return
self.assertLength(1, repo._fallback_repositories)
+ # New repositories are write locked.
+ self.assertTrue(repo.is_write_locked())
+ repo.unlock()
def test_format_initialize_on_transport_ex_repo_fmt_name_None(self):
t = self.get_transport('dir')
@@ -1259,6 +1265,9 @@
# must stay with the all-in-one-format.
repo_name = self.bzrdir_format.network_name()
self.assertEqual(repo_name, repo._format.network_name())
+ # New repositories are write locked.
+ self.assertTrue(repo.is_write_locked())
+ repo.unlock()
def assertInitializeEx(self, t, need_meta=False, **kwargs):
"""Execute initialize_on_transport_ex and check it succeeded correctly.
=== modified file 'bzrlib/tests/test_fetch.py'
--- a/bzrlib/tests/test_fetch.py 2009-04-01 15:49:55 +0000
+++ b/bzrlib/tests/test_fetch.py 2009-04-29 04:14:26 +0000
@@ -321,11 +321,13 @@
self.assertEqual(1, self._count_log_matches('branch/format', http_logs))
self.assertEqual(1, self._count_log_matches('repository/format',
http_logs))
+ self.assertEqual(1, self._count_log_matches('revisions.kndx',
+ http_logs))
self.assertTrue(1 >= self._count_log_matches('revision-history',
http_logs))
self.assertTrue(1 >= self._count_log_matches('last-revision',
http_logs))
- self.assertEqual(4, len(http_logs))
+ self.assertLength(5, http_logs)
class TestKnitToPackFetch(TestCaseWithTransport):
=== modified file 'bzrlib/tests/test_smart.py'
--- a/bzrlib/tests/test_smart.py 2009-04-27 03:34:12 +0000
+++ b/bzrlib/tests/test_smart.py 2009-04-28 03:55:56 +0000
@@ -362,7 +362,7 @@
name = self.make_bzrdir('reference')._format.network_name()
request = smart.bzrdir.SmartServerRequestBzrDirInitializeEx(backing)
self.assertEqual(SmartServerResponse(('', '', '', '', '', '', name,
- 'False', '', '')),
+ 'False', '', '', '')),
request.execute(name, '', 'True', 'False', 'False', '', '', '', '',
'False'))
made_dir = bzrdir.BzrDir.open_from_transport(backing)
More information about the bazaar-commits
mailing list