Rev 2462: Repository.tarball operation to speed initial checkouts in file:///home/pqm/archives/thelove/bzr/%2Btrunk/
Canonical.com Patch Queue Manager
pqm at pqm.ubuntu.com
Thu Apr 26 09:34:23 BST 2007
At file:///home/pqm/archives/thelove/bzr/%2Btrunk/
------------------------------------------------------------
revno: 2462
revision-id: pqm at pqm.ubuntu.com-20070426083414-8xgtmyk47txgquaw
parent: pqm at pqm.ubuntu.com-20070426070825-6xw10b1el98su02i
parent: mbp at sourcefrog.net-20070426074805-va53nylsxqt7ur7u
committer: Canonical.com Patch Queue Manager<pqm at pqm.ubuntu.com>
branch nick: +trunk
timestamp: Thu 2007-04-26 09:34:14 +0100
message:
Repository.tarball operation to speed initial checkouts
modified:
bzrlib/bzrdir.py bzrdir.py-20060131065624-156dfea39c4387cb
bzrlib/remote.py remote.py-20060720103555-yeeg2x51vn0rbtdp-1
bzrlib/repository.py rev_storage.py-20051111201905-119e9401e46257e3
bzrlib/smart/repository.py repository.py-20061128022038-vr5wy5bubyb8xttk-1
bzrlib/smart/request.py request.py-20061108095550-gunadhxmzkdjfeek-1
bzrlib/tests/__init__.py selftest.py-20050531073622-8d0e3c8845c97a64
bzrlib/tests/test_errors.py test_errors.py-20060210110251-41aba2deddf936a8
bzrlib/tests/test_remote.py test_remote.py-20060720103555-yeeg2x51vn0rbtdp-2
bzrlib/tests/test_smart.py test_smart.py-20061122024551-ol0l0o0oofsu9b3t-2
bzrlib/tests/test_transport.py testtransport.py-20050718175618-e5cdb99f4555ddce
bzrlib/tests/test_transport_implementations.py test_transport_implementations.py-20051227111451-f97c5c7d5c49fce7
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.25
merged: mbp at sourcefrog.net-20070426074805-va53nylsxqt7ur7u
parent: mbp at sourcefrog.net-20070423120654-7k0q70jyjrvb5g38
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Thu 2007-04-26 17:48:05 +1000
message:
Repository.tarball fixes for python2.4
Use 'r|bz2' to extract since r:bz2 is not supported
Replace extractall, which is not in python2.4
RemoteRepository._get_tarball returns a TemporaryFile
-------------- This line and the following will be ignored --------------
modified:
bzrlib/remote.py
bzrlib/tests/test_remote.py
bzrlib/tests/test_smart.py
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.24
merged: mbp at sourcefrog.net-20070423120654-7k0q70jyjrvb5g38
parent: mbp at sourcefrog.net-20070423071826-0vcm0vzp4jp3ajax
parent: mbp at sourcefrog.net-20070423095250-xzaleukzs05x9lp0
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Mon 2007-04-23 22:06:54 +1000
message:
Merge Repository.sprout refactoring, and make that use Repository.tarball
------------------------------------------------------------
revno: 2440.1.1
merged: mbp at sourcefrog.net-20070423095250-xzaleukzs05x9lp0
parent: pqm at pqm.ubuntu.com-20070421151139-5wau2ukbpx5z1hv2
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: sprout-cleanup
timestamp: Mon 2007-04-23 19:52:50 +1000
message:
Add new Repository.sprout,
Cleaner in intention and purpose than copy_content_into. It doesn't copy the
extra settings of the repository (like working-trees and shared), which is
normally what you'll want.
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.23
merged: mbp at sourcefrog.net-20070423071826-0vcm0vzp4jp3ajax
parent: mbp at sourcefrog.net-20070423050344-7r4w0fvmjbevcn0r
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Mon 2007-04-23 17:18:26 +1000
message:
review cleanups
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.22
merged: mbp at sourcefrog.net-20070423050344-7r4w0fvmjbevcn0r
parent: mbp at sourcefrog.net-20070416153631-425bo8mmp7kkitox
parent: pqm at pqm.ubuntu.com-20070421151139-5wau2ukbpx5z1hv2
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Mon 2007-04-23 15:03:44 +1000
message:
merge bzr.dev
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.21
merged: mbp at sourcefrog.net-20070416153631-425bo8mmp7kkitox
parent: mbp at sourcefrog.net-20070416153233-bbvwu8649jdl8d4c
parent: pqm at pqm.ubuntu.com-20070416080254-bf3rfk77k5bgfdl7
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Tue 2007-04-17 01:36:31 +1000
message:
merge bzr.dev
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.20
merged: mbp at sourcefrog.net-20070416153233-bbvwu8649jdl8d4c
parent: mbp at sourcefrog.net-20070416153202-l0tgct3ib1vk77fl
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Tue 2007-04-17 01:32:33 +1000
message:
Route branch operations through remote copy_content_into
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.19
merged: mbp at sourcefrog.net-20070416153202-l0tgct3ib1vk77fl
parent: mbp at sourcefrog.net-20070416143438-gev59fyob49myi74
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Tue 2007-04-17 01:32:02 +1000
message:
fix up test data format
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.18
merged: mbp at sourcefrog.net-20070416143438-gev59fyob49myi74
parent: mbp at sourcefrog.net-20070416134530-a9p6c12yp8l1wnr1
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Tue 2007-04-17 00:34:38 +1000
message:
reformat
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.17
merged: mbp at sourcefrog.net-20070416134530-a9p6c12yp8l1wnr1
parent: mbp at sourcefrog.net-20070416100344-xuvua8ydn5ma6syg
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Mon 2007-04-16 23:45:30 +1000
message:
Update and reenable rpc-level tests for Repository.tarball
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.16
merged: mbp at sourcefrog.net-20070416100344-xuvua8ydn5ma6syg
parent: mbp at sourcefrog.net-20070416100133-whdocdv9cnlre1ye
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Mon 2007-04-16 20:03:44 +1000
message:
doc
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.15
merged: mbp at sourcefrog.net-20070416100133-whdocdv9cnlre1ye
parent: mbp at sourcefrog.net-20070416095757-t5eud8iy6l1kw70b
parent: andrew.bennetts at canonical.com-20070416065300-8te6vwujl287yh5p
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Mon 2007-04-16 20:01:33 +1000
message:
Merge more from hpss
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.14
merged: mbp at sourcefrog.net-20070416095757-t5eud8iy6l1kw70b
parent: mbp at sourcefrog.net-20070416042255-rs6sxqr68n374fbi
parent: andrew.bennetts at canonical.com-20070416025619-v6rjozkjjnrg970w
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Mon 2007-04-16 19:57:57 +1000
message:
merge hpss again; restore incorrectly removed RemoteRepository.break_lock
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.13
merged: mbp at sourcefrog.net-20070416042255-rs6sxqr68n374fbi
parent: mbp at sourcefrog.net-20070416033524-7w3l94gisf1p9w3a
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Mon 2007-04-16 14:22:55 +1000
message:
Remove obsolete test
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.12
merged: mbp at sourcefrog.net-20070416033524-7w3l94gisf1p9w3a
parent: mbp at sourcefrog.net-20070416032143-6pmuezr2i7umkj9q
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Mon 2007-04-16 13:35:24 +1000
message:
small test cleanups
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.11
merged: mbp at sourcefrog.net-20070416032143-6pmuezr2i7umkj9q
parent: mbp at sourcefrog.net-20070411100702-r12ixsycxcohqu4s
parent: andrew.bennetts at canonical.com-20070414142229-633813p69cryl6gm
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Mon 2007-04-16 13:21:43 +1000
message:
merge hpss changes
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.10
merged: mbp at sourcefrog.net-20070411100702-r12ixsycxcohqu4s
parent: mbp at sourcefrog.net-20070411091045-yhvop9nb1ngwvi00
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Wed 2007-04-11 20:07:02 +1000
message:
copy_content_into from Remote repositories by using temporary directories on both ends.
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.9
merged: mbp at sourcefrog.net-20070411091045-yhvop9nb1ngwvi00
parent: ian.clatworthy at internode.on.net-20070330063856-c41baqaxj78otvri
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Wed 2007-04-11 19:10:45 +1000
message:
remote Repository.tarball builds a temporary directory and tars that
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.8
merged: ian.clatworthy at internode.on.net-20070330063856-c41baqaxj78otvri
parent: mbp at sourcefrog.net-20070330035956-lmp3z220xq1kdusx
committer: Ian Clatworthy <ian.clatworthy at internode.on.net>
branch nick: hpss-faster-copy
timestamp: Fri 2007-03-30 16:38:56 +1000
message:
Tarball proxy code & tests
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.7
merged: mbp at sourcefrog.net-20070330035956-lmp3z220xq1kdusx
parent: mbp at sourcefrog.net-20070330022302-h7yo29evei09rpj1
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Fri 2007-03-30 13:59:56 +1000
message:
(broken) Start addng client proxy test for Repository.tarball
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.6
merged: mbp at sourcefrog.net-20070330022302-h7yo29evei09rpj1
parent: mbp at sourcefrog.net-20070330022108-5dxt1f34qla13yl9
parent: andrew.bennetts at canonical.com-20070330014631-a3xragco77jx67pv
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Fri 2007-03-30 12:23:02 +1000
message:
merge hpss
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.5
merged: mbp at sourcefrog.net-20070330022108-5dxt1f34qla13yl9
parent: mbp at sourcefrog.net-20070330020514-7601dsj0nt3nievp
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Fri 2007-03-30 12:21:08 +1000
message:
Repository.tarball locks repository while running for consistency
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.4
merged: mbp at sourcefrog.net-20070330020514-7601dsj0nt3nievp
parent: mbp at sourcefrog.net-20070330020456-yck6eki5w4txsr19
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Fri 2007-03-30 12:05:14 +1000
message:
Change Transport.local_abspath to raise NotLocalUrl, and test.
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.3
merged: mbp at sourcefrog.net-20070330020456-yck6eki5w4txsr19
parent: mbp at sourcefrog.net-20070329072503-ir70ask331e138ux
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Fri 2007-03-30 12:04:56 +1000
message:
Cherrypick assertRaises enhancement
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.2
merged: mbp at sourcefrog.net-20070329072503-ir70ask331e138ux
parent: mbp at sourcefrog.net-20070329060803-dywopr2sg30oj23e
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Thu 2007-03-29 17:25:03 +1000
message:
smart method Repository.tarball actually returns the tarball
------------------------------------------------------------
revno: 2018.1.11.1.25.1.13.1.123.1.1
merged: mbp at sourcefrog.net-20070329060803-dywopr2sg30oj23e
parent: robertc at robertcollins.net-20070329042156-28vdpsyvcbrw7wy9
committer: Martin Pool <mbp at sourcefrog.net>
branch nick: hpss-faster-copy
timestamp: Thu 2007-03-29 16:08:03 +1000
message:
Add stub Repository.tarball smart method
=== modified file 'bzrlib/bzrdir.py'
--- a/bzrlib/bzrdir.py 2007-04-20 08:33:47 +0000
+++ b/bzrlib/bzrdir.py 2007-04-23 12:06:54 +0000
@@ -781,13 +781,13 @@
result.create_repository()
elif source_repository is not None and result_repo is None:
# have source, and want to make a new target repo
- # we don't clone the repo because that preserves attributes
- # like is_shared(), and we have not yet implemented a
- # repository sprout().
- result_repo = result.create_repository()
- if result_repo is not None:
+ result_repo = source_repository.sprout(result, revision_id=revision_id)
+ else:
# fetch needed content into target.
if source_repository is not None:
+ # would rather do
+ # source_repository.copy_content_into(result_repo, revision_id=revision_id)
+ # so we can override the copy method
result_repo.fetch(source_repository, revision_id=revision_id)
if source_branch is not None:
source_branch.sprout(result, revision_id=revision_id)
=== modified file 'bzrlib/remote.py'
--- a/bzrlib/remote.py 2007-04-24 19:40:13 +0000
+++ b/bzrlib/remote.py 2007-04-26 08:34:14 +0000
@@ -28,6 +28,7 @@
from bzrlib.lockable_files import LockableFiles
from bzrlib.revision import NULL_REVISION
from bzrlib.smart import client, vfs
+from bzrlib.trace import note
# Note: RemoteBzrDirFormat is in bzrdir.py
@@ -431,6 +432,30 @@
self._ensure_real()
return self._real_repository.break_lock()
+ def _get_tarball(self, compression):
+ """Return a TemporaryFile containing a repository tarball"""
+ import tempfile
+ path = self.bzrdir._path_for_remote_call(self._client)
+ response, protocol = self._client.call_expecting_body(
+ 'Repository.tarball', path, compression)
+ assert response[0] in ('ok', 'failure'), \
+ 'unexpected response code %s' % (response,)
+ if response[0] == 'ok':
+ # Extract the tarball and return it
+ t = tempfile.NamedTemporaryFile()
+ # TODO: rpc layer should read directly into it...
+ t.write(protocol.read_body_bytes())
+ t.seek(0)
+ return t
+ else:
+ raise errors.SmartServerError(error_code=response)
+
+ def sprout(self, to_bzrdir, revision_id=None):
+ # TODO: Option to control what format is created?
+ to_repo = to_bzrdir.create_repository()
+ self._copy_repository_tarball(to_repo, revision_id)
+ return to_repo
+
### These methods are just thin shims to the VFS object for now.
def revision_tree(self, revision_id):
@@ -571,6 +596,35 @@
return self._real_repository.copy_content_into(
destination, revision_id=revision_id)
+ def _copy_repository_tarball(self, destination, revision_id=None):
+ # get a tarball of the remote repository, and copy from that into the
+ # destination
+ from bzrlib import osutils
+ import tarfile
+ import tempfile
+ from StringIO import StringIO
+ # TODO: Maybe a progress bar while streaming the tarball?
+ note("Copying repository content as tarball...")
+ tar_file = self._get_tarball('bz2')
+ try:
+ tar = tarfile.open('repository', fileobj=tar_file,
+ mode='r|bz2')
+ tmpdir = tempfile.mkdtemp()
+ try:
+ _extract_tar(tar, tmpdir)
+ tmp_bzrdir = BzrDir.open(tmpdir)
+ tmp_repo = tmp_bzrdir.open_repository()
+ tmp_repo.copy_content_into(destination, revision_id)
+ finally:
+ osutils.rmtree(tmpdir)
+ finally:
+ tar_file.close()
+ # TODO: if the server doesn't support this operation, maybe do it the
+ # slow way using the _real_repository?
+ #
+ # TODO: Suggestion from john: using external tar is much faster than
+ # python's tarfile library, but it may not work on windows.
+
def set_make_working_trees(self, new_value):
raise NotImplementedError(self.set_make_working_trees)
@@ -927,9 +981,8 @@
# format, because RemoteBranches can't be created at arbitrary URLs.
# XXX: if to_bzrdir is a RemoteBranch, this should perhaps do
# to_bzrdir.create_branch...
- self._ensure_real()
result = branch.BranchFormat.get_default_format().initialize(to_bzrdir)
- self._real_branch.copy_content_into(result, revision_id=revision_id)
+ self.copy_content_into(result, revision_id=revision_id)
result.set_parent(self.bzrdir.root_transport.base)
return result
@@ -991,3 +1044,11 @@
self._branch_data_config = TreeConfig(self.branch._real_branch)
return self._branch_data_config
+
+def _extract_tar(tar, to_dir):
+ """Extract all the contents of a tarfile object.
+
+ A replacement for extractall, which is not present in python2.4
+ """
+ for tarinfo in tar:
+ tar.extract(tarinfo, to_dir)
=== modified file 'bzrlib/repository.py'
--- a/bzrlib/repository.py 2007-04-18 05:38:31 +0000
+++ b/bzrlib/repository.py 2007-04-23 09:52:50 +0000
@@ -395,6 +395,23 @@
:return: The newly created destination repository.
"""
+ # TODO: deprecate after 0.16; cloning this with all its settings is
+ # probably not very useful -- mbp 20070423
+ dest_repo = self._create_sprouting_repo(a_bzrdir, shared=self.is_shared())
+ self.copy_content_into(dest_repo, revision_id)
+ return dest_repo
+
+ @needs_read_lock
+ def sprout(self, to_bzrdir, revision_id=None):
+ """Create a descendent repository for new development.
+
+ Unlike clone, this does not copy the settings of the repository.
+ """
+ dest_repo = self._create_sprouting_repo(to_bzrdir, shared=False)
+ dest_repo.fetch(self, revision_id=revision_id)
+ return dest_repo
+
+ def _create_sprouting_repo(self, a_bzrdir, shared):
if not isinstance(a_bzrdir._format, self.bzrdir._format.__class__):
# use target default format.
dest_repo = a_bzrdir.create_repository()
@@ -402,10 +419,9 @@
# Most control formats need the repository to be specifically
# created, but on some old all-in-one formats it's not needed
try:
- dest_repo = self._format.initialize(a_bzrdir, shared=self.is_shared())
+ dest_repo = self._format.initialize(a_bzrdir, shared=shared)
except errors.UninitializableFormat:
dest_repo = a_bzrdir.open_repository()
- self.copy_content_into(dest_repo, revision_id)
return dest_repo
@needs_read_lock
@@ -772,7 +788,7 @@
reconciler = RepoReconciler(self, thorough=thorough)
reconciler.reconcile()
return reconciler
-
+
@needs_read_lock
def revision_tree(self, revision_id):
"""Return Tree for a revision on this branch.
@@ -1385,6 +1401,9 @@
@needs_write_lock
def copy_content(self, revision_id=None):
"""Make a complete copy of the content in self into destination.
+
+ This copies both the repository's revision data, and configuration information
+ such as the make_working_trees setting.
This is a destructive operation! Do not use it on existing
repositories.
=== modified file 'bzrlib/smart/repository.py'
--- a/bzrlib/smart/repository.py 2007-04-16 17:23:40 +0000
+++ b/bzrlib/smart/repository.py 2007-04-23 05:03:44 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2006 Canonical Ltd
+# Copyright (C) 2006, 2007 Canonical Ltd
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -16,10 +16,14 @@
"""Server-side repository related request implmentations."""
+import sys
+import tempfile
+import tarfile
from bzrlib import errors
from bzrlib.bzrdir import BzrDir
from bzrlib.smart.request import SmartServerRequest, SmartServerResponse
+from bzrlib.transport.local import LocalTransport
class SmartServerRepositoryRequest(SmartServerRequest):
@@ -173,3 +177,59 @@
repository.unlock()
return SmartServerResponse(('ok',))
+
+class SmartServerRepositoryTarball(SmartServerRepositoryRequest):
+ """Get the raw repository files as a tarball.
+
+ The returned tarball contains a .bzr control directory which in turn
+ contains a repository.
+
+ This takes one parameter, compression, which currently must be
+ "", "gz", or "bz2".
+
+ This is used to implement the Repository.copy_content_into operation.
+ """
+
+ def do_repository_request(self, repository, compression):
+ from bzrlib import osutils
+ repo_transport = repository.control_files._transport
+ tmp_dirname, tmp_repo = self._copy_to_tempdir(repository)
+ try:
+ controldir_name = tmp_dirname + '/.bzr'
+ return self._tarfile_response(controldir_name, compression)
+ finally:
+ osutils.rmtree(tmp_dirname)
+
+ def _copy_to_tempdir(self, from_repo):
+ tmp_dirname = tempfile.mkdtemp(prefix='tmpbzrclone')
+ tmp_bzrdir = from_repo.bzrdir._format.initialize(tmp_dirname)
+ tmp_repo = from_repo._format.initialize(tmp_bzrdir)
+ from_repo.copy_content_into(tmp_repo)
+ return tmp_dirname, tmp_repo
+
+ def _tarfile_response(self, tmp_dirname, compression):
+ temp = tempfile.NamedTemporaryFile()
+ try:
+ self._tarball_of_dir(tmp_dirname, compression, temp.name)
+ # all finished; write the tempfile out to the network
+ temp.seek(0)
+ return SmartServerResponse(('ok',), temp.read())
+ # FIXME: Don't read the whole thing into memory here; rather stream it
+ # out from the file onto the network. mbp 20070411
+ finally:
+ temp.close()
+
+ def _tarball_of_dir(self, dirname, compression, tarfile_name):
+ tarball = tarfile.open(tarfile_name, mode='w:' + compression)
+ try:
+ # The tarball module only accepts ascii names, and (i guess)
+ # packs them with their 8bit names. We know all the files
+ # within the repository have ASCII names so the should be safe
+ # to pack in.
+ dirname = dirname.encode(sys.getfilesystemencoding())
+ # python's tarball module includes the whole path by default so
+ # override it
+ assert dirname.endswith('.bzr')
+ tarball.add(dirname, '.bzr') # recursive by default
+ finally:
+ tarball.close()
=== modified file 'bzrlib/smart/request.py'
--- a/bzrlib/smart/request.py 2007-04-24 09:01:50 +0000
+++ b/bzrlib/smart/request.py 2007-04-26 08:34:14 +0000
@@ -295,6 +295,9 @@
request_handlers.register_lazy(
'Repository.unlock', 'bzrlib.smart.repository', 'SmartServerRepositoryUnlock')
request_handlers.register_lazy(
+ 'Repository.tarball', 'bzrlib.smart.repository',
+ 'SmartServerRepositoryTarball')
+request_handlers.register_lazy(
'rmdir', 'bzrlib.smart.vfs', 'RmdirRequest')
request_handlers.register_lazy(
'stat', 'bzrlib.smart.vfs', 'StatRequest')
=== modified file 'bzrlib/tests/__init__.py'
--- a/bzrlib/tests/__init__.py 2007-04-25 06:50:22 +0000
+++ b/bzrlib/tests/__init__.py 2007-04-26 08:34:14 +0000
@@ -780,11 +780,12 @@
bzrlib.smart.server.SmartTCPServer: bzrlib.smart.server.SmartTCPServer.hooks,
}
self.addCleanup(self._restoreHooks)
- # this list of hooks must be kept in sync with the defaults
- # in branch.py
+ # reset all hooks to an empty instance of the appropriate type
bzrlib.branch.Branch.hooks = bzrlib.branch.BranchHooks()
- bzrlib.smart.server.SmartTCPServer.hooks = \
- bzrlib.smart.server.SmartServerHooks()
+ bzrlib.smart.server.SmartTCPServer.hooks = bzrlib.smart.server.SmartServerHooks()
+ # FIXME: Rather than constructing new objects like this, how about
+ # having save() and clear() methods on the base Hook class? mbp
+ # 20070416
def _silenceUI(self):
"""Turn off UI for duration of test"""
@@ -1745,7 +1746,13 @@
def get_vfs_only_url(self, relpath=None):
"""Get a URL (or maybe a path for the plain old vfs transport.
- This will never be a smart protocol.
+ This will never be a smart protocol. It always has all the
+ capabilities of the local filesystem, but it might actually be a
+ MemoryTransport or some other similar virtual filesystem.
+
+ This is the backing transport (if any) of the server returned by
+ get_url and get_readonly_url.
+
:param relpath: provides for clients to get a path relative to the base
url. These should only be downwards relative, not upwards.
"""
@@ -1811,7 +1818,14 @@
raise TestSkipped("Format %s is not initializable." % format)
def make_repository(self, relpath, shared=False, format=None):
- """Create a repository on our default transport at relpath."""
+ """Create a repository on our default transport at relpath.
+
+ Note that relpath must be a relative path, not a full url.
+ """
+ # FIXME: If you create a remoterepository this returns the underlying
+ # real format, which is incorrect. Actually we should make sure that
+ # RemoteBzrDir returns a RemoteRepository.
+ # maybe mbp 20070410
made_control = self.make_bzrdir(relpath, format=format)
return made_control.create_repository(shared=shared)
=== modified file 'bzrlib/tests/test_errors.py'
--- a/bzrlib/tests/test_errors.py 2007-04-23 06:55:58 +0000
+++ b/bzrlib/tests/test_errors.py 2007-04-26 08:34:14 +0000
@@ -1,5 +1,6 @@
-# Copyright (C) 2006 Canonical Ltd
+# Copyright (C) 2006, 2007 Canonical Ltd
# Authors: Robert Collins <robert.collins at canonical.com>
+# and others
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
=== modified file 'bzrlib/tests/test_remote.py'
--- a/bzrlib/tests/test_remote.py 2007-04-20 05:11:46 +0000
+++ b/bzrlib/tests/test_remote.py 2007-04-26 07:48:05 +0000
@@ -19,13 +19,17 @@
These are proxy objects which act on remote objects by sending messages
through a smart client. The proxies are to be created when attempting to open
the object given a transport that supports smartserver rpc operations.
+
+These tests correspond to tests.test_smart, which exercises the server side.
"""
from cStringIO import StringIO
from bzrlib import (
+ bzrdir,
errors,
remote,
+ repository,
tests,
)
from bzrlib.branch import Branch
@@ -369,9 +373,22 @@
class TestRemoteRepository(tests.TestCase):
+ """Base for testing RemoteRepository protocol usage.
+
+ These tests contain frozen requests and responses. We want any changes to
+ what is sent or expected to be require a thoughtful update to these tests
+ because they might break compatibility with different-versioned servers.
+ """
def setup_fake_client_and_repository(self, responses, transport_path):
- """Create the fake client and repository for testing with."""
+ """Create the fake client and repository for testing with.
+
+ There's no real server here; we just have canned responses sent
+ back one by one.
+
+ :param transport_path: Path below the root of the MemoryTransport
+ where the repository will be created.
+ """
client = FakeClient(responses)
transport = MemoryTransport()
transport.mkdir(transport_path)
@@ -609,3 +626,77 @@
# The remote repo shouldn't be accessed.
self.assertEqual([], client._calls)
+
+
+class TestRepositoryTarball(TestRemoteRepository):
+
+ # This is a canned tarball reponse we can validate against
+ tarball_content = (
+ 'QlpoOTFBWSZTWdGkj3wAAWF/k8aQACBIB//A9+8cIX/v33AACEAYABAECEACNz'
+ 'JqsgJJFPTSnk1A3qh6mTQAAAANPUHkagkSTEkaA09QaNAAAGgAAAcwCYCZGAEY'
+ 'mJhMJghpiaYBUkKammSHqNMZQ0NABkNAeo0AGneAevnlwQoGzEzNVzaYxp/1Uk'
+ 'xXzA1CQX0BJMZZLcPBrluJir5SQyijWHYZ6ZUtVqqlYDdB2QoCwa9GyWwGYDMA'
+ 'OQYhkpLt/OKFnnlT8E0PmO8+ZNSo2WWqeCzGB5fBXZ3IvV7uNJVE7DYnWj6qwB'
+ 'k5DJDIrQ5OQHHIjkS9KqwG3mc3t+F1+iujb89ufyBNIKCgeZBWrl5cXxbMGoMs'
+ 'c9JuUkg5YsiVcaZJurc6KLi6yKOkgCUOlIlOpOoXyrTJjK8ZgbklReDdwGmFgt'
+ 'dkVsAIslSVCd4AtACSLbyhLHryfb14PKegrVDba+U8OL6KQtzdM5HLjAc8/p6n'
+ '0lgaWU8skgO7xupPTkyuwheSckejFLK5T4ZOo0Gda9viaIhpD1Qn7JqqlKAJqC'
+ 'QplPKp2nqBWAfwBGaOwVrz3y1T+UZZNismXHsb2Jq18T+VaD9k4P8DqE3g70qV'
+ 'JLurpnDI6VS5oqDDPVbtVjMxMxMg4rzQVipn2Bv1fVNK0iq3Gl0hhnnHKm/egy'
+ 'nWQ7QH/F3JFOFCQ0aSPfA='
+ ).decode('base64')
+
+ def test_repository_tarball(self):
+ # Test that Repository.tarball generates the right operations
+ transport_path = 'repo'
+ expected_responses = [(('ok',), self.tarball_content),
+ ]
+ expected_calls = [('call_expecting_body', 'Repository.tarball',
+ ('///repo/', 'bz2',),),
+ ]
+ remote_repo, client = self.setup_fake_client_and_repository(
+ expected_responses, transport_path)
+ # Now actually ask for the tarball
+ tarball_file = remote_repo._get_tarball('bz2')
+ try:
+ self.assertEqual(expected_calls, client._calls)
+ self.assertEqual(self.tarball_content, tarball_file.read())
+ finally:
+ tarball_file.close()
+
+ def test_sprout_uses_tarball(self):
+ # RemoteRepository.sprout should try to use the
+ # tarball command rather than accessing all the files
+ transport_path = 'srcrepo'
+ expected_responses = [(('ok',), self.tarball_content),
+ ]
+ expected_calls = [('call2', 'Repository.tarball', ('///srcrepo/', 'bz2',),),
+ ]
+ remote_repo, client = self.setup_fake_client_and_repository(
+ expected_responses, transport_path)
+ # make a regular local repository to receive the results
+ dest_transport = MemoryTransport()
+ dest_transport.mkdir('destrepo')
+ bzrdir_format = bzrdir.format_registry.make_bzrdir('default')
+ dest_bzrdir = bzrdir_format.initialize_on_transport(dest_transport)
+ # try to copy...
+ remote_repo.sprout(dest_bzrdir)
+
+
+class TestRemoteRepositoryCopyContent(tests.TestCaseWithTransport):
+ """RemoteRepository.copy_content_into optimizations"""
+
+ def test_copy_content_remote_to_local(self):
+ self.transport_server = server.SmartTCPServer_for_testing
+ src_repo = self.make_repository('repo1')
+ src_repo = repository.Repository.open(self.get_url('repo1'))
+ # At the moment the tarball-based copy_content_into can't write back
+ # into a smart server. It would be good if it could upload the
+ # tarball; once that works we'd have to create repositories of
+ # different formats. -- mbp 20070410
+ dest_url = self.get_vfs_only_url('repo2')
+ dest_bzrdir = BzrDir.create(dest_url)
+ dest_repo = dest_bzrdir.create_repository()
+ self.assertFalse(isinstance(dest_repo, RemoteRepository))
+ self.assertTrue(isinstance(src_repo, RemoteRepository))
+ src_repo.copy_content_into(dest_repo)
=== modified file 'bzrlib/tests/test_smart.py'
--- a/bzrlib/tests/test_smart.py 2007-04-19 04:29:20 +0000
+++ b/bzrlib/tests/test_smart.py 2007-04-26 07:48:05 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2006 Canonical Ltd
+# Copyright (C) 2006, 2007 Canonical Ltd
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -16,6 +16,10 @@
"""Tests for the smart wire/domain protococl."""
+from StringIO import StringIO
+import tempfile
+import tarfile
+
from bzrlib import bzrdir, errors, smart, tests
from bzrlib.smart.request import SmartServerResponse
import bzrlib.smart.bzrdir
@@ -738,6 +742,30 @@
SmartServerResponse(('TokenMismatch',)), response)
+class TestSmartServerRepositoryTarball(tests.TestCaseWithTransport):
+
+ def test_repository_tarball(self):
+ backing = self.get_transport()
+ request = smart.repository.SmartServerRepositoryTarball(backing)
+ repository = self.make_repository('.')
+ # make some extraneous junk in the repository directory which should
+ # not be copied
+ self.build_tree(['.bzr/repository/extra-junk'])
+ response = request.execute(backing.local_abspath(''), 'bz2')
+ self.assertEqual(('ok',), response.args)
+ # body should be a tbz2
+ body_file = StringIO(response.body)
+ body_tar = tarfile.open('body_tar.tbz2', fileobj=body_file,
+ mode='r|bz2')
+ # let's make sure there are some key repository components inside it.
+ # the tarfile returns directories with trailing slashes...
+ names = set([n.rstrip('/') for n in body_tar.getnames()])
+ self.assertTrue('.bzr/repository/lock' in names)
+ self.assertTrue('.bzr/repository/format' in names)
+ self.assertTrue('.bzr/repository/extra-junk' not in names,
+ "extraneous file present in tar file")
+
+
class TestSmartServerIsReadonly(tests.TestCaseWithTransport):
def test_is_readonly_no(self):
@@ -806,5 +834,8 @@
smart.request.request_handlers.get('Repository.unlock'),
smart.repository.SmartServerRepositoryUnlock)
self.assertEqual(
+ smart.request.request_handlers.get('Repository.tarball'),
+ smart.repository.SmartServerRepositoryTarball)
+ self.assertEqual(
smart.request.request_handlers.get('Transport.is_readonly'),
smart.request.SmartServerIsReadonly)
=== modified file 'bzrlib/tests/test_transport.py'
--- a/bzrlib/tests/test_transport.py 2007-04-21 11:01:23 +0000
+++ b/bzrlib/tests/test_transport.py 2007-04-26 08:34:14 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2004, 2005, 2006 Canonical Ltd
+# Copyright (C) 2004, 2005, 2006, 2007 Canonical Ltd
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -21,7 +21,10 @@
from cStringIO import StringIO
import bzrlib
-from bzrlib import urlutils
+from bzrlib import (
+ errors,
+ urlutils,
+ )
from bzrlib.errors import (ConnectionError,
DependencyNotPresent,
FileExists,
@@ -125,6 +128,12 @@
self.assertEqual('/etc',
t._combine_paths('/home/sarah', '/etc'))
+ def test_local_abspath_non_local_transport(self):
+ # the base implementation should throw
+ t = MemoryTransport()
+ e = self.assertRaises(errors.NotLocalUrl, t.local_abspath, 't')
+ self.assertEqual('memory:///t is not a local path.', str(e))
+
class TestCoalesceOffsets(TestCase):
@@ -465,7 +474,7 @@
transport = self.get_nfs_transport('.')
self.build_tree(['from/', 'from/foo', 'to/', 'to/bar'],
transport=transport)
- self.assertRaises(bzrlib.errors.ResourceBusy,
+ self.assertRaises(errors.ResourceBusy,
transport.rename, 'from', 'to')
@@ -564,6 +573,11 @@
self.assertIsInstance(t, LocalTransport)
self.assertEquals(t.base, here_url)
+ def test_local_abspath(self):
+ here = os.path.abspath('.')
+ t = get_transport(here)
+ self.assertEquals(t.local_abspath(''), here)
+
class TestWin32LocalTransport(TestCase):
=== modified file 'bzrlib/tests/test_transport_implementations.py'
--- a/bzrlib/tests/test_transport_implementations.py 2007-04-20 08:33:47 +0000
+++ b/bzrlib/tests/test_transport_implementations.py 2007-04-23 05:03:44 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2004, 2005, 2006 Canonical Ltd
+# Copyright (C) 2004, 2005, 2006, 2007 Canonical Ltd
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -1150,8 +1150,9 @@
transport = self.get_transport()
try:
p = transport.local_abspath('.')
- except NotLocalUrl:
- pass # This is not a local transport
+ except (errors.NotLocalUrl, TransportNotPossible), e:
+ # should be formattable
+ s = str(e)
else:
self.assertEqual(getcwd(), p)
More information about the bazaar-commits
mailing list