Rev 3983: (vila) Progress bar at socket level for http in file:///home/pqm/archives/thelove/bzr/%2Btrunk/
Canonical.com Patch Queue Manager
pqm at pqm.ubuntu.com
Thu Feb 5 11:00:43 GMT 2009
At file:///home/pqm/archives/thelove/bzr/%2Btrunk/
------------------------------------------------------------
revno: 3983
revision-id: pqm at pqm.ubuntu.com-20090205110039-w9oelsyvyx160qwy
parent: pqm at pqm.ubuntu.com-20090205054228-3qyiv92vtgs94e0c
parent: v.ladeuil+lp at free.fr-20090205102051-gyivrlud1hk1c12g
committer: Canonical.com Patch Queue Manager <pqm at pqm.ubuntu.com>
branch nick: +trunk
timestamp: Thu 2009-02-05 11:00:39 +0000
message:
(vila) Progress bar at socket level for http
modified:
NEWS NEWS-20050323055033-4e00b5db738777ff
bzrlib/tests/https_server.py https_server.py-20071121173708-aj8zczi0ziwbwz21-1
bzrlib/tests/test_http.py testhttp.py-20051018020158-b2eef6e867c514d9
bzrlib/transport/http/__init__.py http_transport.py-20050711212304-506c5fd1059ace96
bzrlib/transport/http/_pycurl.py pycurlhttp.py-20060110060940-4e2a705911af77a6
bzrlib/transport/http/_urllib.py _urlgrabber.py-20060113083826-0bbf7d992fbf090c
bzrlib/transport/http/_urllib2_wrappers.py _urllib2_wrappers.py-20060913231729-ha9ugi48ktx481ao-1
bzrlib/transport/http/response.py _response.py-20060613154423-a2ci7hd4iw5c7fnt-1
------------------------------------------------------------
revno: 3982.1.1
revision-id: v.ladeuil+lp at free.fr-20090205102051-gyivrlud1hk1c12g
parent: pqm at pqm.ubuntu.com-20090205054228-3qyiv92vtgs94e0c
parent: v.ladeuil+lp at free.fr-20090130004941-820fpd2ryyo127vv
committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
branch nick: bzr.integration
timestamp: Thu 2009-02-05 11:20:51 +0100
message:
Progress bar at socket level for http
modified:
NEWS NEWS-20050323055033-4e00b5db738777ff
bzrlib/tests/https_server.py https_server.py-20071121173708-aj8zczi0ziwbwz21-1
bzrlib/tests/test_http.py testhttp.py-20051018020158-b2eef6e867c514d9
bzrlib/transport/http/__init__.py http_transport.py-20050711212304-506c5fd1059ace96
bzrlib/transport/http/_pycurl.py pycurlhttp.py-20060110060940-4e2a705911af77a6
bzrlib/transport/http/_urllib.py _urlgrabber.py-20060113083826-0bbf7d992fbf090c
bzrlib/transport/http/_urllib2_wrappers.py _urllib2_wrappers.py-20060913231729-ha9ugi48ktx481ao-1
bzrlib/transport/http/response.py _response.py-20060613154423-a2ci7hd4iw5c7fnt-1
------------------------------------------------------------
revno: 3945.1.8
revision-id: v.ladeuil+lp at free.fr-20090130004941-820fpd2ryyo127vv
parent: v.ladeuil+lp at free.fr-20090129193101-w0008aa1ufoe7cmd
committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
branch nick: pb-http
timestamp: Fri 2009-01-30 01:49:41 +0100
message:
Add more tests, fix pycurl double handling, revert previous tracking.
* bzrlib/tests/test_http.py:
(PredefinedRequestHandler): Renamed from
PreRecordedRequestHandler.
(PredefinedRequestHandler.handle_one_request): Get the canned
response from the test server directly.
(ActivityServerMixin): Make it a true object and intialize the
attributes in the constructor. Tests can now set the
canned_response attribute before querying the server.
(TestActivity.setUp, TestActivity.tearDown,
TestActivity.get_transport, TestActivity.assertActivitiesMatch):
Extracted from test_get to be able to write other tests.
(TestActivity.test_has, TestActivity.test_readv,
TestActivity.test_post): New tests, all cases should be covered
now.
* bzrlib/transport/http/response.py:
(RangeFile.__init__, RangeFile.read, handle_response): Revert
previous tracking, both http implementations can now report
activity from the socket.
* bzrlib/transport/http/_pycurl.py:
(PyCurlTransport._get_ranged, PyCurlTransport._post): Revert
previous tracking.
modified:
BRANCH.TODO BRANCH.TODO-20060103052123-79ac4969351c03a9
NEWS NEWS-20050323055033-4e00b5db738777ff
bzrlib/tests/test_http.py testhttp.py-20051018020158-b2eef6e867c514d9
bzrlib/transport/http/_pycurl.py pycurlhttp.py-20060110060940-4e2a705911af77a6
bzrlib/transport/http/response.py _response.py-20060613154423-a2ci7hd4iw5c7fnt-1
------------------------------------------------------------
revno: 3945.1.7
revision-id: v.ladeuil+lp at free.fr-20090129193101-w0008aa1ufoe7cmd
parent: v.ladeuil+lp at free.fr-20090129165458-o52vbf2nl4dxadoc
committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
branch nick: pb-http
timestamp: Thu 2009-01-29 20:31:01 +0100
message:
Test against https.
* bzrlib/tests/test_http.py:
(load_tests): Get rid of TestAdapter daughter classes, they are
useless. Add parametrization against http/https for activity
tests.
(ActivityServerMixin, ActivityHTTPServer, ActivityHTTPSServer):
Make HTTP[S] servers.
(PreRecordedRequestHandler.handle_one_request): Be defensive
against threads being non-deterministic.
(TestActivity.test_http_get): Use parametrized test server.
* bzrlib/tests/https_server.py:
(HTTPSServer.__init__): Add protocol_version parameter for tests
purposes.
modified:
BRANCH.TODO BRANCH.TODO-20060103052123-79ac4969351c03a9
bzrlib/tests/https_server.py https_server.py-20071121173708-aj8zczi0ziwbwz21-1
bzrlib/tests/test_http.py testhttp.py-20051018020158-b2eef6e867c514d9
------------------------------------------------------------
revno: 3945.1.6
revision-id: v.ladeuil+lp at free.fr-20090129165458-o52vbf2nl4dxadoc
parent: v.ladeuil+lp at free.fr-20090129142728-1iu8017zgso23w0i
committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
branch nick: pb-http
timestamp: Thu 2009-01-29 17:54:58 +0100
message:
Fix debug handling for pycurl and implement pycurl http activity
reporting at socket level.
* bzrlib/transport/http/_pycurl.py:
(PyCurlTransport._debug_cb): Two birds with one stone: correctly
handle the debug traces with mutter (no more stderr pollution !)
and report activity for all the transmitted headers and data in
both directions.
(PyCurlTransport._set_curl_options): Setup a pycurl debug callback
for both activity reporting and http debug flag handling.
modified:
BRANCH.TODO BRANCH.TODO-20060103052123-79ac4969351c03a9
NEWS NEWS-20050323055033-4e00b5db738777ff
bzrlib/transport/http/_pycurl.py pycurlhttp.py-20060110060940-4e2a705911af77a6
------------------------------------------------------------
revno: 3945.1.5
revision-id: v.ladeuil+lp at free.fr-20090129142728-1iu8017zgso23w0i
parent: v.ladeuil+lp at free.fr-20090126081302-ad8l8nhn2sscymxt
committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
branch nick: pb-http
timestamp: Thu 2009-01-29 15:27:28 +0100
message:
Start implementing http activity reporting at socket level.
* bzrlib/transport/http/_urllib2_wrappers.py:
(_ReportingFileSocket, _ReportingSocket): Wrappers for
socket._socket and socket_filesocket objects providing activity
reporting.
(Response.__init__): Deleted. Wrapping the file socket object is
now done handled at connection time.
(AbstractHTTPConnection._wrap_socket_for_reporting): Wrap the
underlying socket into a reporting one.
(HTTPConnection.__init__, HTTPSConnection.__init__): Accept a
report_activity parameter.
(HTTPSConnection.connect): Wrap the newly created socket.
(HTTPSConnection.connect_to_origin): Wrap the newly created *or*
connected ssl socket.
(ConnectionHandler.__init__): Accept a report_activity parameter.
(ConnectionHandler.create_connection): Pass the report_activity
parameter to all created connections.
(Opener.__init__): Accept a report_activity parameter.
* bzrlib/transport/http/_urllib.py:
(HttpTransport_urllib.__init__): Provide the report activity
function to the opener.
(HttpTransport_urllib._get, HttpTransport_urllib._post): Don't use
the RangeFile report activity facility anymore.
* bzrlib/transport/http/__init__.py:
(HttpTransportBase.get_bytes): Deleted, this gratuitously add a
useless level of buffering.
* bzrlib/tests/test_http.py:
(ActivityHttpServer, PreRecoredRequestHandler, TestActivity): Test
activity reporting by http clients.
modified:
BRANCH.TODO BRANCH.TODO-20060103052123-79ac4969351c03a9
bzrlib/tests/test_http.py testhttp.py-20051018020158-b2eef6e867c514d9
bzrlib/transport/http/__init__.py http_transport.py-20050711212304-506c5fd1059ace96
bzrlib/transport/http/_urllib.py _urlgrabber.py-20060113083826-0bbf7d992fbf090c
bzrlib/transport/http/_urllib2_wrappers.py _urllib2_wrappers.py-20060913231729-ha9ugi48ktx481ao-1
------------------------------------------------------------
revno: 3945.1.4
revision-id: v.ladeuil+lp at free.fr-20090126081302-ad8l8nhn2sscymxt
parent: v.ladeuil+lp at free.fr-20090122112003-pcp5bu8pfc41u7iz
parent: v.ladeuil+lp at free.fr-20090126081250-m310ky1gjotwcw37
committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
branch nick: pb-http
timestamp: Mon 2009-01-26 09:13:02 +0100
message:
Merge emacs-ui-fixes into pb-http
added:
bzrlib/tests/branch_implementations/test_dotted_revno_to_revision_id.py test_dotted_revno_to-20090121014844-6x7d9jtri5sspg1o-1
bzrlib/tests/branch_implementations/test_iter_merge_sorted_revisions.py test_merge_sorted_re-20090121004847-to3gvjwigstu93eh-1
bzrlib/tests/branch_implementations/test_revision_id_to_dotted_revno.py test_revision_id_to_-20090122052032-g3czslif6sdqfkh3-1
modified:
NEWS NEWS-20050323055033-4e00b5db738777ff
bzrlib/branch.py branch.py-20050309040759-e4baf4e0d046576e
bzrlib/help_topics/__init__.py help_topics.py-20060920210027-rnim90q9e0bwxvy4-1
bzrlib/help_topics/en/rules.txt rules.txt-20080516063844-ghr5l6pvvrhiycun-1
bzrlib/knit.py knit.py-20051212171256-f056ac8f0fbe1bd9
bzrlib/log.py log.py-20050505065812-c40ce11702fe5fb1
bzrlib/merge.py merge.py-20050513021216-953b65a438527106
bzrlib/osutils.py osutils.py-20050309040759-eeaff12fbf77ac86
bzrlib/progress.py progress.py-20050610070202-df9faaab791964c0
bzrlib/remote.py remote.py-20060720103555-yeeg2x51vn0rbtdp-1
bzrlib/revisionspec.py revisionspec.py-20050907152633-17567659fd5c0ddb
bzrlib/rules.py properties.py-20080506032617-9k06uqalkf09ck0z-1
bzrlib/tests/branch_implementations/__init__.py __init__.py-20060123013057-b12a52c3f361daf4
bzrlib/tests/test_knit.py test_knit.py-20051212171302-95d4c00dd5f11f2b
bzrlib/tests/test_log.py testlog.py-20050728115707-1a514809d7d49309
bzrlib/tests/test_merge.py testmerge.py-20050905070950-c1b5aa49ff911024
bzrlib/tests/test_osutils.py test_osutils.py-20051201224856-e48ee24c12182989
bzrlib/tests/test_progress.py test_progress.py-20060308160359-978c397bc79b7fda
bzrlib/tests/test_rules.py test_properties.py-20080506033501-3p9kmuob25dho8xl-1
bzrlib/tests/test_ui.py test_ui.py-20051130162854-458e667a7414af09
bzrlib/transport/http/__init__.py http_transport.py-20050711212304-506c5fd1059ace96
bzrlib/transport/http/_pycurl.py pycurlhttp.py-20060110060940-4e2a705911af77a6
bzrlib/transport/http/_urllib.py _urlgrabber.py-20060113083826-0bbf7d992fbf090c
bzrlib/transport/http/response.py _response.py-20060613154423-a2ci7hd4iw5c7fnt-1
bzrlib/transport/sftp.py sftp.py-20051019050329-ab48ce71b7e32dfe
bzrlib/ui/__init__.py ui.py-20050824083933-8cf663c763ba53a9
bzrlib/ui/text.py text.py-20051130153916-2e438cffc8afc478
------------------------------------------------------------
revno: 3945.2.1
revision-id: v.ladeuil+lp at free.fr-20090126081250-m310ky1gjotwcw37
parent: v.ladeuil+lp at free.fr-20090119130947-1ldks301mpsymf8r
parent: pqm at pqm.ubuntu.com-20090124185051-8oryvqq68n6repso
committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
branch nick: emacs-ui-fixes
timestamp: Mon 2009-01-26 09:12:50 +0100
message:
Merge bzr.dev into emacs-ui-fixes
added:
bzrlib/tests/branch_implementations/test_dotted_revno_to_revision_id.py test_dotted_revno_to-20090121014844-6x7d9jtri5sspg1o-1
bzrlib/tests/branch_implementations/test_iter_merge_sorted_revisions.py test_merge_sorted_re-20090121004847-to3gvjwigstu93eh-1
bzrlib/tests/branch_implementations/test_revision_id_to_dotted_revno.py test_revision_id_to_-20090122052032-g3czslif6sdqfkh3-1
modified:
NEWS NEWS-20050323055033-4e00b5db738777ff
bzrlib/branch.py branch.py-20050309040759-e4baf4e0d046576e
bzrlib/help_topics/__init__.py help_topics.py-20060920210027-rnim90q9e0bwxvy4-1
bzrlib/help_topics/en/rules.txt rules.txt-20080516063844-ghr5l6pvvrhiycun-1
bzrlib/knit.py knit.py-20051212171256-f056ac8f0fbe1bd9
bzrlib/log.py log.py-20050505065812-c40ce11702fe5fb1
bzrlib/merge.py merge.py-20050513021216-953b65a438527106
bzrlib/osutils.py osutils.py-20050309040759-eeaff12fbf77ac86
bzrlib/progress.py progress.py-20050610070202-df9faaab791964c0
bzrlib/remote.py remote.py-20060720103555-yeeg2x51vn0rbtdp-1
bzrlib/revisionspec.py revisionspec.py-20050907152633-17567659fd5c0ddb
bzrlib/rules.py properties.py-20080506032617-9k06uqalkf09ck0z-1
bzrlib/tests/branch_implementations/__init__.py __init__.py-20060123013057-b12a52c3f361daf4
bzrlib/tests/test_knit.py test_knit.py-20051212171302-95d4c00dd5f11f2b
bzrlib/tests/test_log.py testlog.py-20050728115707-1a514809d7d49309
bzrlib/tests/test_merge.py testmerge.py-20050905070950-c1b5aa49ff911024
bzrlib/tests/test_osutils.py test_osutils.py-20051201224856-e48ee24c12182989
bzrlib/tests/test_progress.py test_progress.py-20060308160359-978c397bc79b7fda
bzrlib/tests/test_rules.py test_properties.py-20080506033501-3p9kmuob25dho8xl-1
bzrlib/tests/test_ui.py test_ui.py-20051130162854-458e667a7414af09
bzrlib/transport/http/__init__.py http_transport.py-20050711212304-506c5fd1059ace96
bzrlib/transport/http/_pycurl.py pycurlhttp.py-20060110060940-4e2a705911af77a6
bzrlib/transport/http/_urllib.py _urlgrabber.py-20060113083826-0bbf7d992fbf090c
bzrlib/transport/http/response.py _response.py-20060613154423-a2ci7hd4iw5c7fnt-1
bzrlib/transport/sftp.py sftp.py-20051019050329-ab48ce71b7e32dfe
bzrlib/ui/__init__.py ui.py-20050824083933-8cf663c763ba53a9
bzrlib/ui/text.py text.py-20051130153916-2e438cffc8afc478
=== modified file 'NEWS'
--- a/NEWS 2009-02-04 04:19:15 +0000
+++ b/NEWS 2009-02-05 10:20:51 +0000
@@ -59,6 +59,11 @@
* Progress bars now show the rate of activity for some sftp
operations, and they are drawn different. (Martin Pool, #172741)
+ * Progress bars now show the rate of activity for urllib and pycurl based
+ http client implementations. The operations are tracked at the socket
+ level for better precision.
+ (Vincent Ladeuil)
+
* Rule-based preferences can now accept multiple patterns for a set of
rules. (Marius Kruger)
=== modified file 'bzrlib/tests/https_server.py'
--- a/bzrlib/tests/https_server.py 2009-01-08 15:28:17 +0000
+++ b/bzrlib/tests/https_server.py 2009-01-29 19:31:01 +0000
@@ -75,9 +75,11 @@
# Provides usable defaults since an https server requires both a
# private key and certificate to work.
def __init__(self, request_handler=http_server.TestingHTTPRequestHandler,
+ protocol_version=None,
key_file=ssl_certs.build_path('server_without_pass.key'),
cert_file=ssl_certs.build_path('server.crt')):
- http_server.HttpServer.__init__(self, request_handler)
+ http_server.HttpServer.__init__(self, request_handler=request_handler,
+ protocol_version=protocol_version)
self.key_file = key_file
self.cert_file = cert_file
self.temp_files = []
=== modified file 'bzrlib/tests/test_http.py'
--- a/bzrlib/tests/test_http.py 2009-01-08 16:57:10 +0000
+++ b/bzrlib/tests/test_http.py 2009-01-30 00:49:41 +0000
@@ -65,100 +65,85 @@
pycurl_present = False
-class TransportAdapter(tests.TestScenarioApplier):
- """Generate the same test for each transport implementation."""
-
- def __init__(self):
- transport_scenarios = [
- ('urllib', dict(_transport=_urllib.HttpTransport_urllib,
- _server=http_server.HttpServer_urllib,
- _qualified_prefix='http+urllib',)),
- ]
- if pycurl_present:
- transport_scenarios.append(
- ('pycurl', dict(_transport=PyCurlTransport,
- _server=http_server.HttpServer_PyCurl,
- _qualified_prefix='http+pycurl',)))
- self.scenarios = transport_scenarios
-
-
-class TransportProtocolAdapter(TransportAdapter):
- """Generate the same test for each protocol implementation.
-
- In addition to the transport adaptatation that we inherit from.
- """
-
- def __init__(self):
- super(TransportProtocolAdapter, self).__init__()
- protocol_scenarios = [
- ('HTTP/1.0', dict(_protocol_version='HTTP/1.0')),
- ('HTTP/1.1', dict(_protocol_version='HTTP/1.1')),
- ]
- self.scenarios = tests.multiply_scenarios(self.scenarios,
- protocol_scenarios)
-
-
-class TransportProtocolAuthenticationAdapter(TransportProtocolAdapter):
- """Generate the same test for each authentication scheme implementation.
-
- In addition to the protocol adaptatation that we inherit from.
- """
-
- def __init__(self):
- super(TransportProtocolAuthenticationAdapter, self).__init__()
- auth_scheme_scenarios = [
- ('basic', dict(_auth_scheme='basic')),
- ('digest', dict(_auth_scheme='digest')),
- ]
-
- self.scenarios = tests.multiply_scenarios(self.scenarios,
- auth_scheme_scenarios)
-
def load_tests(standard_tests, module, loader):
"""Multiply tests for http clients and protocol versions."""
+ result = loader.suiteClass()
+ adapter = tests.TestScenarioApplier()
+ remaining_tests = standard_tests
+
# one for each transport
- t_adapter = TransportAdapter()
- t_classes= (TestHttpTransportRegistration,
+ t_tests, remaining_tests = tests.split_suite_by_condition(
+ remaining_tests, tests.condition_isinstance((
+ TestHttpTransportRegistration,
TestHttpTransportUrls,
Test_redirected_to,
- )
- is_testing_for_transports = tests.condition_isinstance(t_classes)
+ )))
+ transport_scenarios = [
+ ('urllib', dict(_transport=_urllib.HttpTransport_urllib,
+ _server=http_server.HttpServer_urllib,
+ _qualified_prefix='http+urllib',)),
+ ]
+ if pycurl_present:
+ transport_scenarios.append(
+ ('pycurl', dict(_transport=PyCurlTransport,
+ _server=http_server.HttpServer_PyCurl,
+ _qualified_prefix='http+pycurl',)))
+ adapter.scenarios = transport_scenarios
+ tests.adapt_tests(t_tests, adapter, result)
# multiplied by one for each protocol version
- tp_adapter = TransportProtocolAdapter()
- tp_classes= (SmartHTTPTunnellingTest,
- TestDoCatchRedirections,
- TestHTTPConnections,
- TestHTTPRedirections,
- TestHTTPSilentRedirections,
- TestLimitedRangeRequestServer,
- TestPost,
- TestProxyHttpServer,
- TestRanges,
- TestSpecificRequestHandler,
- )
- is_also_testing_for_protocols = tests.condition_isinstance(tp_classes)
+ tp_tests, remaining_tests = tests.split_suite_by_condition(
+ remaining_tests, tests.condition_isinstance((
+ SmartHTTPTunnellingTest,
+ TestDoCatchRedirections,
+ TestHTTPConnections,
+ TestHTTPRedirections,
+ TestHTTPSilentRedirections,
+ TestLimitedRangeRequestServer,
+ TestPost,
+ TestProxyHttpServer,
+ TestRanges,
+ TestSpecificRequestHandler,
+ )))
+ protocol_scenarios = [
+ ('HTTP/1.0', dict(_protocol_version='HTTP/1.0')),
+ ('HTTP/1.1', dict(_protocol_version='HTTP/1.1')),
+ ]
+ tp_scenarios = tests.multiply_scenarios(adapter.scenarios,
+ protocol_scenarios)
+ adapter.scenarios = tp_scenarios
+ tests.adapt_tests(tp_tests, adapter, result)
# multiplied by one for each authentication scheme
- tpa_adapter = TransportProtocolAuthenticationAdapter()
- tpa_classes = (TestAuth,
- )
- is_also_testing_for_authentication = tests.condition_isinstance(
- tpa_classes)
-
- result = loader.suiteClass()
- for test_class in tests.iter_suite_tests(standard_tests):
- # Each test class is either standalone or testing for some combination
- # of transport, protocol version, authentication scheme. Use the right
- # adpater (or none) depending on the class.
- if is_testing_for_transports(test_class):
- result.addTests(t_adapter.adapt(test_class))
- elif is_also_testing_for_protocols(test_class):
- result.addTests(tp_adapter.adapt(test_class))
- elif is_also_testing_for_authentication(test_class):
- result.addTests(tpa_adapter.adapt(test_class))
- else:
- result.addTest(test_class)
+ tpa_tests, remaining_tests = tests.split_suite_by_condition(
+ remaining_tests, tests.condition_isinstance((
+ TestAuth,
+ )))
+ auth_scheme_scenarios = [
+ ('basic', dict(_auth_scheme='basic')),
+ ('digest', dict(_auth_scheme='digest')),
+ ]
+ adapter.scenarios = tests.multiply_scenarios(adapter.scenarios,
+ auth_scheme_scenarios)
+ tests.adapt_tests(tpa_tests, adapter, result)
+
+ tpact_tests, remaining_tests = tests.split_suite_by_condition(
+ remaining_tests, tests.condition_isinstance((
+ TestActivity,
+ )))
+ activity_scenarios = [
+ ('http', dict(_activity_server=ActivityHTTPServer)),
+ ]
+ if tests.HTTPSServerFeature.available():
+ activity_scenarios.append(
+ ('https', dict(_activity_server=ActivityHTTPSServer,)))
+ adapter.scenarios = tests.multiply_scenarios(tp_scenarios,
+ activity_scenarios)
+ tests.adapt_tests(tpact_tests, adapter, result)
+
+ # No parametrization for the remaining tests
+ result.addTests(remaining_tests)
+
return result
@@ -1807,3 +1792,211 @@
'https://foo.example.com/foo')
self.assertIsInstance(r, type(t))
self.assertEquals(t._user, r._user)
+
+
+class PredefinedRequestHandler(http_server.TestingHTTPRequestHandler):
+ """Request handler for a unique and pre-defined request.
+
+ The only thing we care about here is how many bytes travel on the wire. But
+ since we want to measure it for a real http client, we have to send it
+ correct responses.
+
+ We expect to receive a *single* request nothing more (and we won't even
+ check what request it is, we just measure the bytes read until an empty
+ line.
+ """
+
+ def handle_one_request(self):
+ tcs = self.server.test_case_server
+ requestline = self.rfile.readline()
+ headers = self.MessageClass(self.rfile, 0)
+ # We just read: the request, the headers, an empty line indicating the
+ # end of the headers.
+ bytes_read = len(requestline)
+ for line in headers.headers:
+ bytes_read += len(line)
+ bytes_read += len('\r\n')
+ if requestline.startswith('POST'):
+ # The body should be a single line (or we don't know where it ends
+ # and we don't want to issue a blocking read)
+ body = self.rfile.readline()
+ bytes_read += len(body)
+ tcs.bytes_read = bytes_read
+
+ # We set the bytes written *before* issuing the write, the client is
+ # supposed to consume every produced byte *before* checking that value.
+
+ # Doing the oppposite may lead to test failure: we may be interrupted
+ # after the write but before updating the value. The client can then
+ # continue and read the value *before* we can update it. And yes,
+ # this has been observed -- vila 20090129
+ tcs.bytes_written = len(tcs.canned_response)
+ self.wfile.write(tcs.canned_response)
+
+
+class ActivityServerMixin(object):
+
+ def __init__(self, protocol_version):
+ super(ActivityServerMixin, self).__init__(
+ request_handler=PredefinedRequestHandler,
+ protocol_version=protocol_version)
+ # Bytes read and written by the server
+ self.bytes_read = 0
+ self.bytes_written = 0
+ self.canned_response = None
+
+
+class ActivityHTTPServer(ActivityServerMixin, http_server.HttpServer):
+ pass
+
+
+if tests.HTTPSServerFeature.available():
+ from bzrlib.tests import https_server
+ class ActivityHTTPSServer(ActivityServerMixin, https_server.HTTPSServer):
+ pass
+
+
+class TestActivity(tests.TestCase):
+ """Test socket activity reporting.
+
+ We use a special purpose server to control the bytes sent and received and
+ be able to predict the activity on the client socket.
+ """
+
+ def setUp(self):
+ tests.TestCase.setUp(self)
+ self.server = self._activity_server(self._protocol_version)
+ self.server.setUp()
+ self.activities = {}
+ def report_activity(t, bytes, direction):
+ count = self.activities.get(direction, 0)
+ count += bytes
+ self.activities[direction] = count
+
+ # We override at class level because constructors may propagate the
+ # bound method and render instance overriding ineffective (an
+ # alternative would be be to define a specific ui factory instead...)
+ self.orig_report_activity = self._transport._report_activity
+ self._transport._report_activity = report_activity
+
+ def tearDown(self):
+ self._transport._report_activity = self.orig_report_activity
+ self.server.tearDown()
+ tests.TestCase.tearDown(self)
+
+ def get_transport(self):
+ return self._transport(self.server.get_url())
+
+ def assertActivitiesMatch(self):
+ self.assertEqual(self.server.bytes_read,
+ self.activities.get('write', 0), 'written bytes')
+ self.assertEqual(self.server.bytes_written,
+ self.activities.get('read', 0), 'read bytes')
+
+ def test_get(self):
+ self.server.canned_response = '''HTTP/1.1 200 OK\r
+Date: Tue, 11 Jul 2006 04:32:56 GMT\r
+Server: Apache/2.0.54 (Fedora)\r
+Last-Modified: Sun, 23 Apr 2006 19:35:20 GMT\r
+ETag: "56691-23-38e9ae00"\r
+Accept-Ranges: bytes\r
+Content-Length: 35\r
+Connection: close\r
+Content-Type: text/plain; charset=UTF-8\r
+\r
+Bazaar-NG meta directory, format 1
+'''
+ t = self.get_transport()
+ self.assertEqual('Bazaar-NG meta directory, format 1\n',
+ t.get('foo/bar').read())
+ self.assertActivitiesMatch()
+
+ def test_has(self):
+ self.server.canned_response = '''HTTP/1.1 200 OK\r
+Server: SimpleHTTP/0.6 Python/2.5.2\r
+Date: Thu, 29 Jan 2009 20:21:47 GMT\r
+Content-type: application/octet-stream\r
+Content-Length: 20\r
+Last-Modified: Thu, 29 Jan 2009 20:21:47 GMT\r
+\r
+'''
+ t = self.get_transport()
+ self.assertTrue(t.has('foo/bar'))
+ self.assertActivitiesMatch()
+
+ def test_readv(self):
+ self.server.canned_response = '''HTTP/1.1 206 Partial Content\r
+Date: Tue, 11 Jul 2006 04:49:48 GMT\r
+Server: Apache/2.0.54 (Fedora)\r
+Last-Modified: Thu, 06 Jul 2006 20:22:05 GMT\r
+ETag: "238a3c-16ec2-805c5540"\r
+Accept-Ranges: bytes\r
+Content-Length: 1534\r
+Connection: close\r
+Content-Type: multipart/byteranges; boundary=418470f848b63279b\r
+\r
+\r
+--418470f848b63279b\r
+Content-type: text/plain; charset=UTF-8\r
+Content-range: bytes 0-254/93890\r
+\r
+mbp at sourcefrog.net-20050309040815-13242001617e4a06
+mbp at sourcefrog.net-20050309040929-eee0eb3e6d1e7627
+mbp at sourcefrog.net-20050309040957-6cad07f466bb0bb8
+mbp at sourcefrog.net-20050309041501-c840e09071de3b67
+mbp at sourcefrog.net-20050309044615-c24a3250be83220a
+\r
+--418470f848b63279b\r
+Content-type: text/plain; charset=UTF-8\r
+Content-range: bytes 1000-2049/93890\r
+\r
+40-fd4ec249b6b139ab
+mbp at sourcefrog.net-20050311063625-07858525021f270b
+mbp at sourcefrog.net-20050311231934-aa3776aff5200bb9
+mbp at sourcefrog.net-20050311231953-73aeb3a131c3699a
+mbp at sourcefrog.net-20050311232353-f5e33da490872c6a
+mbp at sourcefrog.net-20050312071639-0a8f59a34a024ff0
+mbp at sourcefrog.net-20050312073432-b2c16a55e0d6e9fb
+mbp at sourcefrog.net-20050312073831-a47c3335ece1920f
+mbp at sourcefrog.net-20050312085412-13373aa129ccbad3
+mbp at sourcefrog.net-20050313052251-2bf004cb96b39933
+mbp at sourcefrog.net-20050313052856-3edd84094687cb11
+mbp at sourcefrog.net-20050313053233-e30a4f28aef48f9d
+mbp at sourcefrog.net-20050313053853-7c64085594ff3072
+mbp at sourcefrog.net-20050313054757-a86c3f5871069e22
+mbp at sourcefrog.net-20050313061422-418f1f73b94879b9
+mbp at sourcefrog.net-20050313120651-497bd231b19df600
+mbp at sourcefrog.net-20050314024931-eae0170ef25a5d1a
+mbp at sourcefrog.net-20050314025438-d52099f915fe65fc
+mbp at sourcefrog.net-20050314025539-637a636692c055cf
+mbp at sourcefrog.net-20050314025737-55eb441f430ab4ba
+mbp at sourcefrog.net-20050314025901-d74aa93bb7ee8f62
+mbp at source\r
+--418470f848b63279b--\r
+'''
+ t = self.get_transport()
+ # Remember that the request is ignored and that the ranges below
+ # doesn't have to match the canned response.
+ l = list(t.readv('/foo/bar', ((0, 255), (1000, 1050))))
+ self.assertEqual(2, len(l))
+ self.assertActivitiesMatch()
+
+ def test_post(self):
+ self.server.canned_response = '''HTTP/1.1 200 OK\r
+Date: Tue, 11 Jul 2006 04:32:56 GMT\r
+Server: Apache/2.0.54 (Fedora)\r
+Last-Modified: Sun, 23 Apr 2006 19:35:20 GMT\r
+ETag: "56691-23-38e9ae00"\r
+Accept-Ranges: bytes\r
+Content-Length: 35\r
+Connection: close\r
+Content-Type: text/plain; charset=UTF-8\r
+\r
+lalala whatever as long as itsssss
+'''
+ t = self.get_transport()
+ # We must send a single line of body bytes, see
+ # PredefinedRequestHandler.handle_one_request
+ code, f = t._post('abc def end-of-body\n')
+ self.assertEqual('lalala whatever as long as itsssss\n', f.read())
+ self.assertActivitiesMatch()
=== modified file 'bzrlib/transport/http/__init__.py'
--- a/bzrlib/transport/http/__init__.py 2009-01-23 21:22:39 +0000
+++ b/bzrlib/transport/http/__init__.py 2009-01-29 14:27:28 +0000
@@ -123,17 +123,13 @@
:param relpath: The relative path to the file
"""
+ code, response_file = self._get(relpath, None)
# FIXME: some callers want an iterable... One step forward, three steps
# backwards :-/ And not only an iterable, but an iterable that can be
# seeked backwards, so we will never be able to do that. One such
# known client is bzrlib.bundle.serializer.v4.get_bundle_reader. At the
# time of this writing it's even the only known client -- vila20071203
- return StringIO(self.get_bytes(relpath))
-
- def get_bytes(self, relpath):
- """See Transport.get_bytes()."""
- code, response_file = self._get(relpath, None)
- return response_file.read()
+ return StringIO(response_file.read())
def _get(self, relpath, ranges, tail_amount=0):
"""Get a file, or part of a file.
=== modified file 'bzrlib/transport/http/_pycurl.py'
--- a/bzrlib/transport/http/_pycurl.py 2009-01-23 21:22:39 +0000
+++ b/bzrlib/transport/http/_pycurl.py 2009-01-30 00:49:41 +0000
@@ -215,8 +215,8 @@
# The parent class use 0 to minimize the requests, but since we can't
# exploit the results as soon as they are received (pycurl limitation) we'd
- # better issue more requests and provide a more responsive UI do the cost
- # of more latency costs.
+ # better issue more requests and provide a more responsive UI incurring
+ # more latency costs.
# If you modify this, think about modifying the comment in http/__init__.py
# too.
_get_max_size = 4 * 1024 * 1024
@@ -245,8 +245,7 @@
'Server return code %d'
% curl.getinfo(pycurl.HTTP_CODE))
msg = self._parse_headers(header)
- return code, response.handle_response(abspath, code, msg, data,
- report_activity=self._report_activity)
+ return code, response.handle_response(abspath, code, msg, data)
def _parse_headers(self, status_and_headers):
"""Transform the headers provided by curl into an HTTPMessage"""
@@ -286,8 +285,7 @@
data.seek(0)
code = curl.getinfo(pycurl.HTTP_CODE)
msg = self._parse_headers(header)
- return code, response.handle_response(abspath, code, msg, data,
- report_activity=self._report_activity)
+ return code, response.handle_response(abspath, code, msg, data)
def _raise_curl_http_error(self, curl, info=None):
@@ -307,15 +305,28 @@
raise errors.InvalidHttpResponse(
url, 'Unable to handle http code %d%s' % (code,msg))
+ def _debug_cb(self, kind, text):
+ if kind in (pycurl.INFOTYPE_HEADER_IN, pycurl.INFOTYPE_DATA_IN,
+ pycurl.INFOTYPE_SSL_DATA_IN):
+ self._report_activity(len(text), 'read')
+ if (kind == pycurl.INFOTYPE_HEADER_IN
+ and 'http' in debug.debug_flags):
+ mutter('< %s' % text)
+ elif kind in (pycurl.INFOTYPE_HEADER_OUT, pycurl.INFOTYPE_DATA_OUT,
+ pycurl.INFOTYPE_SSL_DATA_OUT):
+ self._report_activity(len(text), 'write')
+ if (kind == pycurl.INFOTYPE_HEADER_OUT
+ and 'http' in debug.debug_flags):
+ mutter('> %s' % text)
+ elif kind == pycurl.INFOTYPE_TEXT and 'http' in debug.debug_flags:
+ mutter('* %s' % text)
+
def _set_curl_options(self, curl):
"""Set options for all requests"""
- if 'http' in debug.debug_flags:
- curl.setopt(pycurl.VERBOSE, 1)
- # pycurl doesn't implement the CURLOPT_STDERR option, so we can't
- # do : curl.setopt(pycurl.STDERR, trace._trace_file)
-
ua_str = 'bzr/%s (pycurl: %s)' % (bzrlib.__version__, pycurl.version)
curl.setopt(pycurl.USERAGENT, ua_str)
+ curl.setopt(pycurl.VERBOSE, 1)
+ curl.setopt(pycurl.DEBUGFUNCTION, self._debug_cb)
if self.cabundle:
curl.setopt(pycurl.CAINFO, self.cabundle)
# Set accepted auth methods
=== modified file 'bzrlib/transport/http/_urllib.py'
--- a/bzrlib/transport/http/_urllib.py 2009-01-23 21:22:39 +0000
+++ b/bzrlib/transport/http/_urllib.py 2009-01-29 14:27:28 +0000
@@ -47,7 +47,8 @@
if _from_transport is not None:
self._opener = _from_transport._opener
else:
- self._opener = self._opener_class()
+ self._opener = self._opener_class(
+ report_activity=self._report_activity)
def _perform(self, request):
"""Send the request to the server and handles common errors.
@@ -100,7 +101,6 @@
def _get(self, relpath, offsets, tail_amount=0):
"""See HttpTransport._get"""
-
abspath = self._remote_path(relpath)
headers = {}
accepted_errors = [200, 404]
@@ -126,8 +126,7 @@
raise errors.InvalidHttpRange(abspath, range_header,
'Server return code %d' % code)
- data = handle_response(abspath, code, response.info(), response,
- report_activity=self._report_activity)
+ data = handle_response(abspath, code, response.info(), response)
return code, data
def _post(self, body_bytes):
@@ -137,8 +136,7 @@
response = self._perform(Request('POST', abspath, body_bytes,
accepted_errors=[200, 403]))
code = response.code
- data = handle_response(abspath, code, response.info(), response,
- report_activity=self._report_activity)
+ data = handle_response(abspath, code, response.info(), response)
return code, data
def _head(self, relpath):
=== modified file 'bzrlib/transport/http/_urllib2_wrappers.py'
--- a/bzrlib/transport/http/_urllib2_wrappers.py 2009-01-08 16:57:10 +0000
+++ b/bzrlib/transport/http/_urllib2_wrappers.py 2009-01-29 14:27:28 +0000
@@ -67,13 +67,54 @@
)
-class _BufferedMakefileSocket(object):
-
- def __init__(self, sock):
+class _ReportingFileSocket(object):
+
+ def __init__(self, filesock, report_activity=None):
+ self.filesock = filesock
+ self._report_activity = report_activity
+
+
+ def read(self, size=1):
+ s = self.filesock.read(size)
+ self._report_activity(len(s), 'read')
+ return s
+
+ def readline(self, size=-1):
+ s = self.filesock.readline(size)
+ self._report_activity(len(s), 'read')
+ return s
+
+ def __getattr__(self, name):
+ return getattr(self.filesock, name)
+
+
+class _ReportingSocket(object):
+
+ def __init__(self, sock, report_activity=None):
self.sock = sock
+ self._report_activity = report_activity
+
+ def send(self, s, *args):
+ self.sock.send(s, *args)
+ self._report_activity(len(s), 'write')
+
+ def sendall(self, s, *args):
+ self.sock.send(s, *args)
+ self._report_activity(len(s), 'write')
+
+ def recv(self, *args):
+ s = self.sock.recv(*args)
+ self._report_activity(len(s), 'read')
+ return s
def makefile(self, mode='r', bufsize=-1):
- return self.sock.makefile(mode, 65536)
+ # httplib creates a fileobject that doesn't do buffering, which
+ # makes fp.readline() very expensive because it only reads one byte
+ # at a time. So we wrap the socket in an object that forces
+ # sock.makefile to make a buffered file.
+ fsock = self.sock.makefile(mode, 65536)
+ # And wrap that into a reporting kind of fileobject
+ return _ReportingFileSocket(fsock, self._report_activity)
def __getattr__(self, name):
return getattr(self.sock, name)
@@ -96,14 +137,6 @@
# 8k chunks should be fine.
_discarded_buf_size = 8192
- def __init__(self, sock, *args, **kwargs):
- # httplib creates a fileobject that doesn't do buffering, which
- # makes fp.readline() very expensive because it only reads one byte
- # at a time. So we wrap the socket in an object that forces
- # sock.makefile to make a buffered file.
- sock = _BufferedMakefileSocket(sock)
- httplib.HTTPResponse.__init__(self, sock, *args, **kwargs)
-
def begin(self):
"""Begin to read the response from the server.
@@ -178,8 +211,10 @@
# we want to warn. But not below a given thresold.
_range_warning_thresold = 1024 * 1024
- def __init__(self):
+ def __init__(self,
+ report_activity=None):
self._response = None
+ self._report_activity = report_activity
self._ranges_received_whole_file = None
def _mutter_connect(self):
@@ -216,12 +251,17 @@
# Restore our preciousss
self.sock = sock
+ def _wrap_socket_for_reporting(self, sock):
+ """Wrap the socket before anybody use it."""
+ self.sock = _ReportingSocket(sock, self._report_activity)
+
class HTTPConnection(AbstractHTTPConnection, httplib.HTTPConnection):
# XXX: Needs refactoring at the caller level.
- def __init__(self, host, port=None, proxied_host=None):
- AbstractHTTPConnection.__init__(self)
+ def __init__(self, host, port=None, proxied_host=None,
+ report_activity=None):
+ AbstractHTTPConnection.__init__(self, report_activity=report_activity)
# Use strict=True since we don't support HTTP/0.9
httplib.HTTPConnection.__init__(self, host, port, strict=True)
self.proxied_host = proxied_host
@@ -230,6 +270,7 @@
if 'http' in debug.debug_flags:
self._mutter_connect()
httplib.HTTPConnection.connect(self)
+ self._wrap_socket_for_reporting(self.sock)
# Build the appropriate socket wrapper for ssl
@@ -248,8 +289,9 @@
class HTTPSConnection(AbstractHTTPConnection, httplib.HTTPSConnection):
def __init__(self, host, port=None, key_file=None, cert_file=None,
- proxied_host=None):
- AbstractHTTPConnection.__init__(self)
+ proxied_host=None,
+ report_activity=None):
+ AbstractHTTPConnection.__init__(self, report_activity=report_activity)
# Use strict=True since we don't support HTTP/0.9
httplib.HTTPSConnection.__init__(self, host, port,
key_file, cert_file, strict=True)
@@ -259,11 +301,14 @@
if 'http' in debug.debug_flags:
self._mutter_connect()
httplib.HTTPConnection.connect(self)
+ self._wrap_socket_for_reporting(self.sock)
if self.proxied_host is None:
self.connect_to_origin()
def connect_to_origin(self):
- self.sock = _ssl_wrap_socket(self.sock, self.key_file, self.cert_file)
+ ssl_sock = _ssl_wrap_socket(self.sock, self.key_file, self.cert_file)
+ # Wrap the ssl socket before anybody use it
+ self._wrap_socket_for_reporting(ssl_sock)
class Request(urllib2.Request):
@@ -355,6 +400,9 @@
handler_order = 1000 # after all pre-processings
+ def __init__(self, report_activity=None):
+ self._report_activity = report_activity
+
def create_connection(self, request, http_connection_class):
host = request.get_host()
if not host:
@@ -366,7 +414,8 @@
# request is made)
try:
connection = http_connection_class(
- host, proxied_host=request.proxied_host)
+ host, proxied_host=request.proxied_host,
+ report_activity=self._report_activity)
except httplib.InvalidURL, exception:
# There is only one occurrence of InvalidURL in httplib
raise errors.InvalidURL(request.get_full_url(),
@@ -1370,9 +1419,11 @@
def __init__(self,
connection=ConnectionHandler,
redirect=HTTPRedirectHandler,
- error=HTTPErrorProcessor,):
- self._opener = urllib2.build_opener( \
- connection, redirect, error,
+ error=HTTPErrorProcessor,
+ report_activity=None):
+ self._opener = urllib2.build_opener(
+ connection(report_activity=report_activity),
+ redirect, error,
ProxyHandler(),
HTTPBasicAuthHandler(),
HTTPDigestAuthHandler(),
=== modified file 'bzrlib/transport/http/response.py'
--- a/bzrlib/transport/http/response.py 2009-01-23 21:22:39 +0000
+++ b/bzrlib/transport/http/response.py 2009-01-30 00:49:41 +0000
@@ -67,18 +67,15 @@
# maximum size of read requests -- used to avoid MemoryError issues in recv
_max_read_size = 512 * 1024
- def __init__(self, path, infile, report_activity=None):
+ def __init__(self, path, infile):
"""Constructor.
:param path: File url, for error reports.
:param infile: File-like socket set at body start.
- :param report_activity: A Transport._report_activity function to call
- as bytes are read.
"""
self._path = path
self._file = infile
self._boundary = None
- self._report_activity = report_activity
# When using multi parts response, this will be set with the headers
# associated with the range currently read.
self._headers = None
@@ -231,8 +228,7 @@
limited = self._start + self._size - self._pos
if size >= 0:
limited = min(limited, size)
- osutils.pumpfile(self._file, buffer, limited, self._max_read_size,
- report_activity=self._report_activity, direction='read')
+ osutils.pumpfile(self._file, buffer, limited, self._max_read_size)
data = buffer.getvalue()
# Update _pos respecting the data effectively read
@@ -281,7 +277,7 @@
return self._pos
-def handle_response(url, code, msg, data, report_activity=None):
+def handle_response(url, code, msg, data):
"""Interpret the code & headers and wrap the provided data in a RangeFile.
This is a factory method which returns an appropriate RangeFile based on
@@ -295,7 +291,7 @@
:return: A file-like object that can seek()+read() the
ranges indicated by the headers.
"""
- rfile = RangeFile(url, data, report_activity=report_activity)
+ rfile = RangeFile(url, data)
if code == 200:
# A whole file
size = msg.getheader('content-length', None)
More information about the bazaar-commits
mailing list