Rev 3983: Progress bar at socket level for http in http://bazaar.launchpad.net/%7Evila/bzr/bzr.integration

Vincent Ladeuil v.ladeuil+lp at free.fr
Thu Feb 5 10:21:35 GMT 2009


At http://bazaar.launchpad.net/%7Evila/bzr/bzr.integration

------------------------------------------------------------
revno: 3983
revision-id: v.ladeuil+lp at free.fr-20090205102051-gyivrlud1hk1c12g
parent: pqm at pqm.ubuntu.com-20090205054228-3qyiv92vtgs94e0c
parent: v.ladeuil+lp at free.fr-20090130004941-820fpd2ryyo127vv
committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
branch nick: bzr.integration
timestamp: Thu 2009-02-05 11:20:51 +0100
message:
  Progress bar at socket level for http
modified:
  NEWS                           NEWS-20050323055033-4e00b5db738777ff
  bzrlib/tests/https_server.py   https_server.py-20071121173708-aj8zczi0ziwbwz21-1
  bzrlib/tests/test_http.py      testhttp.py-20051018020158-b2eef6e867c514d9
  bzrlib/transport/http/__init__.py http_transport.py-20050711212304-506c5fd1059ace96
  bzrlib/transport/http/_pycurl.py pycurlhttp.py-20060110060940-4e2a705911af77a6
  bzrlib/transport/http/_urllib.py _urlgrabber.py-20060113083826-0bbf7d992fbf090c
  bzrlib/transport/http/_urllib2_wrappers.py _urllib2_wrappers.py-20060913231729-ha9ugi48ktx481ao-1
  bzrlib/transport/http/response.py _response.py-20060613154423-a2ci7hd4iw5c7fnt-1
    ------------------------------------------------------------
    revno: 3945.1.8
    revision-id: v.ladeuil+lp at free.fr-20090130004941-820fpd2ryyo127vv
    parent: v.ladeuil+lp at free.fr-20090129193101-w0008aa1ufoe7cmd
    committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
    branch nick: pb-http
    timestamp: Fri 2009-01-30 01:49:41 +0100
    message:
      Add more tests, fix pycurl double handling, revert previous tracking.
      
      * bzrlib/tests/test_http.py:
      (PredefinedRequestHandler): Renamed from
      PreRecordedRequestHandler.
      (PredefinedRequestHandler.handle_one_request): Get the canned
      response from the test server directly.
      (ActivityServerMixin): Make it a true object and intialize the
      attributes in the constructor. Tests can now set the
      canned_response attribute before querying the server.
      (TestActivity.setUp, TestActivity.tearDown,
      TestActivity.get_transport, TestActivity.assertActivitiesMatch):
      Extracted from test_get to be able to write other tests.
      (TestActivity.test_has, TestActivity.test_readv,
      TestActivity.test_post): New tests, all cases should be covered
      now.
      
      * bzrlib/transport/http/response.py:
      (RangeFile.__init__, RangeFile.read, handle_response): Revert
      previous tracking, both http implementations can now report
      activity from the socket.
      
      * bzrlib/transport/http/_pycurl.py:
      (PyCurlTransport._get_ranged, PyCurlTransport._post): Revert
      previous tracking.
    modified:
      BRANCH.TODO                    BRANCH.TODO-20060103052123-79ac4969351c03a9
      NEWS                           NEWS-20050323055033-4e00b5db738777ff
      bzrlib/tests/test_http.py      testhttp.py-20051018020158-b2eef6e867c514d9
      bzrlib/transport/http/_pycurl.py pycurlhttp.py-20060110060940-4e2a705911af77a6
      bzrlib/transport/http/response.py _response.py-20060613154423-a2ci7hd4iw5c7fnt-1
    ------------------------------------------------------------
    revno: 3945.1.7
    revision-id: v.ladeuil+lp at free.fr-20090129193101-w0008aa1ufoe7cmd
    parent: v.ladeuil+lp at free.fr-20090129165458-o52vbf2nl4dxadoc
    committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
    branch nick: pb-http
    timestamp: Thu 2009-01-29 20:31:01 +0100
    message:
      Test against https.
      
      * bzrlib/tests/test_http.py:
      (load_tests): Get rid of TestAdapter daughter classes, they are
      useless. Add parametrization against http/https for activity
      tests.
      (ActivityServerMixin, ActivityHTTPServer, ActivityHTTPSServer):
      Make HTTP[S] servers.
      (PreRecordedRequestHandler.handle_one_request): Be defensive
      against threads being non-deterministic.
      (TestActivity.test_http_get): Use parametrized test server.
      
      * bzrlib/tests/https_server.py:
      (HTTPSServer.__init__): Add protocol_version parameter for tests
      purposes.
    modified:
      BRANCH.TODO                    BRANCH.TODO-20060103052123-79ac4969351c03a9
      bzrlib/tests/https_server.py   https_server.py-20071121173708-aj8zczi0ziwbwz21-1
      bzrlib/tests/test_http.py      testhttp.py-20051018020158-b2eef6e867c514d9
    ------------------------------------------------------------
    revno: 3945.1.6
    revision-id: v.ladeuil+lp at free.fr-20090129165458-o52vbf2nl4dxadoc
    parent: v.ladeuil+lp at free.fr-20090129142728-1iu8017zgso23w0i
    committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
    branch nick: pb-http
    timestamp: Thu 2009-01-29 17:54:58 +0100
    message:
      Fix debug handling for pycurl and implement pycurl http activity
      reporting at socket level.
      
      * bzrlib/transport/http/_pycurl.py:
      (PyCurlTransport._debug_cb): Two birds with one stone: correctly
      handle the debug traces with mutter (no more stderr pollution !)
      and report activity for all the transmitted headers and data in
      both directions.
      (PyCurlTransport._set_curl_options): Setup a pycurl debug callback
      for both activity reporting and http debug flag handling.
    modified:
      BRANCH.TODO                    BRANCH.TODO-20060103052123-79ac4969351c03a9
      NEWS                           NEWS-20050323055033-4e00b5db738777ff
      bzrlib/transport/http/_pycurl.py pycurlhttp.py-20060110060940-4e2a705911af77a6
    ------------------------------------------------------------
    revno: 3945.1.5
    revision-id: v.ladeuil+lp at free.fr-20090129142728-1iu8017zgso23w0i
    parent: v.ladeuil+lp at free.fr-20090126081302-ad8l8nhn2sscymxt
    committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
    branch nick: pb-http
    timestamp: Thu 2009-01-29 15:27:28 +0100
    message:
      Start implementing http activity reporting at socket level.
      
      * bzrlib/transport/http/_urllib2_wrappers.py:
      (_ReportingFileSocket, _ReportingSocket): Wrappers for
      socket._socket and socket_filesocket objects providing activity
      reporting.
      (Response.__init__): Deleted. Wrapping the file socket object is
      now done handled at connection time.
      (AbstractHTTPConnection._wrap_socket_for_reporting): Wrap the
      underlying socket into a reporting one.
      (HTTPConnection.__init__, HTTPSConnection.__init__): Accept a
      report_activity parameter.
      (HTTPSConnection.connect): Wrap the newly created socket.
      (HTTPSConnection.connect_to_origin): Wrap the newly created *or*
      connected ssl socket.
      (ConnectionHandler.__init__): Accept a report_activity parameter.
      (ConnectionHandler.create_connection): Pass the report_activity
      parameter to all created connections.
      (Opener.__init__): Accept a report_activity parameter.
      
      * bzrlib/transport/http/_urllib.py:
      (HttpTransport_urllib.__init__): Provide the report activity
      function to the opener.
      (HttpTransport_urllib._get, HttpTransport_urllib._post): Don't use
      the RangeFile report activity facility anymore.
      
      * bzrlib/transport/http/__init__.py:
      (HttpTransportBase.get_bytes): Deleted, this gratuitously add a
      useless level of buffering.
      
      * bzrlib/tests/test_http.py:
      (ActivityHttpServer, PreRecoredRequestHandler, TestActivity): Test
      activity reporting by http clients.
    modified:
      BRANCH.TODO                    BRANCH.TODO-20060103052123-79ac4969351c03a9
      bzrlib/tests/test_http.py      testhttp.py-20051018020158-b2eef6e867c514d9
      bzrlib/transport/http/__init__.py http_transport.py-20050711212304-506c5fd1059ace96
      bzrlib/transport/http/_urllib.py _urlgrabber.py-20060113083826-0bbf7d992fbf090c
      bzrlib/transport/http/_urllib2_wrappers.py _urllib2_wrappers.py-20060913231729-ha9ugi48ktx481ao-1
    ------------------------------------------------------------
    revno: 3945.1.4
    revision-id: v.ladeuil+lp at free.fr-20090126081302-ad8l8nhn2sscymxt
    parent: v.ladeuil+lp at free.fr-20090122112003-pcp5bu8pfc41u7iz
    parent: v.ladeuil+lp at free.fr-20090126081250-m310ky1gjotwcw37
    committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
    branch nick: pb-http
    timestamp: Mon 2009-01-26 09:13:02 +0100
    message:
      Merge emacs-ui-fixes into pb-http
    added:
      bzrlib/tests/branch_implementations/test_dotted_revno_to_revision_id.py test_dotted_revno_to-20090121014844-6x7d9jtri5sspg1o-1
      bzrlib/tests/branch_implementations/test_iter_merge_sorted_revisions.py test_merge_sorted_re-20090121004847-to3gvjwigstu93eh-1
      bzrlib/tests/branch_implementations/test_revision_id_to_dotted_revno.py test_revision_id_to_-20090122052032-g3czslif6sdqfkh3-1
    modified:
      NEWS                           NEWS-20050323055033-4e00b5db738777ff
      bzrlib/branch.py               branch.py-20050309040759-e4baf4e0d046576e
      bzrlib/help_topics/__init__.py help_topics.py-20060920210027-rnim90q9e0bwxvy4-1
      bzrlib/help_topics/en/rules.txt rules.txt-20080516063844-ghr5l6pvvrhiycun-1
      bzrlib/knit.py                 knit.py-20051212171256-f056ac8f0fbe1bd9
      bzrlib/log.py                  log.py-20050505065812-c40ce11702fe5fb1
      bzrlib/merge.py                merge.py-20050513021216-953b65a438527106
      bzrlib/osutils.py              osutils.py-20050309040759-eeaff12fbf77ac86
      bzrlib/progress.py             progress.py-20050610070202-df9faaab791964c0
      bzrlib/remote.py               remote.py-20060720103555-yeeg2x51vn0rbtdp-1
      bzrlib/revisionspec.py         revisionspec.py-20050907152633-17567659fd5c0ddb
      bzrlib/rules.py                properties.py-20080506032617-9k06uqalkf09ck0z-1
      bzrlib/tests/branch_implementations/__init__.py __init__.py-20060123013057-b12a52c3f361daf4
      bzrlib/tests/test_knit.py      test_knit.py-20051212171302-95d4c00dd5f11f2b
      bzrlib/tests/test_log.py       testlog.py-20050728115707-1a514809d7d49309
      bzrlib/tests/test_merge.py     testmerge.py-20050905070950-c1b5aa49ff911024
      bzrlib/tests/test_osutils.py   test_osutils.py-20051201224856-e48ee24c12182989
      bzrlib/tests/test_progress.py  test_progress.py-20060308160359-978c397bc79b7fda
      bzrlib/tests/test_rules.py     test_properties.py-20080506033501-3p9kmuob25dho8xl-1
      bzrlib/tests/test_ui.py        test_ui.py-20051130162854-458e667a7414af09
      bzrlib/transport/http/__init__.py http_transport.py-20050711212304-506c5fd1059ace96
      bzrlib/transport/http/_pycurl.py pycurlhttp.py-20060110060940-4e2a705911af77a6
      bzrlib/transport/http/_urllib.py _urlgrabber.py-20060113083826-0bbf7d992fbf090c
      bzrlib/transport/http/response.py _response.py-20060613154423-a2ci7hd4iw5c7fnt-1
      bzrlib/transport/sftp.py       sftp.py-20051019050329-ab48ce71b7e32dfe
      bzrlib/ui/__init__.py          ui.py-20050824083933-8cf663c763ba53a9
      bzrlib/ui/text.py              text.py-20051130153916-2e438cffc8afc478
    ------------------------------------------------------------
    revno: 3945.2.1
    revision-id: v.ladeuil+lp at free.fr-20090126081250-m310ky1gjotwcw37
    parent: v.ladeuil+lp at free.fr-20090119130947-1ldks301mpsymf8r
    parent: pqm at pqm.ubuntu.com-20090124185051-8oryvqq68n6repso
    committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
    branch nick: emacs-ui-fixes
    timestamp: Mon 2009-01-26 09:12:50 +0100
    message:
      Merge bzr.dev into emacs-ui-fixes
    added:
      bzrlib/tests/branch_implementations/test_dotted_revno_to_revision_id.py test_dotted_revno_to-20090121014844-6x7d9jtri5sspg1o-1
      bzrlib/tests/branch_implementations/test_iter_merge_sorted_revisions.py test_merge_sorted_re-20090121004847-to3gvjwigstu93eh-1
      bzrlib/tests/branch_implementations/test_revision_id_to_dotted_revno.py test_revision_id_to_-20090122052032-g3czslif6sdqfkh3-1
    modified:
      NEWS                           NEWS-20050323055033-4e00b5db738777ff
      bzrlib/branch.py               branch.py-20050309040759-e4baf4e0d046576e
      bzrlib/help_topics/__init__.py help_topics.py-20060920210027-rnim90q9e0bwxvy4-1
      bzrlib/help_topics/en/rules.txt rules.txt-20080516063844-ghr5l6pvvrhiycun-1
      bzrlib/knit.py                 knit.py-20051212171256-f056ac8f0fbe1bd9
      bzrlib/log.py                  log.py-20050505065812-c40ce11702fe5fb1
      bzrlib/merge.py                merge.py-20050513021216-953b65a438527106
      bzrlib/osutils.py              osutils.py-20050309040759-eeaff12fbf77ac86
      bzrlib/progress.py             progress.py-20050610070202-df9faaab791964c0
      bzrlib/remote.py               remote.py-20060720103555-yeeg2x51vn0rbtdp-1
      bzrlib/revisionspec.py         revisionspec.py-20050907152633-17567659fd5c0ddb
      bzrlib/rules.py                properties.py-20080506032617-9k06uqalkf09ck0z-1
      bzrlib/tests/branch_implementations/__init__.py __init__.py-20060123013057-b12a52c3f361daf4
      bzrlib/tests/test_knit.py      test_knit.py-20051212171302-95d4c00dd5f11f2b
      bzrlib/tests/test_log.py       testlog.py-20050728115707-1a514809d7d49309
      bzrlib/tests/test_merge.py     testmerge.py-20050905070950-c1b5aa49ff911024
      bzrlib/tests/test_osutils.py   test_osutils.py-20051201224856-e48ee24c12182989
      bzrlib/tests/test_progress.py  test_progress.py-20060308160359-978c397bc79b7fda
      bzrlib/tests/test_rules.py     test_properties.py-20080506033501-3p9kmuob25dho8xl-1
      bzrlib/tests/test_ui.py        test_ui.py-20051130162854-458e667a7414af09
      bzrlib/transport/http/__init__.py http_transport.py-20050711212304-506c5fd1059ace96
      bzrlib/transport/http/_pycurl.py pycurlhttp.py-20060110060940-4e2a705911af77a6
      bzrlib/transport/http/_urllib.py _urlgrabber.py-20060113083826-0bbf7d992fbf090c
      bzrlib/transport/http/response.py _response.py-20060613154423-a2ci7hd4iw5c7fnt-1
      bzrlib/transport/sftp.py       sftp.py-20051019050329-ab48ce71b7e32dfe
      bzrlib/ui/__init__.py          ui.py-20050824083933-8cf663c763ba53a9
      bzrlib/ui/text.py              text.py-20051130153916-2e438cffc8afc478
-------------- next part --------------
=== modified file 'NEWS'
--- a/NEWS	2009-02-04 04:19:15 +0000
+++ b/NEWS	2009-02-05 10:20:51 +0000
@@ -59,6 +59,11 @@
     * Progress bars now show the rate of activity for some sftp 
       operations, and they are drawn different.  (Martin Pool, #172741)
 
+    * Progress bars now show the rate of activity for urllib and pycurl based
+      http client implementations. The operations are tracked at the socket
+      level for better precision.
+      (Vincent Ladeuil)
+
     * Rule-based preferences can now accept multiple patterns for a set of
       rules.  (Marius Kruger)
 

=== modified file 'bzrlib/tests/https_server.py'
--- a/bzrlib/tests/https_server.py	2009-01-08 15:28:17 +0000
+++ b/bzrlib/tests/https_server.py	2009-01-29 19:31:01 +0000
@@ -75,9 +75,11 @@
     # Provides usable defaults since an https server requires both a
     # private key and certificate to work.
     def __init__(self, request_handler=http_server.TestingHTTPRequestHandler,
+                 protocol_version=None,
                  key_file=ssl_certs.build_path('server_without_pass.key'),
                  cert_file=ssl_certs.build_path('server.crt')):
-        http_server.HttpServer.__init__(self, request_handler)
+        http_server.HttpServer.__init__(self, request_handler=request_handler,
+                                        protocol_version=protocol_version)
         self.key_file = key_file
         self.cert_file = cert_file
         self.temp_files = []

=== modified file 'bzrlib/tests/test_http.py'
--- a/bzrlib/tests/test_http.py	2009-01-08 16:57:10 +0000
+++ b/bzrlib/tests/test_http.py	2009-01-30 00:49:41 +0000
@@ -65,100 +65,85 @@
     pycurl_present = False
 
 
-class TransportAdapter(tests.TestScenarioApplier):
-    """Generate the same test for each transport implementation."""
-
-    def __init__(self):
-        transport_scenarios = [
-            ('urllib', dict(_transport=_urllib.HttpTransport_urllib,
-                            _server=http_server.HttpServer_urllib,
-                            _qualified_prefix='http+urllib',)),
-            ]
-        if pycurl_present:
-            transport_scenarios.append(
-                ('pycurl', dict(_transport=PyCurlTransport,
-                                _server=http_server.HttpServer_PyCurl,
-                                _qualified_prefix='http+pycurl',)))
-        self.scenarios = transport_scenarios
-
-
-class TransportProtocolAdapter(TransportAdapter):
-    """Generate the same test for each protocol implementation.
-
-    In addition to the transport adaptatation that we inherit from.
-    """
-
-    def __init__(self):
-        super(TransportProtocolAdapter, self).__init__()
-        protocol_scenarios = [
-            ('HTTP/1.0',  dict(_protocol_version='HTTP/1.0')),
-            ('HTTP/1.1',  dict(_protocol_version='HTTP/1.1')),
-            ]
-        self.scenarios = tests.multiply_scenarios(self.scenarios,
-                                                  protocol_scenarios)
-
-
-class TransportProtocolAuthenticationAdapter(TransportProtocolAdapter):
-    """Generate the same test for each authentication scheme implementation.
-
-    In addition to the protocol adaptatation that we inherit from.
-    """
-
-    def __init__(self):
-        super(TransportProtocolAuthenticationAdapter, self).__init__()
-        auth_scheme_scenarios = [
-            ('basic', dict(_auth_scheme='basic')),
-            ('digest', dict(_auth_scheme='digest')),
-            ]
-
-        self.scenarios = tests.multiply_scenarios(self.scenarios,
-                                                  auth_scheme_scenarios)
-
 def load_tests(standard_tests, module, loader):
     """Multiply tests for http clients and protocol versions."""
+    result = loader.suiteClass()
+    adapter = tests.TestScenarioApplier()
+    remaining_tests = standard_tests
+
     # one for each transport
-    t_adapter = TransportAdapter()
-    t_classes= (TestHttpTransportRegistration,
+    t_tests, remaining_tests = tests.split_suite_by_condition(
+        remaining_tests, tests.condition_isinstance((
+                TestHttpTransportRegistration,
                 TestHttpTransportUrls,
                 Test_redirected_to,
-                )
-    is_testing_for_transports = tests.condition_isinstance(t_classes)
+                )))
+    transport_scenarios = [
+        ('urllib', dict(_transport=_urllib.HttpTransport_urllib,
+                        _server=http_server.HttpServer_urllib,
+                        _qualified_prefix='http+urllib',)),
+        ]
+    if pycurl_present:
+        transport_scenarios.append(
+            ('pycurl', dict(_transport=PyCurlTransport,
+                            _server=http_server.HttpServer_PyCurl,
+                            _qualified_prefix='http+pycurl',)))
+    adapter.scenarios = transport_scenarios
+    tests.adapt_tests(t_tests, adapter, result)
 
     # multiplied by one for each protocol version
-    tp_adapter = TransportProtocolAdapter()
-    tp_classes= (SmartHTTPTunnellingTest,
-                 TestDoCatchRedirections,
-                 TestHTTPConnections,
-                 TestHTTPRedirections,
-                 TestHTTPSilentRedirections,
-                 TestLimitedRangeRequestServer,
-                 TestPost,
-                 TestProxyHttpServer,
-                 TestRanges,
-                 TestSpecificRequestHandler,
-                 )
-    is_also_testing_for_protocols = tests.condition_isinstance(tp_classes)
+    tp_tests, remaining_tests = tests.split_suite_by_condition(
+        remaining_tests, tests.condition_isinstance((
+                SmartHTTPTunnellingTest,
+                TestDoCatchRedirections,
+                TestHTTPConnections,
+                TestHTTPRedirections,
+                TestHTTPSilentRedirections,
+                TestLimitedRangeRequestServer,
+                TestPost,
+                TestProxyHttpServer,
+                TestRanges,
+                TestSpecificRequestHandler,
+                )))
+    protocol_scenarios = [
+            ('HTTP/1.0',  dict(_protocol_version='HTTP/1.0')),
+            ('HTTP/1.1',  dict(_protocol_version='HTTP/1.1')),
+            ]
+    tp_scenarios = tests.multiply_scenarios(adapter.scenarios,
+                                            protocol_scenarios)
+    adapter.scenarios = tp_scenarios
+    tests.adapt_tests(tp_tests, adapter, result)
 
     # multiplied by one for each authentication scheme
-    tpa_adapter = TransportProtocolAuthenticationAdapter()
-    tpa_classes = (TestAuth,
-                   )
-    is_also_testing_for_authentication = tests.condition_isinstance(
-        tpa_classes)
-
-    result = loader.suiteClass()
-    for test_class in tests.iter_suite_tests(standard_tests):
-        # Each test class is either standalone or testing for some combination
-        # of transport, protocol version, authentication scheme. Use the right
-        # adpater (or none) depending on the class.
-        if is_testing_for_transports(test_class):
-            result.addTests(t_adapter.adapt(test_class))
-        elif is_also_testing_for_protocols(test_class):
-            result.addTests(tp_adapter.adapt(test_class))
-        elif is_also_testing_for_authentication(test_class):
-            result.addTests(tpa_adapter.adapt(test_class))
-        else:
-            result.addTest(test_class)
+    tpa_tests, remaining_tests = tests.split_suite_by_condition(
+        remaining_tests, tests.condition_isinstance((
+                TestAuth,
+                )))
+    auth_scheme_scenarios = [
+        ('basic', dict(_auth_scheme='basic')),
+        ('digest', dict(_auth_scheme='digest')),
+        ]
+    adapter.scenarios = tests.multiply_scenarios(adapter.scenarios,
+                                                 auth_scheme_scenarios)
+    tests.adapt_tests(tpa_tests, adapter, result)
+
+    tpact_tests, remaining_tests = tests.split_suite_by_condition(
+        remaining_tests, tests.condition_isinstance((
+                TestActivity,
+                )))
+    activity_scenarios = [
+        ('http', dict(_activity_server=ActivityHTTPServer)),
+        ]
+    if tests.HTTPSServerFeature.available():
+        activity_scenarios.append(
+            ('https', dict(_activity_server=ActivityHTTPSServer,)))
+    adapter.scenarios = tests.multiply_scenarios(tp_scenarios,
+                                                 activity_scenarios)
+    tests.adapt_tests(tpact_tests, adapter, result)
+
+    # No parametrization for the remaining tests
+    result.addTests(remaining_tests)
+
     return result
 
 
@@ -1807,3 +1792,211 @@
                              'https://foo.example.com/foo')
         self.assertIsInstance(r, type(t))
         self.assertEquals(t._user, r._user)
+
+
+class PredefinedRequestHandler(http_server.TestingHTTPRequestHandler):
+    """Request handler for a unique and pre-defined request.
+
+    The only thing we care about here is how many bytes travel on the wire. But
+    since we want to measure it for a real http client, we have to send it
+    correct responses.
+
+    We expect to receive a *single* request nothing more (and we won't even
+    check what request it is, we just measure the bytes read until an empty
+    line.
+    """
+
+    def handle_one_request(self):
+        tcs = self.server.test_case_server
+        requestline = self.rfile.readline()
+        headers = self.MessageClass(self.rfile, 0)
+        # We just read: the request, the headers, an empty line indicating the
+        # end of the headers.
+        bytes_read = len(requestline)
+        for line in headers.headers:
+            bytes_read += len(line)
+        bytes_read += len('\r\n')
+        if requestline.startswith('POST'):
+            # The body should be a single line (or we don't know where it ends
+            # and we don't want to issue a blocking read)
+            body = self.rfile.readline()
+            bytes_read += len(body)
+        tcs.bytes_read = bytes_read
+
+        # We set the bytes written *before* issuing the write, the client is
+        # supposed to consume every produced byte *before* checking that value.
+
+        # Doing the oppposite may lead to test failure: we may be interrupted
+        # after the write but before updating the value. The client can then
+        # continue and read the value *before* we can update it. And yes,
+        # this has been observed -- vila 20090129
+        tcs.bytes_written = len(tcs.canned_response)
+        self.wfile.write(tcs.canned_response)
+
+
+class ActivityServerMixin(object):
+
+    def __init__(self, protocol_version):
+        super(ActivityServerMixin, self).__init__(
+            request_handler=PredefinedRequestHandler,
+            protocol_version=protocol_version)
+        # Bytes read and written by the server
+        self.bytes_read = 0
+        self.bytes_written = 0
+        self.canned_response = None
+
+
+class ActivityHTTPServer(ActivityServerMixin, http_server.HttpServer):
+    pass
+
+
+if tests.HTTPSServerFeature.available():
+    from bzrlib.tests import https_server
+    class ActivityHTTPSServer(ActivityServerMixin, https_server.HTTPSServer):
+        pass
+
+
+class TestActivity(tests.TestCase):
+    """Test socket activity reporting.
+
+    We use a special purpose server to control the bytes sent and received and
+    be able to predict the activity on the client socket.
+    """
+
+    def setUp(self):
+        tests.TestCase.setUp(self)
+        self.server = self._activity_server(self._protocol_version)
+        self.server.setUp()
+        self.activities = {}
+        def report_activity(t, bytes, direction):
+            count = self.activities.get(direction, 0)
+            count += bytes
+            self.activities[direction] = count
+
+        # We override at class level because constructors may propagate the
+        # bound method and render instance overriding ineffective (an
+        # alternative would be be to define a specific ui factory instead...)
+        self.orig_report_activity = self._transport._report_activity
+        self._transport._report_activity = report_activity
+
+    def tearDown(self):
+        self._transport._report_activity = self.orig_report_activity
+        self.server.tearDown()
+        tests.TestCase.tearDown(self)
+
+    def get_transport(self):
+        return self._transport(self.server.get_url())
+
+    def assertActivitiesMatch(self):
+        self.assertEqual(self.server.bytes_read,
+                         self.activities.get('write', 0), 'written bytes')
+        self.assertEqual(self.server.bytes_written,
+                         self.activities.get('read', 0), 'read bytes')
+
+    def test_get(self):
+        self.server.canned_response = '''HTTP/1.1 200 OK\r
+Date: Tue, 11 Jul 2006 04:32:56 GMT\r
+Server: Apache/2.0.54 (Fedora)\r
+Last-Modified: Sun, 23 Apr 2006 19:35:20 GMT\r
+ETag: "56691-23-38e9ae00"\r
+Accept-Ranges: bytes\r
+Content-Length: 35\r
+Connection: close\r
+Content-Type: text/plain; charset=UTF-8\r
+\r
+Bazaar-NG meta directory, format 1
+'''
+        t = self.get_transport()
+        self.assertEqual('Bazaar-NG meta directory, format 1\n',
+                         t.get('foo/bar').read())
+        self.assertActivitiesMatch()
+
+    def test_has(self):
+        self.server.canned_response = '''HTTP/1.1 200 OK\r
+Server: SimpleHTTP/0.6 Python/2.5.2\r
+Date: Thu, 29 Jan 2009 20:21:47 GMT\r
+Content-type: application/octet-stream\r
+Content-Length: 20\r
+Last-Modified: Thu, 29 Jan 2009 20:21:47 GMT\r
+\r
+'''
+        t = self.get_transport()
+        self.assertTrue(t.has('foo/bar'))
+        self.assertActivitiesMatch()
+
+    def test_readv(self):
+        self.server.canned_response = '''HTTP/1.1 206 Partial Content\r
+Date: Tue, 11 Jul 2006 04:49:48 GMT\r
+Server: Apache/2.0.54 (Fedora)\r
+Last-Modified: Thu, 06 Jul 2006 20:22:05 GMT\r
+ETag: "238a3c-16ec2-805c5540"\r
+Accept-Ranges: bytes\r
+Content-Length: 1534\r
+Connection: close\r
+Content-Type: multipart/byteranges; boundary=418470f848b63279b\r
+\r
+\r
+--418470f848b63279b\r
+Content-type: text/plain; charset=UTF-8\r
+Content-range: bytes 0-254/93890\r
+\r
+mbp at sourcefrog.net-20050309040815-13242001617e4a06
+mbp at sourcefrog.net-20050309040929-eee0eb3e6d1e7627
+mbp at sourcefrog.net-20050309040957-6cad07f466bb0bb8
+mbp at sourcefrog.net-20050309041501-c840e09071de3b67
+mbp at sourcefrog.net-20050309044615-c24a3250be83220a
+\r
+--418470f848b63279b\r
+Content-type: text/plain; charset=UTF-8\r
+Content-range: bytes 1000-2049/93890\r
+\r
+40-fd4ec249b6b139ab
+mbp at sourcefrog.net-20050311063625-07858525021f270b
+mbp at sourcefrog.net-20050311231934-aa3776aff5200bb9
+mbp at sourcefrog.net-20050311231953-73aeb3a131c3699a
+mbp at sourcefrog.net-20050311232353-f5e33da490872c6a
+mbp at sourcefrog.net-20050312071639-0a8f59a34a024ff0
+mbp at sourcefrog.net-20050312073432-b2c16a55e0d6e9fb
+mbp at sourcefrog.net-20050312073831-a47c3335ece1920f
+mbp at sourcefrog.net-20050312085412-13373aa129ccbad3
+mbp at sourcefrog.net-20050313052251-2bf004cb96b39933
+mbp at sourcefrog.net-20050313052856-3edd84094687cb11
+mbp at sourcefrog.net-20050313053233-e30a4f28aef48f9d
+mbp at sourcefrog.net-20050313053853-7c64085594ff3072
+mbp at sourcefrog.net-20050313054757-a86c3f5871069e22
+mbp at sourcefrog.net-20050313061422-418f1f73b94879b9
+mbp at sourcefrog.net-20050313120651-497bd231b19df600
+mbp at sourcefrog.net-20050314024931-eae0170ef25a5d1a
+mbp at sourcefrog.net-20050314025438-d52099f915fe65fc
+mbp at sourcefrog.net-20050314025539-637a636692c055cf
+mbp at sourcefrog.net-20050314025737-55eb441f430ab4ba
+mbp at sourcefrog.net-20050314025901-d74aa93bb7ee8f62
+mbp at source\r
+--418470f848b63279b--\r
+'''
+        t = self.get_transport()
+        # Remember that the request is ignored and that the ranges below
+        # doesn't have to match the canned response.
+        l = list(t.readv('/foo/bar', ((0, 255), (1000, 1050))))
+        self.assertEqual(2, len(l))
+        self.assertActivitiesMatch()
+
+    def test_post(self):
+        self.server.canned_response = '''HTTP/1.1 200 OK\r
+Date: Tue, 11 Jul 2006 04:32:56 GMT\r
+Server: Apache/2.0.54 (Fedora)\r
+Last-Modified: Sun, 23 Apr 2006 19:35:20 GMT\r
+ETag: "56691-23-38e9ae00"\r
+Accept-Ranges: bytes\r
+Content-Length: 35\r
+Connection: close\r
+Content-Type: text/plain; charset=UTF-8\r
+\r
+lalala whatever as long as itsssss
+'''
+        t = self.get_transport()
+        # We must send a single line of body bytes, see
+        # PredefinedRequestHandler.handle_one_request
+        code, f = t._post('abc def end-of-body\n')
+        self.assertEqual('lalala whatever as long as itsssss\n', f.read())
+        self.assertActivitiesMatch()

=== modified file 'bzrlib/transport/http/__init__.py'
--- a/bzrlib/transport/http/__init__.py	2009-01-23 21:22:39 +0000
+++ b/bzrlib/transport/http/__init__.py	2009-01-29 14:27:28 +0000
@@ -123,17 +123,13 @@
 
         :param relpath: The relative path to the file
         """
+        code, response_file = self._get(relpath, None)
         # FIXME: some callers want an iterable... One step forward, three steps
         # backwards :-/ And not only an iterable, but an iterable that can be
         # seeked backwards, so we will never be able to do that.  One such
         # known client is bzrlib.bundle.serializer.v4.get_bundle_reader. At the
         # time of this writing it's even the only known client -- vila20071203
-        return StringIO(self.get_bytes(relpath))
-
-    def get_bytes(self, relpath):
-        """See Transport.get_bytes()."""
-        code, response_file = self._get(relpath, None)
-        return response_file.read()
+        return StringIO(response_file.read())
 
     def _get(self, relpath, ranges, tail_amount=0):
         """Get a file, or part of a file.

=== modified file 'bzrlib/transport/http/_pycurl.py'
--- a/bzrlib/transport/http/_pycurl.py	2009-01-23 21:22:39 +0000
+++ b/bzrlib/transport/http/_pycurl.py	2009-01-30 00:49:41 +0000
@@ -215,8 +215,8 @@
 
     # The parent class use 0 to minimize the requests, but since we can't
     # exploit the results as soon as they are received (pycurl limitation) we'd
-    # better issue more requests and provide a more responsive UI do the cost
-    # of more latency costs.
+    # better issue more requests and provide a more responsive UI incurring
+    # more latency costs.
     # If you modify this, think about modifying the comment in http/__init__.py
     # too.
     _get_max_size = 4 * 1024 * 1024
@@ -245,8 +245,7 @@
                                           'Server return code %d'
                                           % curl.getinfo(pycurl.HTTP_CODE))
         msg = self._parse_headers(header)
-        return code, response.handle_response(abspath, code, msg, data,
-            report_activity=self._report_activity)
+        return code, response.handle_response(abspath, code, msg, data)
 
     def _parse_headers(self, status_and_headers):
         """Transform the headers provided by curl into an HTTPMessage"""
@@ -286,8 +285,7 @@
         data.seek(0)
         code = curl.getinfo(pycurl.HTTP_CODE)
         msg = self._parse_headers(header)
-        return code, response.handle_response(abspath, code, msg, data,
-            report_activity=self._report_activity)
+        return code, response.handle_response(abspath, code, msg, data)
 
 
     def _raise_curl_http_error(self, curl, info=None):
@@ -307,15 +305,28 @@
             raise errors.InvalidHttpResponse(
                 url, 'Unable to handle http code %d%s' % (code,msg))
 
+    def _debug_cb(self, kind, text):
+        if kind in (pycurl.INFOTYPE_HEADER_IN, pycurl.INFOTYPE_DATA_IN,
+                    pycurl.INFOTYPE_SSL_DATA_IN):
+            self._report_activity(len(text), 'read')
+            if (kind == pycurl.INFOTYPE_HEADER_IN
+                and 'http' in debug.debug_flags):
+                mutter('< %s' % text)
+        elif kind in (pycurl.INFOTYPE_HEADER_OUT, pycurl.INFOTYPE_DATA_OUT,
+                      pycurl.INFOTYPE_SSL_DATA_OUT):
+            self._report_activity(len(text), 'write')
+            if (kind == pycurl.INFOTYPE_HEADER_OUT
+                and 'http' in debug.debug_flags):
+                mutter('> %s' % text)
+        elif kind == pycurl.INFOTYPE_TEXT and 'http' in debug.debug_flags:
+            mutter('* %s' % text)
+
     def _set_curl_options(self, curl):
         """Set options for all requests"""
-        if 'http' in debug.debug_flags:
-            curl.setopt(pycurl.VERBOSE, 1)
-            # pycurl doesn't implement the CURLOPT_STDERR option, so we can't
-            # do : curl.setopt(pycurl.STDERR, trace._trace_file)
-
         ua_str = 'bzr/%s (pycurl: %s)' % (bzrlib.__version__, pycurl.version)
         curl.setopt(pycurl.USERAGENT, ua_str)
+        curl.setopt(pycurl.VERBOSE, 1)
+        curl.setopt(pycurl.DEBUGFUNCTION, self._debug_cb)
         if self.cabundle:
             curl.setopt(pycurl.CAINFO, self.cabundle)
         # Set accepted auth methods

=== modified file 'bzrlib/transport/http/_urllib.py'
--- a/bzrlib/transport/http/_urllib.py	2009-01-23 21:22:39 +0000
+++ b/bzrlib/transport/http/_urllib.py	2009-01-29 14:27:28 +0000
@@ -47,7 +47,8 @@
         if _from_transport is not None:
             self._opener = _from_transport._opener
         else:
-            self._opener = self._opener_class()
+            self._opener = self._opener_class(
+                report_activity=self._report_activity)
 
     def _perform(self, request):
         """Send the request to the server and handles common errors.
@@ -100,7 +101,6 @@
 
     def _get(self, relpath, offsets, tail_amount=0):
         """See HttpTransport._get"""
-
         abspath = self._remote_path(relpath)
         headers = {}
         accepted_errors = [200, 404]
@@ -126,8 +126,7 @@
             raise errors.InvalidHttpRange(abspath, range_header,
                                           'Server return code %d' % code)
 
-        data = handle_response(abspath, code, response.info(), response,
-            report_activity=self._report_activity)
+        data = handle_response(abspath, code, response.info(), response)
         return code, data
 
     def _post(self, body_bytes):
@@ -137,8 +136,7 @@
         response = self._perform(Request('POST', abspath, body_bytes,
                                          accepted_errors=[200, 403]))
         code = response.code
-        data = handle_response(abspath, code, response.info(), response,
-            report_activity=self._report_activity)
+        data = handle_response(abspath, code, response.info(), response)
         return code, data
 
     def _head(self, relpath):

=== modified file 'bzrlib/transport/http/_urllib2_wrappers.py'
--- a/bzrlib/transport/http/_urllib2_wrappers.py	2009-01-08 16:57:10 +0000
+++ b/bzrlib/transport/http/_urllib2_wrappers.py	2009-01-29 14:27:28 +0000
@@ -67,13 +67,54 @@
     )
 
 
-class _BufferedMakefileSocket(object):
-
-    def __init__(self, sock):
+class _ReportingFileSocket(object):
+
+    def __init__(self, filesock, report_activity=None):
+        self.filesock = filesock
+        self._report_activity = report_activity
+
+
+    def read(self, size=1):
+        s = self.filesock.read(size)
+        self._report_activity(len(s), 'read')
+        return s
+
+    def readline(self, size=-1):
+        s = self.filesock.readline(size)
+        self._report_activity(len(s), 'read')
+        return s
+
+    def __getattr__(self, name):
+        return getattr(self.filesock, name)
+
+
+class _ReportingSocket(object):
+
+    def __init__(self, sock, report_activity=None):
         self.sock = sock
+        self._report_activity = report_activity
+
+    def send(self, s, *args):
+        self.sock.send(s, *args)
+        self._report_activity(len(s), 'write')
+
+    def sendall(self, s, *args):
+        self.sock.send(s, *args)
+        self._report_activity(len(s), 'write')
+
+    def recv(self, *args):
+        s = self.sock.recv(*args)
+        self._report_activity(len(s), 'read')
+        return s
 
     def makefile(self, mode='r', bufsize=-1):
-        return self.sock.makefile(mode, 65536)
+        # httplib creates a fileobject that doesn't do buffering, which
+        # makes fp.readline() very expensive because it only reads one byte
+        # at a time.  So we wrap the socket in an object that forces
+        # sock.makefile to make a buffered file.
+        fsock = self.sock.makefile(mode, 65536)
+        # And wrap that into a reporting kind of fileobject
+        return _ReportingFileSocket(fsock, self._report_activity)
 
     def __getattr__(self, name):
         return getattr(self.sock, name)
@@ -96,14 +137,6 @@
     # 8k chunks should be fine.
     _discarded_buf_size = 8192
 
-    def __init__(self, sock, *args, **kwargs):
-        # httplib creates a fileobject that doesn't do buffering, which
-        # makes fp.readline() very expensive because it only reads one byte
-        # at a time.  So we wrap the socket in an object that forces
-        # sock.makefile to make a buffered file.
-        sock = _BufferedMakefileSocket(sock)
-        httplib.HTTPResponse.__init__(self, sock, *args, **kwargs)
-
     def begin(self):
         """Begin to read the response from the server.
 
@@ -178,8 +211,10 @@
     # we want to warn. But not below a given thresold.
     _range_warning_thresold = 1024 * 1024
 
-    def __init__(self):
+    def __init__(self,
+                 report_activity=None):
         self._response = None
+        self._report_activity = report_activity
         self._ranges_received_whole_file = None
 
     def _mutter_connect(self):
@@ -216,12 +251,17 @@
         # Restore our preciousss
         self.sock = sock
 
+    def _wrap_socket_for_reporting(self, sock):
+        """Wrap the socket before anybody use it."""
+        self.sock = _ReportingSocket(sock, self._report_activity)
+
 
 class HTTPConnection(AbstractHTTPConnection, httplib.HTTPConnection):
 
     # XXX: Needs refactoring at the caller level.
-    def __init__(self, host, port=None, proxied_host=None):
-        AbstractHTTPConnection.__init__(self)
+    def __init__(self, host, port=None, proxied_host=None,
+                 report_activity=None):
+        AbstractHTTPConnection.__init__(self, report_activity=report_activity)
         # Use strict=True since we don't support HTTP/0.9
         httplib.HTTPConnection.__init__(self, host, port, strict=True)
         self.proxied_host = proxied_host
@@ -230,6 +270,7 @@
         if 'http' in debug.debug_flags:
             self._mutter_connect()
         httplib.HTTPConnection.connect(self)
+        self._wrap_socket_for_reporting(self.sock)
 
 
 # Build the appropriate socket wrapper for ssl
@@ -248,8 +289,9 @@
 class HTTPSConnection(AbstractHTTPConnection, httplib.HTTPSConnection):
 
     def __init__(self, host, port=None, key_file=None, cert_file=None,
-                 proxied_host=None):
-        AbstractHTTPConnection.__init__(self)
+                 proxied_host=None,
+                 report_activity=None):
+        AbstractHTTPConnection.__init__(self, report_activity=report_activity)
         # Use strict=True since we don't support HTTP/0.9
         httplib.HTTPSConnection.__init__(self, host, port,
                                          key_file, cert_file, strict=True)
@@ -259,11 +301,14 @@
         if 'http' in debug.debug_flags:
             self._mutter_connect()
         httplib.HTTPConnection.connect(self)
+        self._wrap_socket_for_reporting(self.sock)
         if self.proxied_host is None:
             self.connect_to_origin()
 
     def connect_to_origin(self):
-        self.sock = _ssl_wrap_socket(self.sock, self.key_file, self.cert_file)
+        ssl_sock = _ssl_wrap_socket(self.sock, self.key_file, self.cert_file)
+        # Wrap the ssl socket before anybody use it
+        self._wrap_socket_for_reporting(ssl_sock)
 
 
 class Request(urllib2.Request):
@@ -355,6 +400,9 @@
 
     handler_order = 1000 # after all pre-processings
 
+    def __init__(self, report_activity=None):
+        self._report_activity = report_activity
+
     def create_connection(self, request, http_connection_class):
         host = request.get_host()
         if not host:
@@ -366,7 +414,8 @@
         # request is made)
         try:
             connection = http_connection_class(
-                host, proxied_host=request.proxied_host)
+                host, proxied_host=request.proxied_host,
+                report_activity=self._report_activity)
         except httplib.InvalidURL, exception:
             # There is only one occurrence of InvalidURL in httplib
             raise errors.InvalidURL(request.get_full_url(),
@@ -1370,9 +1419,11 @@
     def __init__(self,
                  connection=ConnectionHandler,
                  redirect=HTTPRedirectHandler,
-                 error=HTTPErrorProcessor,):
-        self._opener = urllib2.build_opener( \
-            connection, redirect, error,
+                 error=HTTPErrorProcessor,
+                 report_activity=None):
+        self._opener = urllib2.build_opener(
+            connection(report_activity=report_activity),
+            redirect, error,
             ProxyHandler(),
             HTTPBasicAuthHandler(),
             HTTPDigestAuthHandler(),

=== modified file 'bzrlib/transport/http/response.py'
--- a/bzrlib/transport/http/response.py	2009-01-23 21:22:39 +0000
+++ b/bzrlib/transport/http/response.py	2009-01-30 00:49:41 +0000
@@ -67,18 +67,15 @@
     # maximum size of read requests -- used to avoid MemoryError issues in recv
     _max_read_size = 512 * 1024
 
-    def __init__(self, path, infile, report_activity=None):
+    def __init__(self, path, infile):
         """Constructor.
 
         :param path: File url, for error reports.
         :param infile: File-like socket set at body start.
-        :param report_activity: A Transport._report_activity function to call
-            as bytes are read.
         """
         self._path = path
         self._file = infile
         self._boundary = None
-        self._report_activity = report_activity
         # When using multi parts response, this will be set with the headers
         # associated with the range currently read.
         self._headers = None
@@ -231,8 +228,7 @@
             limited = self._start + self._size - self._pos
             if size >= 0:
                 limited = min(limited, size)
-        osutils.pumpfile(self._file, buffer, limited, self._max_read_size,
-            report_activity=self._report_activity, direction='read')
+        osutils.pumpfile(self._file, buffer, limited, self._max_read_size)
         data = buffer.getvalue()
 
         # Update _pos respecting the data effectively read
@@ -281,7 +277,7 @@
         return self._pos
 
 
-def handle_response(url, code, msg, data, report_activity=None):
+def handle_response(url, code, msg, data):
     """Interpret the code & headers and wrap the provided data in a RangeFile.
 
     This is a factory method which returns an appropriate RangeFile based on
@@ -295,7 +291,7 @@
     :return: A file-like object that can seek()+read() the 
              ranges indicated by the headers.
     """
-    rfile = RangeFile(url, data, report_activity=report_activity)
+    rfile = RangeFile(url, data)
     if code == 200:
         # A whole file
         size = msg.getheader('content-length', None)



More information about the bazaar-commits mailing list