Rev 452: Bring in the load-test updates as well. in http://bazaar.launchpad.net/~jameinel/loggerhead/trunk-into-experimental

John Arbash Meinel john at arbash-meinel.com
Thu Feb 10 23:09:07 UTC 2011


At http://bazaar.launchpad.net/~jameinel/loggerhead/trunk-into-experimental

------------------------------------------------------------
revno: 452 [merge]
revision-id: john at arbash-meinel.com-20110210230900-7xuqstphs1efolvp
parent: john at arbash-meinel.com-20110210015144-0kqtadsgnenl0w52
parent: john at arbash-meinel.com-20110210230659-ejcgl2r4z768beho
committer: John Arbash Meinel <john at arbash-meinel.com>
branch nick: trunk-into-experimental
timestamp: Thu 2011-02-10 17:09:00 -0600
message:
  Bring in the load-test updates as well.
added:
  HACKING                        hacking-20110210230514-0nni6rcgjisxbyj6-1
  load_test_scripts/             load_test_scripts-20110210005009-joifj89nhdn34aix-1
  load_test_scripts/multiple_instances.script multiple_instances.s-20110210005705-laogm0i7xp7t2o0a-1
  load_test_scripts/simple.script load_test.script-20110210004226-8f1448651hqu0zn0-1
  loggerhead/load_test.py        load_test.py-20110209211518-hww0u9oztvmw5jfh-1
  loggerhead/tests/test_load_test.py test_load_test.py-20110209211518-hww0u9oztvmw5jfh-2
modified:
  NEWS                           news-20070121024650-6cwmhprgtcegpxvm-1
  __init__.py                    __init__.py-20090123174919-ekabddqbmvwuci2i-1
  loggerhead/tests/__init__.py   __init__.py-20061211064342-102iqirsciyvgtcf-29
-------------- next part --------------
=== added file 'HACKING'
--- a/HACKING	1970-01-01 00:00:00 +0000
+++ b/HACKING	2011-02-10 23:05:31 +0000
@@ -0,0 +1,59 @@
+Loggerhead
+==========
+
+Overview
+--------
+
+This document attempts to give some hints for people that are wanting to work
+on Loggerhead.
+
+
+Testing
+-------
+
+You can run the loggerhead test suite as a bzr plugin. To run just the
+loggerhead tests::
+
+  bzr selftest -s bp.loggerhead
+
+
+Load Testing
+------------
+
+As a web service, Loggerhead will often be hit by multiple requests. We want
+to make sure that loggerhead can scale with many requests, without performing
+poorly or crashing under the load.
+
+There is a command ``bzr load-test-loggerhead`` that can be run to stress
+loggerhead. A script is given, describing what requests to make, against what
+URLs, and for what level of parallel activity.
+
+
+Load Testing Multiple Instances
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+One way that Launchpad provides both high availability and performance scaling
+is by running multiple instances of loggerhead, serving the same content. A
+proxy is then used to load balance the requests. This also allows us to shut
+down one instance for upgrading, without interupting service (requests are
+just routed to the other instance).
+
+However, multiple processes poses an even greater risk that caches will
+conflict. As such, it is useful to test that changes don't introduce coherency
+issues at load. ``bzr load-test-loggerhead`` can be configured with a script
+that will make requests against multiple loggerhead instances concurrently.
+
+To run multiple instances, it is often sufficient to just spawn multiple
+servers on different ports. For example::
+
+  $ bzr serve --http --port=8080 &
+  $ bzr serve --http --port=8081 &
+
+There is a simple example script already in the source tree::
+
+  $ bzr load-test-loggerhead load_test_scripts/multiple_instances.script
+
+
+
+.. vim: ft=rst tw=78
+

=== modified file 'NEWS'
--- a/NEWS	2011-02-10 01:32:13 +0000
+++ b/NEWS	2011-02-10 23:09:00 +0000
@@ -4,6 +4,11 @@
 dev [future]
 ------------
 
+    - Add ``bzr load-test-loggerhead`` as a way to make sure loggerhead can
+      handle concurrent requests, etc. Scripts can be written that spawn
+      multiple threads, and issue concurrent requests.
+      (John Arbash Meinel)
+
     - If we get a HEAD request, there is no reason to expand the template, we
       shouldn't be returning body content anyway.
       (John Arbash Meinel, #716201, #716217)

=== modified file '__init__.py'
--- a/__init__.py	2011-02-10 01:32:13 +0000
+++ b/__init__.py	2011-02-10 23:09:00 +0000
@@ -1,4 +1,4 @@
-# Copyright 2009, 2010 Canonical Ltd
+# Copyright 2009, 2010, 2011 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -38,6 +38,7 @@
 if __name__ == 'bzrlib.plugins.loggerhead':
     import bzrlib
     from bzrlib.api import require_any_api
+    from bzrlib import commands
 
     require_any_api(bzrlib, bzr_compatible_versions)
 
@@ -106,6 +107,26 @@
 
     transport_server_registry.register('http', serve_http, help=HELP)
 
+    class cmd_load_test_loggerhead(commands.Command):
+        """Run a load test against a live loggerhead instance.
+
+        Pass in the name of a script file to run. See loggerhead/load_test.py
+        for a description of the file format.
+        """
+
+        takes_args = ["filename"]
+
+        def run(self, filename):
+            from bzrlib.plugins.loggerhead.loggerhead import load_test
+            script = load_test.run_script(filename)
+            for thread_id in sorted(script._threads):
+                worker = script._threads[thread_id][0]
+                for url, success, time in worker.stats:
+                    self.outf.write(' %5.3fs %s %s\n'
+                                    % (time, str(success)[0], url))
+
+    commands.register_command(cmd_load_test_loggerhead)
+
     def load_tests(standard_tests, module, loader):
         _ensure_loggerhead_path()
         try:

=== added directory 'load_test_scripts'
=== added file 'load_test_scripts/multiple_instances.script'
--- a/load_test_scripts/multiple_instances.script	1970-01-01 00:00:00 +0000
+++ b/load_test_scripts/multiple_instances.script	2011-02-10 01:00:32 +0000
@@ -0,0 +1,19 @@
+{
+    "comment": "Connect to multiple loggerhead instances and make requests on each. One should be on :8080, one should be on :8081. Multiple threads will place requests on each.",
+    "parameters": {"base_url": "http://localhost"},
+    "requests": [
+        {"thread": "1", "relpath": ":8080/changes"},
+        {"thread": "2", "relpath": ":8080/files"},
+        {"thread": "3", "relpath": ":8081/files"},
+        {"thread": "4", "relpath": ":8081/changes"},
+        {"thread": "1", "relpath": ":8080/changes"},
+        {"thread": "2", "relpath": ":8080/files"},
+        {"thread": "3", "relpath": ":8081/files"},
+        {"thread": "4", "relpath": ":8081/changes"},
+        {"thread": "1", "relpath": ":8080/changes"},
+        {"thread": "2", "relpath": ":8080/files"},
+        {"thread": "3", "relpath": ":8081/files"},
+        {"thread": "4", "relpath": ":8081/changes"}
+    ]
+}
+

=== added file 'load_test_scripts/simple.script'
--- a/load_test_scripts/simple.script	1970-01-01 00:00:00 +0000
+++ b/load_test_scripts/simple.script	2011-02-10 01:00:32 +0000
@@ -0,0 +1,9 @@
+{
+    "comment": "A fairly trivial load test script. It just loads the main two pages from a loggerhead install running directly on a branch.",
+    "parameters": {"base_url": "http://localhost:8080"},
+    "requests": [
+        {"relpath": "/changes"},
+        {"relpath": "/files"}
+    ]
+}
+

=== added file 'loggerhead/load_test.py'
--- a/loggerhead/load_test.py	1970-01-01 00:00:00 +0000
+++ b/loggerhead/load_test.py	2011-02-10 01:00:32 +0000
@@ -0,0 +1,233 @@
+# Copyright 2011 Canonical Ltd
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+"""Code to do some load testing against a loggerhead instance.
+
+This is basically meant to take a list of actions to take, and run it against a
+real host, and see how the results respond.::
+
+    {"parameters": {
+         "base_url": "http://localhost:8080",
+     },
+     "requests": [
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"}
+     ],
+    }
+
+All threads have a Queue length of 1. When a third request for a given thread
+is seen, no more requests are queued until that thread finishes its current
+job. So this results in all requests being issued sequentially::
+
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"}
+
+While this would cause all requests to be sent in parallel:
+
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "2", "relpath": "/changes"},
+        {"thread": "3", "relpath": "/changes"},
+        {"thread": "4", "relpath": "/changes"}
+
+This should keep 2 threads pipelined with activity, as long as they finish in
+approximately the same speed. We'll start the first thread running, and the
+second thread, and queue up both with a second request once the first finishes.
+When we get to the third request for thread "1", we block on queuing up more
+work until the first thread 1 request has finished.
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "2", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "2", "relpath": "/changes"},
+        {"thread": "1", "relpath": "/changes"},
+        {"thread": "2", "relpath": "/changes"}
+
+There is not currently a way to say "run all these requests keeping exactly 2
+threads active". Though if you know the load pattern, you could approximate
+this.
+"""
+
+import threading
+import time
+import Queue
+
+try:
+    import simplejson
+except ImportError:
+    import json as simplejson
+
+from bzrlib import (
+    errors,
+    transport,
+    urlutils,
+    )
+
+# This code will be doing multi-threaded requests against bzrlib.transport
+# code. We want to make sure to load everything ahead of time, so we don't get
+# lazy-import failures
+_ = transport.get_transport('http://example.com')
+
+
+class RequestDescription(object):
+    """Describes info about a request."""
+
+    def __init__(self, descrip_dict):
+        self.thread = descrip_dict.get('thread', '1')
+        self.relpath = descrip_dict['relpath']
+
+
+class RequestWorker(object):
+    """Process requests in a worker thread."""
+
+    _timer = time.time
+
+    def __init__(self, identifier, blocking_time=1.0, _queue_size=1):
+        self.identifier = identifier
+        self.queue = Queue.Queue(_queue_size)
+        self.start_time = self.end_time = None
+        self.stats = []
+        self.blocking_time = blocking_time
+
+    def step_next(self):
+        url = self.queue.get(True, self.blocking_time)
+        if url == '<noop>':
+            # This is usually an indicator that we want to stop, so just skip
+            # this one.
+            self.queue.task_done()
+            return
+        self.start_time = self._timer()
+        success = self.process(url)
+        self.end_time = self._timer()
+        self.update_stats(url, success)
+        self.queue.task_done()
+
+    def run(self, stop_event):
+        while not stop_event.isSet():
+            try:
+                self.step_next()
+            except Queue.Empty:
+                pass
+
+    def process(self, url):
+        base, path = urlutils.split(url)
+        t = transport.get_transport(base)
+        try:
+            # TODO: We should probably look into using some part of
+            #       blocking_timeout to decide when to stop trying to read
+            #       content
+            content = t.get_bytes(path)
+        except (errors.TransportError, errors.NoSuchFile):
+            return False
+        return True
+
+    def update_stats(self, url, success):
+        self.stats.append((url, success, self.end_time - self.start_time))
+
+
+class ActionScript(object):
+    """This tracks the actions that we want to perform."""
+
+    _worker_class = RequestWorker
+    _default_base_url = 'http://localhost:8080'
+    _default_blocking_timeout = 60.0
+
+    def __init__(self):
+        self.base_url = self._default_base_url
+        self.blocking_timeout = self._default_blocking_timeout
+        self._requests = []
+        self._threads = {}
+        self.stop_event = threading.Event()
+
+    @classmethod
+    def parse(cls, content):
+        script = cls()
+        json_dict = simplejson.loads(content)
+        if 'parameters' not in json_dict:
+            raise ValueError('Missing "parameters" section')
+        if 'requests' not in json_dict:
+            raise ValueError('Missing "requests" section')
+        param_dict = json_dict['parameters']
+        request_list = json_dict['requests']
+        base_url = param_dict.get('base_url', None)
+        if base_url is not None:
+            script.base_url = base_url
+        blocking_timeout = param_dict.get('blocking_timeout', None)
+        if blocking_timeout is not None:
+            script.blocking_timeout = blocking_timeout
+        for request_dict in request_list:
+            script.add_request(request_dict)
+        return script
+
+    def add_request(self, request_dict):
+        request = RequestDescription(request_dict)
+        self._requests.append(request)
+
+    def _get_worker(self, thread_id):
+        if thread_id in self._threads:
+            return self._threads[thread_id][0]
+        handler = self._worker_class(thread_id,
+                                     blocking_time=self.blocking_timeout/10.0)
+
+        t = threading.Thread(target=handler.run, args=(self.stop_event,),
+                             name='Thread-%s' % (thread_id,))
+        self._threads[thread_id] = (handler, t)
+        t.start()
+        return handler
+
+    def finish_queues(self):
+        """Wait for all queues of all children to finish."""
+        for h, t in self._threads.itervalues():
+            h.queue.join()
+
+    def stop_and_join(self):
+        """Stop all running workers, and return.
+
+        This will stop even if workers still have work items.
+        """
+        self.stop_event.set()
+        for h, t in self._threads.itervalues():
+            # Signal the queue that it should stop blocking, we don't have to
+            # wait for the queue to empty, because we may see stop_event before
+            # we see the <noop>
+            h.queue.put('<noop>')
+            # And join the controlling thread
+            for i in range(10):
+                t.join(self.blocking_timeout / 10.0)
+                if not t.isAlive():
+                    break
+
+    def _full_url(self, relpath):
+        return self.base_url + relpath
+
+    def run(self):
+        self.stop_event.clear()
+        for request in self._requests:
+            full_url = self._full_url(request.relpath)
+            worker = self._get_worker(request.thread)
+            worker.queue.put(full_url, True, self.blocking_timeout)
+        self.finish_queues()
+        self.stop_and_join()
+
+
+def run_script(filename):
+    with open(filename, 'rb') as f:
+        content = f.read()
+    script = ActionScript.parse(content)
+    script.run()
+    return script

=== modified file 'loggerhead/tests/__init__.py'
--- a/loggerhead/tests/__init__.py	2011-01-28 22:44:19 +0000
+++ b/loggerhead/tests/__init__.py	2011-02-10 23:09:00 +0000
@@ -21,6 +21,7 @@
             'history_db_tests',
             'test_controllers',
             'test_corners',
+            'test_load_test',
             'test_simple',
             'test_templating',
         ]]))

=== added file 'loggerhead/tests/test_load_test.py'
--- a/loggerhead/tests/test_load_test.py	1970-01-01 00:00:00 +0000
+++ b/loggerhead/tests/test_load_test.py	2011-02-10 00:49:06 +0000
@@ -0,0 +1,337 @@
+# Copyright 2011 Canonical Ltd
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#
+
+"""Tests for the load testing code."""
+
+import socket
+import time
+import threading
+import Queue
+
+from bzrlib import tests
+from bzrlib.tests import http_server
+
+from bzrlib.plugins.loggerhead.loggerhead import load_test
+
+
+empty_script = """{
+    "parameters": {},
+    "requests": []
+}"""
+
+class TestRequestDescription(tests.TestCase):
+
+    def test_init_from_dict(self):
+        rd = load_test.RequestDescription({'thread': '10', 'relpath': '/foo'})
+        self.assertEqual('10', rd.thread)
+        self.assertEqual('/foo', rd.relpath)
+
+    def test_default_thread_is_1(self):
+        rd = load_test.RequestDescription({'relpath': '/bar'})
+        self.assertEqual('1', rd.thread)
+        self.assertEqual('/bar', rd.relpath)
+
+
+_cur_time = time.time()
+def one_sec_timer():
+    """Every time this timer is called, it increments by 1 second."""
+    global _cur_time
+    _cur_time += 1.0
+    return _cur_time
+
+
+class NoopRequestWorker(load_test.RequestWorker):
+
+    # Every call to _timer will increment by one
+    _timer = staticmethod(one_sec_timer)
+
+    # Ensure that process never does anything
+    def process(self, url):
+        return True
+
+
+class TestRequestWorkerInfrastructure(tests.TestCase):
+    """Tests various infrastructure bits, without doing actual requests."""
+
+    def test_step_next_tracks_time(self):
+        rt = NoopRequestWorker('id')
+        rt.queue.put('item')
+        rt.step_next()
+        self.assertTrue(rt.queue.empty())
+        self.assertEqual([('item', True, 1.0)], rt.stats)
+
+    def test_step_multiple_items(self):
+        rt = NoopRequestWorker('id')
+        rt.queue.put('item')
+        rt.step_next()
+        rt.queue.put('next-item')
+        rt.step_next()
+        self.assertTrue(rt.queue.empty())
+        self.assertEqual([('item', True, 1.0), ('next-item', True, 1.0)],
+                         rt.stats)
+
+    def test_step_next_does_nothing_for_noop(self):
+        rt = NoopRequestWorker('id')
+        rt.queue.put('item')
+        rt.step_next()
+        rt.queue.put('<noop>')
+        rt.step_next()
+        self.assertEqual([('item', True, 1.0)], rt.stats)
+
+    def test_step_next_will_timeout(self):
+        # We don't want step_next to block forever
+        rt = NoopRequestWorker('id', blocking_time=0.001)
+        self.assertRaises(Queue.Empty, rt.step_next)
+
+    def test_run_stops_for_stop_event(self):
+        rt = NoopRequestWorker('id', blocking_time=0.001, _queue_size=2)
+        rt.queue.put('item1')
+        rt.queue.put('item2')
+        event = threading.Event()
+        t = threading.Thread(target=rt.run, args=(event,))
+        t.start()
+        # Wait for the queue to be processed
+        rt.queue.join()
+        # Now we can queue up another one, and wait for it
+        rt.queue.put('item3')
+        rt.queue.join()
+        # Now set the stopping event
+        event.set()
+        # Add another item to the queue, which might get processed, but the
+        # next item won't
+        rt.queue.put('item4')
+        rt.queue.put('item5')
+        t.join()
+        self.assertEqual([('item1', True, 1.0), ('item2', True, 1.0),
+                          ('item3', True, 1.0)],
+                         rt.stats[:3])
+        # The last event might be item4 or might be item3, the important thing
+        # is that even though there are still queued events, we won't
+        # process anything past the first
+        self.assertNotEqual('item5', rt.stats[-1][0])
+
+
+class TestRequestWorker(tests.TestCaseWithTransport):
+
+    def setUp(self):
+        super(TestRequestWorker, self).setUp()
+        self.transport_readonly_server = http_server.HttpServer
+
+    def test_request_items(self):
+        rt = load_test.RequestWorker('id', blocking_time=0.01, _queue_size=2)
+        self.build_tree(['file1', 'file2'])
+        readonly_url1 = self.get_readonly_url('file1')
+        self.assertStartsWith(readonly_url1, 'http://')
+        readonly_url2 = self.get_readonly_url('file2')
+        rt.queue.put(readonly_url1)
+        rt.queue.put(readonly_url2)
+        rt.step_next()
+        rt.step_next()
+        self.assertEqual(readonly_url1, rt.stats[0][0])
+        self.assertEqual(readonly_url2, rt.stats[1][0])
+
+    def test_request_nonexistant_items(self):
+        rt = load_test.RequestWorker('id', blocking_time=0.01, _queue_size=2)
+        readonly_url1 = self.get_readonly_url('no-file1')
+        rt.queue.put(readonly_url1)
+        rt.step_next()
+        self.assertEqual(readonly_url1, rt.stats[0][0])
+        self.assertEqual(False, rt.stats[0][1])
+
+    def test_no_server(self):
+        s = socket.socket()
+        # Bind to a port, but don't listen on it
+        s.bind(('localhost', 0))
+        url = 'http://%s:%s/path' % s.getsockname()
+        rt = load_test.RequestWorker('id', blocking_time=0.01, _queue_size=2)
+        rt.queue.put(url)
+        rt.step_next()
+        self.assertEqual((url, False), rt.stats[0][:2])
+
+
+class NoActionScript(load_test.ActionScript):
+
+    _thread_class = NoopRequestWorker
+    _default_blocking_timeout = 0.01
+
+
+class TestActionScriptInfrastructure(tests.TestCase):
+
+    def test_parse_requires_parameters_and_requests(self):
+        self.assertRaises(ValueError,
+            load_test.ActionScript.parse, '')
+        self.assertRaises(ValueError,
+            load_test.ActionScript.parse, '{}')
+        self.assertRaises(ValueError,
+            load_test.ActionScript.parse, '{"parameters": {}}')
+        self.assertRaises(ValueError,
+            load_test.ActionScript.parse, '{"requests": []}')
+        load_test.ActionScript.parse(
+            '{"parameters": {}, "requests": [], "comment": "section"}')
+        script = load_test.ActionScript.parse(
+            empty_script)
+        self.assertIsNot(None, script)
+
+    def test_parse_default_base_url(self):
+        script = load_test.ActionScript.parse(empty_script)
+        self.assertEqual('http://localhost:8080', script.base_url)
+
+    def test_parse_find_base_url(self):
+        script = load_test.ActionScript.parse(
+            '{"parameters": {"base_url": "http://example.com"},'
+            ' "requests": []}')
+        self.assertEqual('http://example.com', script.base_url)
+
+    def test_parse_default_blocking_timeout(self):
+        script = load_test.ActionScript.parse(empty_script)
+        self.assertEqual(60.0, script.blocking_timeout)
+
+    def test_parse_find_blocking_timeout(self):
+        script = load_test.ActionScript.parse(
+            '{"parameters": {"blocking_timeout": 10.0},'
+            ' "requests": []}'
+        )
+        self.assertEqual(10.0, script.blocking_timeout)
+
+    def test_parse_finds_requests(self):
+        script = load_test.ActionScript.parse(
+            '{"parameters": {}, "requests": ['
+            ' {"relpath": "/foo"},'
+            ' {"relpath": "/bar"}'
+            ' ]}')
+        self.assertEqual(2, len(script._requests))
+        self.assertEqual("/foo", script._requests[0].relpath)
+        self.assertEqual("/bar", script._requests[1].relpath)
+
+    def test__get_worker(self):
+        script = NoActionScript()
+        # If an id is found, then we should create it
+        self.assertEqual({}, script._threads)
+        worker = script._get_worker('id')
+        self.assertTrue('id' in script._threads)
+        # We should have set the blocking timeout
+        self.assertEqual(script.blocking_timeout / 10.0,
+                         worker.blocking_time)
+
+        # Another request will return the identical object
+        self.assertIs(worker, script._get_worker('id'))
+
+        # And the stop event will stop the thread
+        script.stop_and_join()
+
+    def test__full_url(self):
+        script = NoActionScript()
+        self.assertEqual('http://localhost:8080/path',
+                         script._full_url('/path'))
+        self.assertEqual('http://localhost:8080/path/to/foo',
+                         script._full_url('/path/to/foo'))
+        script.base_url = 'http://example.com'
+        self.assertEqual('http://example.com/path/to/foo',
+                         script._full_url('/path/to/foo'))
+        script.base_url = 'http://example.com/base'
+        self.assertEqual('http://example.com/base/path/to/foo',
+                         script._full_url('/path/to/foo'))
+        script.base_url = 'http://example.com'
+        self.assertEqual('http://example.com:8080/path',
+                         script._full_url(':8080/path'))
+
+    def test_single_threaded(self):
+        script = NoActionScript.parse("""{
+            "parameters": {"base_url": ""},
+            "requests": [
+                {"thread": "1", "relpath": "first"},
+                {"thread": "1", "relpath": "second"},
+                {"thread": "1", "relpath": "third"},
+                {"thread": "1", "relpath": "fourth"}
+            ]}""")
+        script.run()
+        worker = script._get_worker("1")
+        self.assertEqual(["first", "second", "third", "fourth"],
+                         [s[0] for s in worker.stats])
+
+    def test_two_threads(self):
+        script = NoActionScript.parse("""{
+            "parameters": {"base_url": ""},
+            "requests": [
+                {"thread": "1", "relpath": "first"},
+                {"thread": "2", "relpath": "second"},
+                {"thread": "1", "relpath": "third"},
+                {"thread": "2", "relpath": "fourth"}
+            ]}""")
+        script.run()
+        worker = script._get_worker("1")
+        self.assertEqual(["first", "third"],
+                         [s[0] for s in worker.stats])
+        worker = script._get_worker("2")
+        self.assertEqual(["second", "fourth"],
+                         [s[0] for s in worker.stats])
+
+
+class TestActionScriptIntegration(tests.TestCaseWithTransport):
+
+    def setUp(self):
+        super(TestActionScriptIntegration, self).setUp()
+        self.transport_readonly_server = http_server.HttpServer
+
+    def test_full_integration(self):
+        self.build_tree(['first', 'second', 'third', 'fourth'])
+        url = self.get_readonly_url()
+        script = load_test.ActionScript.parse("""{
+            "parameters": {"base_url": "%s", "blocking_timeout": 0.1},
+            "requests": [
+                {"thread": "1", "relpath": "first"},
+                {"thread": "2", "relpath": "second"},
+                {"thread": "1", "relpath": "no-this"},
+                {"thread": "2", "relpath": "or-this"},
+                {"thread": "1", "relpath": "third"},
+                {"thread": "2", "relpath": "fourth"}
+            ]}""" % (url,))
+        script.run()
+        worker = script._get_worker("1")
+        self.assertEqual([("first", True), ('no-this', False),
+                          ("third", True)],
+                         [(s[0].rsplit('/', 1)[1], s[1])
+                          for s in worker.stats])
+        worker = script._get_worker("2")
+        self.assertEqual([("second", True), ('or-this', False),
+                          ("fourth", True)],
+                         [(s[0].rsplit('/', 1)[1], s[1])
+                          for s in worker.stats])
+
+
+class TestRunScript(tests.TestCaseWithTransport):
+
+    def setUp(self):
+        super(TestRunScript, self).setUp()
+        self.transport_readonly_server = http_server.HttpServer
+
+    def test_run_script(self):
+        self.build_tree(['file1', 'file2', 'file3', 'file4'])
+        url = self.get_readonly_url()
+        self.build_tree_contents([('localhost.script', """{
+    "parameters": {"base_url": "%s", "blocking_timeout": 0.1},
+    "requests": [
+        {"thread": "1", "relpath": "file1"},
+        {"thread": "2", "relpath": "file2"},
+        {"thread": "1", "relpath": "file3"},
+        {"thread": "2", "relpath": "file4"}
+    ]
+}""" % (url,))])
+        script = load_test.run_script('localhost.script')
+        worker = script._threads["1"][0]
+        self.assertEqual([("file1", True), ('file3', True)],
+                         [(s[0].rsplit('/', 1)[1], s[1])
+                          for s in worker.stats])
+        worker = script._threads["2"][0]
+        self.assertEqual([("file2", True), ("file4", True)],
+                         [(s[0].rsplit('/', 1)[1], s[1])
+                          for s in worker.stats])



More information about the bazaar-commits mailing list