Rev 2379: (robertc) Merge bzrlib.strace module which adds support for stracing individual callables from within bzrlib. in http://people.ubuntu.com/~robertc/baz2.0/integration
Robert Collins
robertc at robertcollins.net
Tue Mar 27 08:58:39 BST 2007
At http://people.ubuntu.com/~robertc/baz2.0/integration
------------------------------------------------------------
revno: 2379
revision-id: robertc at robertcollins.net-20070327075834-51wgidn6o63h8lqo
parent: pqm at pqm.ubuntu.com-20070327070037-cd7e6fa939b1d08d
parent: robertc at robertcollins.net-20070327073712-6v27yeyl9unj6p9o
committer: Robert Collins <robertc at robertcollins.net>
branch nick: integration
timestamp: Tue 2007-03-27 17:58:34 +1000
message:
(robertc) Merge bzrlib.strace module which adds support for stracing individual callables from within bzrlib.
added:
bzrlib/strace.py strace.py-20070323001526-6zquhhw8leb9m6j8-1
bzrlib/tests/test_strace.py test_strace.py-20070323001526-6zquhhw8leb9m6j8-2
modified:
NEWS NEWS-20050323055033-4e00b5db738777ff
bzrlib/builtins.py builtins.py-20050830033751-fc01482b9ca23183
bzrlib/tests/__init__.py selftest.py-20050531073622-8d0e3c8845c97a64
bzrlib/tests/test_selftest.py test_selftest.py-20051202044319-c110a115d8c0456a
------------------------------------------------------------
revno: 2367.1.9
merged: robertc at robertcollins.net-20070327073712-6v27yeyl9unj6p9o
parent: robertc at robertcollins.net-20070327070732-jrx0o1jgqond0xpb
committer: Robert Collins <robertc at robertcollins.net>
branch nick: benchmark-strace
timestamp: Tue 2007-03-27 17:37:12 +1000
message:
Review feedback.
------------------------------------------------------------
revno: 2367.1.8
merged: robertc at robertcollins.net-20070327070732-jrx0o1jgqond0xpb
parent: robertc at robertcollins.net-20070323001831-czc15dtustiat7j7
committer: Robert Collins <robertc at robertcollins.net>
branch nick: benchmark-strace
timestamp: Tue 2007-03-27 17:07:32 +1000
message:
Whitespace.
------------------------------------------------------------
revno: 2367.1.7
merged: robertc at robertcollins.net-20070323001831-czc15dtustiat7j7
parent: robertc at robertcollins.net-20070322214342-58nmsg7pvh6ghc8b
committer: Robert Collins <robertc at robertcollins.net>
branch nick: benchmark-strace
timestamp: Fri 2007-03-23 11:18:31 +1100
message:
Added ``bzrlib.strace.strace`` which will strace a single callable and
return a StraceResult object which contains just the syscalls involved
in running it. (Robert Collins)
------------------------------------------------------------
revno: 2367.1.6
merged: robertc at robertcollins.net-20070322214342-58nmsg7pvh6ghc8b
parent: robertc at robertcollins.net-20070322121903-8sqxzfr5eqvsx5yp
committer: Robert Collins <robertc at robertcollins.net>
branch nick: test-prereqs
timestamp: Fri 2007-03-23 08:43:42 +1100
message:
Allow per-test-fixture feature requirements via 'requireFeature'.(Robert Collins)
------------------------------------------------------------
revno: 2367.1.5
merged: robertc at robertcollins.net-20070322121903-8sqxzfr5eqvsx5yp
parent: robertc at robertcollins.net-20070322105438-gt9qu83u9ml5aubo
committer: Robert Collins <robertc at robertcollins.net>
branch nick: test-prereqs
timestamp: Thu 2007-03-22 23:19:03 +1100
message:
Implement reporting of Unsupported tests in the bzr test result and runner
classes. (Robert Collins)
------------------------------------------------------------
revno: 2367.1.4
merged: robertc at robertcollins.net-20070322105438-gt9qu83u9ml5aubo
parent: robertc at robertcollins.net-20070322100744-96m81fcue8hgsfnd
committer: Robert Collins <robertc at robertcollins.net>
branch nick: test-prereqs
timestamp: Thu 2007-03-22 21:54:38 +1100
message:
Add operating system Feature model to bzrlib.tests to allow writing tests
that can declare their needed dependencies and be cleanly disabled.
------------------------------------------------------------
revno: 2367.1.3
merged: robertc at robertcollins.net-20070322100744-96m81fcue8hgsfnd
parent: robertc at robertcollins.net-20070322090459-q0gjq21qgrj453hf
committer: Robert Collins <robertc at robertcollins.net>
branch nick: test-prereqs
timestamp: Thu 2007-03-22 21:07:44 +1100
message:
Add support for calling addNotSupported on TestResults to bzr TestCase's
when _test_needs_features contains unavailable features. (Robert Collins)
------------------------------------------------------------
revno: 2367.1.2
merged: robertc at robertcollins.net-20070322090459-q0gjq21qgrj453hf
parent: robertc at robertcollins.net-20070321042802-1bxr1t97046woojb
committer: Robert Collins <robertc at robertcollins.net>
branch nick: test-prereqs
timestamp: Thu 2007-03-22 20:04:59 +1100
message:
Some minor cleanups of test code, and implement KnownFailure support as
per http://bazaar-vcs.org/BzrExtendTestSuite. (Robert Collins)
=== added file 'bzrlib/strace.py'
--- a/bzrlib/strace.py 1970-01-01 00:00:00 +0000
+++ b/bzrlib/strace.py 2007-03-27 07:37:12 +0000
@@ -0,0 +1,88 @@
+# Copyright (C) 2007 Canonical Ltd
+# Authors: Robert Collins <robert.collins at canonical.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+"""Support for running strace against the current process."""
+
+import os
+import signal
+import subprocess
+import tempfile
+
+# this is currently test-focused, so importing bzrlib.tests is ok. We might
+# want to move feature to its own module though.
+from bzrlib.tests import Feature
+
+
+def strace(function, *args, **kwargs):
+ """Invoke strace on function.
+
+ :return: a tuple: function-result, a StraceResult.
+ """
+ # capture strace output to a file
+ log_file = tempfile.TemporaryFile()
+ log_file_fd = log_file.fileno()
+ pid = os.getpid()
+ # start strace
+ proc = subprocess.Popen(['strace',
+ '-f', '-r', '-tt', '-p', str(pid),
+ ],
+ stderr=log_file_fd,
+ stdout=log_file_fd)
+ # TODO? confirm its started (test suite should be sufficient)
+ # (can loop on proc.pid, but that may not indicate started and attached.)
+ result = function(*args, **kwargs)
+ # stop strace
+ os.kill(proc.pid, signal.SIGQUIT)
+ proc.communicate()
+ # grab the log
+ log_file.seek(0)
+ log = log_file.read()
+ log_file.close()
+ return result, StraceResult(log)
+
+
+class StraceResult(object):
+ """The result of stracing a function."""
+
+ def __init__(self, raw_log):
+ """Create a StraceResult.
+
+ :param raw_log: The output that strace created.
+ """
+ self.raw_log = raw_log
+
+
+class _StraceFeature(Feature):
+
+ def _probe(self):
+ try:
+ proc = subprocess.Popen(['strace'],
+ stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE)
+ proc.communicate()
+ return True
+ except OSError, e:
+ if e.errno == errno.ENOENT:
+ # strace is not installed
+ return False
+ else:
+ raise
+
+ def feature_name(self):
+ return 'strace'
+
+StraceFeature = _StraceFeature()
=== added file 'bzrlib/tests/test_strace.py'
--- a/bzrlib/tests/test_strace.py 1970-01-01 00:00:00 +0000
+++ b/bzrlib/tests/test_strace.py 2007-03-27 07:37:12 +0000
@@ -0,0 +1,68 @@
+# Copyright (C) 2007 Canonical Ltd
+# Authors: Robert Collins <robert.collins at canonical.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+"""Tests for the strace-invoking support."""
+
+import subprocess
+
+from bzrlib.strace import StraceFeature, strace, StraceResult
+from bzrlib.tests import TestCaseWithTransport
+
+
+class TestStraceFeature(TestCaseWithTransport):
+
+ def test_strace_detection(self):
+ """Strace is available if its runnable."""
+ try:
+ proc = subprocess.Popen(['strace'],
+ stderr=subprocess.PIPE,
+ stdout=subprocess.PIPE)
+ proc.communicate()
+ found_strace = True
+ except OSError, e:
+ if e.errno == errno.ENOENT:
+ # strace is not installed
+ found_strace = False
+ else:
+ raise
+ self.assertEqual(found_strace, StraceFeature.available())
+
+
+class TestStrace(TestCaseWithTransport):
+
+ _test_needs_features = [StraceFeature]
+
+ def test_strace_callable_is_called(self):
+ output = []
+ def function(positional, *args, **kwargs):
+ output.append((positional, args, kwargs))
+ strace(function, "a", "b", c="c")
+ self.assertEqual([("a", ("b",), {"c":"c"})], output)
+
+ def test_strace_callable_result(self):
+ def function():
+ return "foo"
+ result, strace_result = strace(function)
+ self.assertEqual("foo", result)
+ self.assertIsInstance(strace_result, StraceResult)
+
+ def test_strace_result_has_raw_log(self):
+ """Checks that a reasonable raw strace log was found by strace."""
+ def function():
+ self.build_tree(['myfile'])
+ _, result = strace(function)
+ self.assertContainsRe(result.raw_log, 'myfile')
=== modified file 'NEWS'
--- a/NEWS 2007-03-27 05:53:02 +0000
+++ b/NEWS 2007-03-27 07:58:34 +0000
@@ -5,6 +5,13 @@
* bzrlib API compatability with 0.8 has been dropped, cleaning up some
code paths. (Robert Collins)
+ TESTING:
+
+ * Added ``bzrlib.strace.strace`` which will strace a single callable and
+ return a StraceResult object which contains just the syscalls involved
+ in running it. (Robert Collins)
+
+
bzr 0.15 (not finalised)
INTERNALS:
=== modified file 'bzrlib/builtins.py'
--- a/bzrlib/builtins.py 2007-03-27 00:35:29 +0000
+++ b/bzrlib/builtins.py 2007-03-27 07:58:34 +0000
@@ -2243,7 +2243,7 @@
@display_command
def printme(self, branch):
- print branch.nick
+ print branch.nick
class cmd_selftest(Command):
=== modified file 'bzrlib/tests/__init__.py'
--- a/bzrlib/tests/__init__.py 2007-03-15 22:35:35 +0000
+++ b/bzrlib/tests/__init__.py 2007-03-23 00:18:31 +0000
@@ -180,7 +180,9 @@
self.num_tests = num_tests
self.error_count = 0
self.failure_count = 0
+ self.known_failure_count = 0
self.skip_count = 0
+ self.unsupported = {}
self.count = 0
self._overall_start_time = time.time()
@@ -221,32 +223,46 @@
"""Record that a test has started."""
self._start_time = time.time()
+ def _cleanupLogFile(self, test):
+ # We can only do this if we have one of our TestCases, not if
+ # we have a doctest.
+ setKeepLogfile = getattr(test, 'setKeepLogfile', None)
+ if setKeepLogfile is not None:
+ setKeepLogfile()
+
def addError(self, test, err):
+ self.extractBenchmarkTime(test)
+ self._cleanupLogFile(test)
if isinstance(err[1], TestSkipped):
- return self.addSkipped(test, err)
+ return self.addSkipped(test, err)
+ elif isinstance(err[1], UnavailableFeature):
+ return self.addNotSupported(test, err[1].args[0])
unittest.TestResult.addError(self, test, err)
- # We can only do this if we have one of our TestCases, not if
- # we have a doctest.
- setKeepLogfile = getattr(test, 'setKeepLogfile', None)
- if setKeepLogfile is not None:
- setKeepLogfile()
- self.extractBenchmarkTime(test)
+ self.error_count += 1
self.report_error(test, err)
if self.stop_early:
self.stop()
def addFailure(self, test, err):
+ self._cleanupLogFile(test)
+ self.extractBenchmarkTime(test)
+ if isinstance(err[1], KnownFailure):
+ return self.addKnownFailure(test, err)
unittest.TestResult.addFailure(self, test, err)
- # We can only do this if we have one of our TestCases, not if
- # we have a doctest.
- setKeepLogfile = getattr(test, 'setKeepLogfile', None)
- if setKeepLogfile is not None:
- setKeepLogfile()
- self.extractBenchmarkTime(test)
+ self.failure_count += 1
self.report_failure(test, err)
if self.stop_early:
self.stop()
+ def addKnownFailure(self, test, err):
+ self.known_failure_count += 1
+ self.report_known_failure(test, err)
+
+ def addNotSupported(self, test, feature):
+ self.unsupported.setdefault(str(feature), 0)
+ self.unsupported[str(feature)] += 1
+ self.report_unsupported(test, feature)
+
def addSuccess(self, test):
self.extractBenchmarkTime(test)
if self._bench_history is not None:
@@ -258,7 +274,6 @@
unittest.TestResult.addSuccess(self, test)
def addSkipped(self, test, skip_excinfo):
- self.extractBenchmarkTime(test)
self.report_skip(test, skip_excinfo)
# seems best to treat this as success from point-of-view of unittest
# -- it actually does nothing so it barely matters :)
@@ -301,12 +316,22 @@
class TextTestResult(ExtendedTestResult):
"""Displays progress and results of tests in text form"""
- def __init__(self, *args, **kw):
- ExtendedTestResult.__init__(self, *args, **kw)
- self.pb = self.ui.nested_progress_bar()
+ def __init__(self, stream, descriptions, verbosity,
+ bench_history=None,
+ num_tests=None,
+ pb=None,
+ ):
+ ExtendedTestResult.__init__(self, stream, descriptions, verbosity,
+ bench_history, num_tests)
+ if pb is None:
+ self.pb = self.ui.nested_progress_bar()
+ self._supplied_pb = False
+ else:
+ self.pb = pb
+ self._supplied_pb = True
self.pb.show_pct = False
self.pb.show_spinner = False
- self.pb.show_eta = False,
+ self.pb.show_eta = False,
self.pb.show_count = False
self.pb.show_bar = False
@@ -322,8 +347,12 @@
a += ', %d errors' % self.error_count
if self.failure_count:
a += ', %d failed' % self.failure_count
+ if self.known_failure_count:
+ a += ', %d known failures' % self.known_failure_count
if self.skip_count:
a += ', %d skipped' % self.skip_count
+ if self.unsupported:
+ a += ', %d missing features' % len(self.unsupported)
a += ']'
return a
@@ -342,19 +371,21 @@
return self._shortened_test_description(test)
def report_error(self, test, err):
- self.error_count += 1
self.pb.note('ERROR: %s\n %s\n',
self._test_description(test),
err[1],
)
def report_failure(self, test, err):
- self.failure_count += 1
self.pb.note('FAIL: %s\n %s\n',
self._test_description(test),
err[1],
)
+ def report_known_failure(self, test, err):
+ self.pb.note('XFAIL: %s\n%s\n',
+ self._test_description(test), err[1])
+
def report_skip(self, test, skip_excinfo):
self.skip_count += 1
if False:
@@ -371,11 +402,15 @@
# progress bar...
self.pb.note('SKIP: %s', skip_excinfo[1])
+ def report_unsupported(self, test, feature):
+ """test cannot be run because feature is missing."""
+
def report_cleaning_up(self):
self.pb.update('cleaning up...')
def finished(self):
- self.pb.finished()
+ if not self._supplied_pb:
+ self.pb.finished()
class VerboseTestResult(ExtendedTestResult):
@@ -414,22 +449,27 @@
return '%s%s' % (indent, err[1])
def report_error(self, test, err):
- self.error_count += 1
self.stream.writeln('ERROR %s\n%s'
% (self._testTimeString(),
self._error_summary(err)))
def report_failure(self, test, err):
- self.failure_count += 1
self.stream.writeln(' FAIL %s\n%s'
% (self._testTimeString(),
self._error_summary(err)))
+ def report_known_failure(self, test, err):
+ self.stream.writeln('XFAIL %s\n%s'
+ % (self._testTimeString(),
+ self._error_summary(err)))
+
def report_success(self, test):
self.stream.writeln(' OK %s' % self._testTimeString())
for bench_called, stats in getattr(test, '_benchcalls', []):
self.stream.writeln('LSProf output for %s(%s, %s)' % bench_called)
stats.pprint(file=self.stream)
+ # flush the stream so that we get smooth output. This verbose mode is
+ # used to show the output in PQM.
self.stream.flush()
def report_skip(self, test, skip_excinfo):
@@ -438,6 +478,12 @@
% (self._testTimeString(),
self._error_summary(skip_excinfo)))
+ def report_unsupported(self, test, feature):
+ """test cannot be run because feature is missing."""
+ self.stream.writeln("NODEP %s\n The feature '%s' is not available."
+ %(self._testTimeString(), feature))
+
+
class TextTestRunner(object):
stop_on_failure = False
@@ -486,13 +532,25 @@
if errored:
if failed: self.stream.write(", ")
self.stream.write("errors=%d" % errored)
+ if result.known_failure_count:
+ if failed or errored: self.stream.write(", ")
+ self.stream.write("known_failure_count=%d" %
+ result.known_failure_count)
self.stream.writeln(")")
else:
- self.stream.writeln("OK")
+ if result.known_failure_count:
+ self.stream.writeln("OK (known_failures=%d)" %
+ result.known_failure_count)
+ else:
+ self.stream.writeln("OK")
if result.skip_count > 0:
skipped = result.skip_count
self.stream.writeln('%d test%s skipped' %
(skipped, skipped != 1 and "s" or ""))
+ if result.unsupported:
+ for feature, count in sorted(result.unsupported.items()):
+ self.stream.writeln("Missing feature '%s' skipped %d tests." %
+ (feature, count))
result.report_cleaning_up()
# This is still a little bogus,
# but only a little. Folk not using our testrunner will
@@ -545,6 +603,23 @@
"""Indicates that a test was intentionally skipped, rather than failing."""
+class KnownFailure(AssertionError):
+ """Indicates that a test failed in a precisely expected manner.
+
+ Such failures dont block the whole test suite from passing because they are
+ indicators of partially completed code or of future work. We have an
+ explicit error for them so that we can ensure that they are always visible:
+ KnownFailures are always shown in the output of bzr selftest.
+ """
+
+
+class UnavailableFeature(Exception):
+ """A feature required for this test was not available.
+
+ The feature should be used to construct the exception.
+ """
+
+
class CommandFailed(Exception):
pass
@@ -970,6 +1045,23 @@
def _restoreHooks(self):
bzrlib.branch.Branch.hooks = self._preserved_hooks
+ def knownFailure(self, reason):
+ """This test has failed for some known reason."""
+ raise KnownFailure(reason)
+
+ def run(self, result=None):
+ if result is None: result = self.defaultTestResult()
+ for feature in getattr(self, '_test_needs_features', []):
+ if not feature.available():
+ result.startTest(self)
+ if getattr(result, 'addNotSupported', None):
+ result.addNotSupported(self, feature)
+ else:
+ result.addSuccess(self)
+ result.stopTest(self)
+ return
+ return unittest.TestCase.run(self, result)
+
def tearDown(self):
self._runCleanups()
unittest.TestCase.tearDown(self)
@@ -1045,6 +1137,14 @@
"""Shortcut that splits cmd into words, runs, and returns stdout"""
return self.run_bzr_captured(cmd.split(), retcode=retcode)[0]
+ def requireFeature(self, feature):
+ """This test requires a specific feature is available.
+
+ :raises UnavailableFeature: When feature is not available.
+ """
+ if not feature.available():
+ raise UnavailableFeature(feature)
+
def run_bzr_captured(self, argv, retcode=0, encoding=None, stdin=None,
working_dir=None):
"""Invoke bzr and return (stdout, stderr).
@@ -2002,6 +2102,7 @@
'bzrlib.tests.test_ssh_transport',
'bzrlib.tests.test_status',
'bzrlib.tests.test_store',
+ 'bzrlib.tests.test_strace',
'bzrlib.tests.test_subsume',
'bzrlib.tests.test_symbol_versioning',
'bzrlib.tests.test_tag',
@@ -2093,3 +2194,31 @@
if not quiet:
print 'delete directory:', i
shutil.rmtree(i)
+
+
+class Feature(object):
+ """An operating system Feature."""
+
+ def __init__(self):
+ self._available = None
+
+ def available(self):
+ """Is the feature available?
+
+ :return: True if the feature is available.
+ """
+ if self._available is None:
+ self._available = self._probe()
+ return self._available
+
+ def _probe(self):
+ """Implement this method in concrete features.
+
+ :return: True if the feature is available.
+ """
+ raise NotImplementedError
+
+ def __str__(self):
+ if getattr(self, 'feature_name', None):
+ return self.feature_name()
+ return self.__class__.__name__
=== modified file 'bzrlib/tests/test_selftest.py'
--- a/bzrlib/tests/test_selftest.py 2007-03-12 20:55:23 +0000
+++ b/bzrlib/tests/test_selftest.py 2007-03-22 21:43:42 +0000
@@ -38,6 +38,9 @@
from bzrlib.symbol_versioning import zero_ten, zero_eleven
from bzrlib.tests import (
ChrootedTestCase,
+ ExtendedTestResult,
+ Feature,
+ KnownFailure,
TestCase,
TestCaseInTempDir,
TestCaseWithMemoryTransport,
@@ -45,6 +48,7 @@
TestSkipped,
TestSuite,
TextTestRunner,
+ UnavailableFeature,
)
from bzrlib.tests.test_sftp_transport import TestCaseWithSFTPServer
from bzrlib.tests.TestUtil import _load_module_by_name
@@ -690,6 +694,186 @@
self.assertContainsRe(output,
r"LSProf output for <type 'unicode'>\(\('world',\), {'errors': 'replace'}\)\n")
+ def test_known_failure(self):
+ """A KnownFailure being raised should trigger several result actions."""
+ class InstrumentedTestResult(ExtendedTestResult):
+
+ def report_test_start(self, test): pass
+ def report_known_failure(self, test, err):
+ self._call = test, err
+ result = InstrumentedTestResult(None, None, None, None)
+ def test_function():
+ raise KnownFailure('failed!')
+ test = unittest.FunctionTestCase(test_function)
+ test.run(result)
+ # it should invoke 'report_known_failure'.
+ self.assertEqual(2, len(result._call))
+ self.assertEqual(test, result._call[0])
+ self.assertEqual(KnownFailure, result._call[1][0])
+ self.assertIsInstance(result._call[1][1], KnownFailure)
+ # we dont introspec the traceback, if the rest is ok, it would be
+ # exceptional for it not to be.
+ # it should update the known_failure_count on the object.
+ self.assertEqual(1, result.known_failure_count)
+ # the result should be successful.
+ self.assertTrue(result.wasSuccessful())
+
+ def test_verbose_report_known_failure(self):
+ # verbose test output formatting
+ result_stream = StringIO()
+ result = bzrlib.tests.VerboseTestResult(
+ unittest._WritelnDecorator(result_stream),
+ descriptions=0,
+ verbosity=2,
+ )
+ test = self.get_passing_test()
+ result.startTest(test)
+ result.extractBenchmarkTime(test)
+ prefix = len(result_stream.getvalue())
+ # the err parameter has the shape:
+ # (class, exception object, traceback)
+ # KnownFailures dont get their tracebacks shown though, so we
+ # can skip that.
+ err = (KnownFailure, KnownFailure('foo'), None)
+ result.report_known_failure(test, err)
+ output = result_stream.getvalue()[prefix:]
+ lines = output.splitlines()
+ self.assertEqual(lines, ['XFAIL 0ms', ' foo'])
+
+ def test_text_report_known_failure(self):
+ # text test output formatting
+ pb = MockProgress()
+ result = bzrlib.tests.TextTestResult(
+ None,
+ descriptions=0,
+ verbosity=1,
+ pb=pb,
+ )
+ test = self.get_passing_test()
+ # this seeds the state to handle reporting the test.
+ result.startTest(test)
+ result.extractBenchmarkTime(test)
+ # the err parameter has the shape:
+ # (class, exception object, traceback)
+ # KnownFailures dont get their tracebacks shown though, so we
+ # can skip that.
+ err = (KnownFailure, KnownFailure('foo'), None)
+ result.report_known_failure(test, err)
+ self.assertEqual(
+ [
+ ('update', '[1 in 0s] passing_test', None, None),
+ ('note', 'XFAIL: %s\n%s\n', ('passing_test', err[1]))
+ ],
+ pb.calls)
+ # known_failures should be printed in the summary, so if we run a test
+ # after there are some known failures, the update prefix should match
+ # this.
+ result.known_failure_count = 3
+ test.run(result)
+ self.assertEqual(
+ [
+ ('update', '[2 in 0s, 3 known failures] passing_test', None, None),
+ ],
+ pb.calls[2:])
+
+ def get_passing_test(self):
+ """Return a test object that can't be run usefully."""
+ def passing_test():
+ pass
+ return unittest.FunctionTestCase(passing_test)
+
+ def test_add_not_supported(self):
+ """Test the behaviour of invoking addNotSupported."""
+ class InstrumentedTestResult(ExtendedTestResult):
+ def report_test_start(self, test): pass
+ def report_unsupported(self, test, feature):
+ self._call = test, feature
+ result = InstrumentedTestResult(None, None, None, None)
+ test = SampleTestCase('_test_pass')
+ feature = Feature()
+ result.startTest(test)
+ result.addNotSupported(test, feature)
+ # it should invoke 'report_unsupported'.
+ self.assertEqual(2, len(result._call))
+ self.assertEqual(test, result._call[0])
+ self.assertEqual(feature, result._call[1])
+ # the result should be successful.
+ self.assertTrue(result.wasSuccessful())
+ # it should record the test against a count of tests not run due to
+ # this feature.
+ self.assertEqual(1, result.unsupported['Feature'])
+ # and invoking it again should increment that counter
+ result.addNotSupported(test, feature)
+ self.assertEqual(2, result.unsupported['Feature'])
+
+ def test_verbose_report_unsupported(self):
+ # verbose test output formatting
+ result_stream = StringIO()
+ result = bzrlib.tests.VerboseTestResult(
+ unittest._WritelnDecorator(result_stream),
+ descriptions=0,
+ verbosity=2,
+ )
+ test = self.get_passing_test()
+ feature = Feature()
+ result.startTest(test)
+ result.extractBenchmarkTime(test)
+ prefix = len(result_stream.getvalue())
+ result.report_unsupported(test, feature)
+ output = result_stream.getvalue()[prefix:]
+ lines = output.splitlines()
+ self.assertEqual(lines, ['NODEP 0ms', " The feature 'Feature' is not available."])
+
+ def test_text_report_unsupported(self):
+ # text test output formatting
+ pb = MockProgress()
+ result = bzrlib.tests.TextTestResult(
+ None,
+ descriptions=0,
+ verbosity=1,
+ pb=pb,
+ )
+ test = self.get_passing_test()
+ feature = Feature()
+ # this seeds the state to handle reporting the test.
+ result.startTest(test)
+ result.extractBenchmarkTime(test)
+ result.report_unsupported(test, feature)
+ # no output on unsupported features
+ self.assertEqual(
+ [('update', '[1 in 0s] passing_test', None, None)
+ ],
+ pb.calls)
+ # the number of missing features should be printed in the progress
+ # summary, so check for that.
+ result.unsupported = {'foo':0, 'bar':0}
+ test.run(result)
+ self.assertEqual(
+ [
+ ('update', '[2 in 0s, 2 missing features] passing_test', None, None),
+ ],
+ pb.calls[1:])
+
+ def test_unavailable_exception(self):
+ """An UnavailableFeature being raised should invoke addNotSupported."""
+ class InstrumentedTestResult(ExtendedTestResult):
+
+ def report_test_start(self, test): pass
+ def addNotSupported(self, test, feature):
+ self._call = test, feature
+ result = InstrumentedTestResult(None, None, None, None)
+ feature = Feature()
+ def test_function():
+ raise UnavailableFeature(feature)
+ test = unittest.FunctionTestCase(test_function)
+ test.run(result)
+ # it should invoke 'addNotSupported'.
+ self.assertEqual(2, len(result._call))
+ self.assertEqual(test, result._call[0])
+ self.assertEqual(feature, result._call[1])
+ # and not count as an error
+ self.assertEqual(0, result.error_count)
+
class TestRunner(TestCase):
@@ -712,6 +896,50 @@
finally:
TestCaseInTempDir.TEST_ROOT = old_root
+ def test_known_failure_failed_run(self):
+ # run a test that generates a known failure which should be printed in
+ # the final output when real failures occur.
+ def known_failure_test():
+ raise KnownFailure('failed')
+ test = unittest.TestSuite()
+ test.addTest(unittest.FunctionTestCase(known_failure_test))
+ def failing_test():
+ raise AssertionError('foo')
+ test.addTest(unittest.FunctionTestCase(failing_test))
+ stream = StringIO()
+ runner = TextTestRunner(stream=stream)
+ result = self.run_test_runner(runner, test)
+ lines = stream.getvalue().splitlines()
+ self.assertEqual([
+ '',
+ '======================================================================',
+ 'FAIL: unittest.FunctionTestCase (failing_test)',
+ '----------------------------------------------------------------------',
+ 'Traceback (most recent call last):',
+ ' raise AssertionError(\'foo\')',
+ 'AssertionError: foo',
+ '',
+ '----------------------------------------------------------------------',
+ '',
+ 'FAILED (failures=1, known_failure_count=1)'],
+ lines[0:5] + lines[6:10] + lines[11:])
+
+ def test_known_failure_ok_run(self):
+ # run a test that generates a known failure which should be printed in the final output.
+ def known_failure_test():
+ raise KnownFailure('failed')
+ test = unittest.FunctionTestCase(known_failure_test)
+ stream = StringIO()
+ runner = TextTestRunner(stream=stream)
+ result = self.run_test_runner(runner, test)
+ self.assertEqual(
+ '\n'
+ '----------------------------------------------------------------------\n'
+ 'Ran 1 test in 0.000s\n'
+ '\n'
+ 'OK (known_failures=1)\n',
+ stream.getvalue())
+
def test_skipped_test(self):
# run a test that is skipped, and check the suite as a whole still
# succeeds.
@@ -765,6 +993,31 @@
# Check if cleanup was called the right number of times.
self.assertEqual(0, test.counter)
+ def test_unsupported_features_listed(self):
+ """When unsupported features are encountered they are detailed."""
+ class Feature1(Feature):
+ def _probe(self): return False
+ class Feature2(Feature):
+ def _probe(self): return False
+ # create sample tests
+ test1 = SampleTestCase('_test_pass')
+ test1._test_needs_features = [Feature1()]
+ test2 = SampleTestCase('_test_pass')
+ test2._test_needs_features = [Feature2()]
+ test = unittest.TestSuite()
+ test.addTest(test1)
+ test.addTest(test2)
+ stream = StringIO()
+ runner = TextTestRunner(stream=stream)
+ result = self.run_test_runner(runner, test)
+ lines = stream.getvalue().splitlines()
+ self.assertEqual([
+ 'OK',
+ "Missing feature 'Feature1' skipped 1 tests.",
+ "Missing feature 'Feature2' skipped 1 tests.",
+ ],
+ lines[-3:])
+
def test_bench_history(self):
# tests that the running the benchmark produces a history file
# containing a timestamp and the revision id of the bzrlib source which
@@ -843,6 +1096,12 @@
self.assertEqual(log, test._log_contents)
+class SampleTestCase(TestCase):
+
+ def _test_pass(self):
+ pass
+
+
class TestTestCase(TestCase):
"""Tests that test the core bzrlib TestCase."""
@@ -924,6 +1183,81 @@
self.assertIsInstance(self._benchcalls[0][1], bzrlib.lsprof.Stats)
self.assertIsInstance(self._benchcalls[1][1], bzrlib.lsprof.Stats)
+ def test_knownFailure(self):
+ """Self.knownFailure() should raise a KnownFailure exception."""
+ self.assertRaises(KnownFailure, self.knownFailure, "A Failure")
+
+ def test_requireFeature_available(self):
+ """self.requireFeature(available) is a no-op."""
+ class Available(Feature):
+ def _probe(self):return True
+ feature = Available()
+ self.requireFeature(feature)
+
+ def test_requireFeature_unavailable(self):
+ """self.requireFeature(unavailable) raises UnavailableFeature."""
+ class Unavailable(Feature):
+ def _probe(self):return False
+ feature = Unavailable()
+ self.assertRaises(UnavailableFeature, self.requireFeature, feature)
+
+ def test_run_no_parameters(self):
+ test = SampleTestCase('_test_pass')
+ test.run()
+
+ def test_run_enabled_unittest_result(self):
+ """Test we revert to regular behaviour when the test is enabled."""
+ test = SampleTestCase('_test_pass')
+ class EnabledFeature(object):
+ def available(self):
+ return True
+ test._test_needs_features = [EnabledFeature()]
+ result = unittest.TestResult()
+ test.run(result)
+ self.assertEqual(1, result.testsRun)
+ self.assertEqual([], result.errors)
+ self.assertEqual([], result.failures)
+
+ def test_run_disabled_unittest_result(self):
+ """Test our compatability for disabled tests with unittest results."""
+ test = SampleTestCase('_test_pass')
+ class DisabledFeature(object):
+ def available(self):
+ return False
+ test._test_needs_features = [DisabledFeature()]
+ result = unittest.TestResult()
+ test.run(result)
+ self.assertEqual(1, result.testsRun)
+ self.assertEqual([], result.errors)
+ self.assertEqual([], result.failures)
+
+ def test_run_disabled_supporting_result(self):
+ """Test disabled tests behaviour with support aware results."""
+ test = SampleTestCase('_test_pass')
+ class DisabledFeature(object):
+ def available(self):
+ return False
+ the_feature = DisabledFeature()
+ test._test_needs_features = [the_feature]
+ class InstrumentedTestResult(unittest.TestResult):
+ def __init__(self):
+ unittest.TestResult.__init__(self)
+ self.calls = []
+ def startTest(self, test):
+ self.calls.append(('startTest', test))
+ def stopTest(self, test):
+ self.calls.append(('stopTest', test))
+ def addNotSupported(self, test, feature):
+ self.calls.append(('addNotSupported', test, feature))
+ result = InstrumentedTestResult()
+ test.run(result)
+ self.assertEqual([
+ ('startTest', test),
+ ('addNotSupported', test, the_feature),
+ ('stopTest', test),
+ ],
+ result.calls)
+
@symbol_versioning.deprecated_function(zero_eleven)
def sample_deprecated_function():
@@ -1085,3 +1419,53 @@
self.assertEquals(['bzr','bzrlib','setup.py',
'test9999.tmp','tests'],
after)
+
+
+class TestKnownFailure(TestCase):
+
+ def test_known_failure(self):
+ """Check that KnownFailure is defined appropriately."""
+ # a KnownFailure is an assertion error for compatability with unaware
+ # runners.
+ self.assertIsInstance(KnownFailure(""), AssertionError)
+
+
+class TestFeature(TestCase):
+
+ def test_caching(self):
+ """Feature._probe is called by the feature at most once."""
+ class InstrumentedFeature(Feature):
+ def __init__(self):
+ Feature.__init__(self)
+ self.calls = []
+ def _probe(self):
+ self.calls.append('_probe')
+ return False
+ feature = InstrumentedFeature()
+ feature.available()
+ self.assertEqual(['_probe'], feature.calls)
+ feature.available()
+ self.assertEqual(['_probe'], feature.calls)
+
+ def test_named_str(self):
+ """Feature.__str__ should thunk to feature_name()."""
+ class NamedFeature(Feature):
+ def feature_name(self):
+ return 'symlinks'
+ feature = NamedFeature()
+ self.assertEqual('symlinks', str(feature))
+
+ def test_default_str(self):
+ """Feature.__str__ should default to __class__.__name__."""
+ class NamedFeature(Feature):
+ pass
+ feature = NamedFeature()
+ self.assertEqual('NamedFeature', str(feature))
+
+
+class TestUnavailableFeature(TestCase):
+
+ def test_access_feature(self):
+ feature = Feature()
+ exception = UnavailableFeature(feature)
+ self.assertIs(feature, exception.args[0])
More information about the bazaar-commits
mailing list