[apparmor] [patch] utils: add python coverage generation

Steve Beattie steve at nxnw.org
Mon Oct 27 19:51:16 UTC 2014


Hello,

On Sun, Oct 26, 2014 at 12:49:12AM +0200, Christian Boltz wrote:
> > To generate the coverage data, in the test subdirectory, do:
> > 
> >   make coverage
> > 
> > This essentially runs make check, using a single python interpreter,
> > and records which lines and branches of the python code were
> > exercised.
> > 
> > To view a text based report, after generating the coverage data, do:
> >
> >   make coverage-report
> 
> Instead of "after generating the coverage data", can't you just use a 
> make dependency like
>     coverage-report: coverage
>     coverage-html: coverage
> ?
> 
> (I noticed that you add coverage to .PHONY, and I understand the reason 
> - getting the dependencies 100% correct would be hard[1]. Nevertheless I 
> hope that we can use a make dependency for coverage-report and coverage-
> html)
> 

You are correct that I worried about getting the dependencies correct,
and/or corner cases where potential users would end up looking at
out-of-sync coverage data.

> [1] hmm - what about   $(wildcard ../aa-* ../apparmor/*.py)   ?

To wit: relying on the above misses any changes to the actual
test-*.py files themselves; one of the primary purposes to generate
coverage data is to give people an incentive to add tests, and the
reported coverage percentages provide a feedback mechanism that might
do so (e.g. "Ooh, I increased the amount covered by 5 percentage
points!"). Editing/adding tests and seeing no change in the coverage
report could be a bit demoralizing.

Anyhow. I went ahead and updated the patch to attempt to do this. I
think I got the dependencies in good shape. Patch follows:

utils: add python coverage generation

This patch adds support for generating test coverage information for the
python utils.

To view a text based report, in the test subdirectory do:

  make coverage-report

To generate detailed html reports, do:

  make coverage-html

And then point your web browser at
$(YOUR_CURRENT_WORKING_TREE)/utils/test/htmlcov/index.html .
An alternate output location can be specified by setting the
COVERAGE_OUT variable, e.g.

  make coverage-html COVERAGE_OUT=/tmp/coverage/

(the output directory does not need to exist beforehand.)

To generate only the coverage data, do:

  make coverage

or

  make .coverage

(The coverage data generated by python is stored in the .coverage
file.)  This essentially runs make check, using a single python
interpreter, and records which lines and branches of the python code
were exercised.

Signed-off-by: Steve Beattie <steve at nxnw.org>
---
 utils/test/Makefile |   24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

Index: b/utils/test/Makefile
===================================================================
--- a/utils/test/Makefile
+++ b/utils/test/Makefile
@@ -26,13 +26,31 @@ common/Make.rules: $(COMMONDIR)/Make.rul
 	ln -sf $(COMMONDIR) .
 endif
 
-.PHONY: clean check
+COVERAGE_OMIT=test-*.py,common_test.py
+ifneq ($(COVERAGE_OUT), )
+HTML_COVR_ARGS=-d $(COVERAGE_OUT)
+endif
+
+.PHONY: clean check coverage coverage-report coverage-html
 ifndef VERBOSE
-.SILENT: clean check
+.SILENT: clean check .coverage coverage coverage-report coverage-html
 endif
 
 clean: _clean
-	rm -rf __pycache__/ common
+	rm -rf __pycache__/ common .coverage htmlcov
 
 check:
 	export PYTHONPATH=.. ; $(foreach test, $(wildcard test-*.py), $(call pyalldo, $(test)))
+
+.coverage: $(wildcard ../aa-* ../apparmor/*.py test-*.py)
+	export PYTHONPATH=.. ; $(foreach test, $(wildcard test-*.py), $(PYTHON) -m coverage run --branch -p $(test); )
+	$(PYTHON) -m coverage combine
+
+coverage: .coverage
+
+coverage-report: .coverage
+	$(PYTHON) -m coverage report --omit="$(COVERAGE_OMIT)"
+
+coverage-html: .coverage
+	$(PYTHON) -m coverage html --omit="$(COVERAGE_OMIT)" $(HTML_COVR_ARGS)
+

-- 
Steve Beattie
<sbeattie at ubuntu.com>
http://NxNW.org/~steve/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/apparmor/attachments/20141027/26388eba/attachment.pgp>


More information about the AppArmor mailing list