autopkgtest verision deployed on production infrastructure updated

Iain Lane laney at ubuntu.com
Mon Dec 10 10:09:53 UTC 2018


Hi all,

I've just pushed a bunch of commits to our deployed branch of
autopkgtestĀ¹ that bring us up to date with upstream, where the
developers have done a lot of work. Many thanks to them. The most
interesting thing to subscribers to this list will be the changes -
mainly additions - to the list of known test restrictions. I've pasted
the diff of README.package-tests.rst below.

I don't currently expect any breakage to result from this, but if you
notice anything weird please reply here or come to #ubuntu-release and
we'll take a look.

Cheers,

-- 
Iain Lane                                  [ iain at orangesquash.org.uk ]
Debian Developer                                   [ laney at debian.org ]
Ubuntu Developer                                   [ laney at ubuntu.com ]

[1] https://code.launchpad.net/~ubuntu-release/autopkgtest/+git/development/

@@ -237,14 +245,84 @@ needs-reboot
     The test wants to reboot the machine using
     ``/tmp/autopkgtest-reboot`` (see below).
 
-needs-recommends
+needs-recommends (deprecated)
     Enable installation of recommended packages in apt for the test
     dependencies. This does not affect build dependencies.
 
+    Don't use this for new tests, we want to remove it altogether.
+    See https://lists.debian.org/debian-ci/2018/06/msg00016.html
+    and bug 896698 for the rationale.
+
+flaky
+    The test is expected to fail intermittently, and is not suitable for
+    gating continuous integration. This indicates a bug in either the
+    package under test, a dependency or the test itself, but such bugs
+    can be difficult to fix, and it is often difficult to know when the
+    bug has been fixed without running the test for a while. If a
+    ``flaky`` test succeeds, it will be treated like any other
+    successful test, but if it fails it will be treated as though it
+    had been skipped.
+
+skippable
+    The test might need to be skipped for reasons that cannot be
+    described by an existing restriction such as isolation-machine or
+    breaks-testbed, but must instead be detected at runtime. If the
+    test exits with status 77 (a convention borrowed from Automake), it
+    will be treated as though it had been skipped. If it exits with any
+    other status, its success or failure will be derived from the exit
+    status and stderr as usual. Test authors must be careful to ensure
+    that ``skippable`` tests never exit with status 77 for reasons that
+    should be treated as a failure.
+
+skip-not-installable
+    This test might have test dependencies that can't be fulfilled on all
+    architectures. Therefore, when apt-get installs the dependencies, it will
+    fail. Don't treat this as a test failure, but instead treat it as if the
+    test was skipped.
+
+hint-testsuite-triggers
+    This test exists purely as a hint to suggest when rerunning the
+    tests is likely to be useful.  Specifically, it exists to
+    influence the way dpkg-source generates the Testsuite-Triggers
+    .dsc header from test metadata: the Depends for this test are
+    to be added to Testsuite-Triggers.  (Just as they are for any other
+    test.)
+
+    The test with the hint-testsuite-triggers restriction should not
+    actually be run.
+
+    The packages listed as Depends for this test are usually indirect
+    dependencies, updates to which are considered to pose a risk of
+    regressions in other tests defined in this package.
+
+    There is currently no way to specify this hint on a per-test
+    basis; but in any case the debian.org machinery is not able to
+    think about triggering individual tests.
+
+superficial
+    The test does not provide significant test coverage, so if it
+    passes, that does not necessarily mean that the package under test
+    is actually functional. If a ``superficial`` test fails, it will be
+    treated like any other failing test, but if it succeeds, this is
+    only a weak indication of success. Continuous integration systems
+    should treat a package where all non-superficial tests are skipped as
+    equivalent to a package where all tests are skipped.
+
+    For example, a C library might have a superficial test that simply
+    compiles, links and executes a "hello world" program against the
+    library under test but does not attempt to make use of the library's
+    functionality, while a Python or Perl library might have a
+    superficial test that runs ``import foo`` or ``require Foo;`` but
+    does not attempt to use the library beyond that.
+
 Defined features
 ----------------
 
-There are no currently defined Features.
+test-name
+    Set an explicit test name for the log heading and the ``summary`` file
+    for a ``Test-Command:`` inline test. When not given, these are
+    enumerated like ``command1``.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20181210/1b28c584/attachment.sig>


More information about the ubuntu-devel mailing list