autopkgtest missed regression for kinetic

[Steve Langasek]

On Sun, May 15, 2022 at 05:46:49PM -0400, Jeremy Bicha wrote:
> On Sun, May 15, 2022 at 4:26 PM Steve Langasek
> <steve.langasek at ubuntu.com> wrote:
> > > The test has never passed; it has had neutral results and failing results,
> > > but by policy, proposed-migration does not treat neutral->fail as a
> > > migration-blocking regression, only pass->fail.  Neutral test results are
> > > informational only.

> > Having said this, I am just this moment looking at cases where britney is
> > treating neutral->fail as a regression.  I still believe the intended policy
> > is as I described, but one way or another there appear to be some bugs here
> > in the implementation ;)

> devhelp's autopkgtest is marked as "superficial". In Debian, a passing
> superficial autopkgtest won't speed up migration to Testing. But if
> the test fails, it indicates that there is a serious bug and migration
> to Testing is blocked. Please see the description of the intent of
> "superficial":

> https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst

You're right, sorry for misremembering.

So the other consideration is that proposed-migration, by necessity, has to
cache a lot of information about the autopkgtest results.  Once it has
decided that a test is not a regression based on available results, it has
no reason to revisit this.

So in
https://people.canonical.com/~ubuntu-archive/proposed-migration/log/kinetic/2022-05-10/22:14:21.log.gz
we see:

I: [2022-05-10T22:43:48+0000] - Fetched test result for devhelp/41.2-2/amd64 20220510_212456_139c1@ (triggers: ['webkit2gtk/2.36.1-1']): fail
I: [2022-05-10T22:43:48+0000] - -> matches pending request devhelp/amd64 for trigger webkit2gtk/2.36.1-1
I: [2022-05-10T22:43:48+0000] - Checking for new results for devhelp/amd64 for trigger migration-reference/0
I: [2022-05-10T22:43:48+0000] - Requesting devhelp autopkgtest on amd64 to verify migration-reference/0
I: [2022-05-10T22:43:48+0000] - Updating pending requested tests in data/kinetic/state/autopkgtest-pending.json

In the corresponding
https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses/kinetic/2022-05-10/22:14:21.yaml.xz
this shows:

      devhelp/41.2-2:
        amd64:
        - RUNNING-REFERENCE
        - https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/amd64/d/devhelp/20220510_212456_139c1@/log.gz
        - https://autopkgtest.ubuntu.com/packages/d/devhelp/kinetic/amd64
        - null
        - https://autopkgtest.ubuntu.com/request.cgi?release=kinetic&arch=amd64&package=devhelp&trigger=webkit2gtk%2F2.36.1-1

i.e. the test failed but we're checking if the baseline failed.

In the next run, https://people.canonical.com/~ubuntu-archive/proposed-migration/log/kinetic/2022-05-10/23:59:06.log.gz
we have:

I: [2022-05-11T00:10:00+0000] - Checking for new results for devhelp/amd64 for trigger migration-reference/0
I: [2022-05-11T00:10:00+0000] - Fetched test result for devhelp/41.2-2/amd64 20220510_230253_250a6@ (triggers: ['migration-reference/0']): neutral
I: [2022-05-11T00:10:00+0000] - -> matches pending request devhelp/amd64 for trigger migration-reference/0

and:

      devhelp/41.2-2:
        amd64:
        - ALWAYSFAIL
        - https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/amd64/d/devhelp/20220510_212456_139c1@/log.gz
        - https://autopkgtest.ubuntu.com/packages/d/devhelp/kinetic/amd64
        - null
        - null

I'm not sure why this is.  It is similar to other issues we've had in the
past where the order in which results are seen cause britney to see failures
as regressions when they're actually progressions, but the underlying cause
here may be different.

That seems pretty obviously a bug; we shouldn't request a baseline retest,
query for the result of that test, and then ignore the result in calculating
whether we've regressed.

I've opened https://bugs.launchpad.net/britney/+bug/1973864 for this issue.

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                   https://www.debian.org/
slangasek at ubuntu.com                                     vorlon at debian.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20220518/570a92c9/attachment.sig>