[Bug 1806347] [NEW] mask current unreliable tests in bionic and cosmic

[Christian Ehrhardt ]

Public bug reported:

Hi,
a recent SRU made me check the success rate of systemd autopkgtests.
It looks grim atm:

None of the fails look related to the actual uploads BTW

Bionic:
last 7 ppc64 - http://autopkgtest.ubuntu.com/packages/s/systemd/bionic/ppc64el
last 7 s390x - http://autopkgtest.ubuntu.com/packages/s/systemd/bionic/s390x
At least those with "last 7" of them is exactly since systemd/237-3ubuntu10.10 and only the "upstream" test is failing.

last 26 amd64 - http://autopkgtest.ubuntu.com/packages/s/systemd/bionic/amd64
last 12 i386 - http://autopkgtest.ubuntu.com/packages/s/systemd/bionic/i386
If anything those fail since the last kernel update, but that isn't sure.

last 80 armhf - http://autopkgtest.ubuntu.com/packages/s/systemd/bionic/armhf
Here always "root-unittests" fails, maybe not container compatible?

Cosmic:
last ~40 ppc64 - http://autopkgtest.ubuntu.com/packages/s/systemd/cosmic/ppc64el
In addition to the "upstream" fails seen in Bionic there are some gdm induced errors which we just fixed in https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3006fedda1d1ca3f04c5f593e8018bb6d1196025

last 4 s390x - http://autopkgtest.ubuntu.com/packages/s/systemd/cosmic/s390x
last 16 on amd64 - http://autopkgtest.ubuntu.com/packages/s/systemd/cosmic/amd64
last 8 arm64 - http://autopkgtest.ubuntu.com/packages/s/systemd/cosmic/arm64
last 4 i386 - http://autopkgtest.ubuntu.com/packages/s/systemd/cosmic/i386
The tests that fail most of the time are "boot-smoke" and upstream"


In general the test "upstream" fails most (amd64/s390x/ppc64).
Maybe that one should be (partially) disabled on the next upload as the rest is at least semi reliable (doable with retries).

The same applies for test "root-unittests" on armhf, please just skip
that one and retain the rest.

So maybe on the next systemd update those could be addressed or at least
improved, but for now this is fake-blocking way too many SRUs - to lets
add an all arch badtest for now.

TL;DR on changes that I'd request for next systemd upload:
- Bionic skip "upstream" test (unless you have the magic dust to make it good again)
- Bionic skip root-unittest on armhf
- Cosmic skip "upstream" test
- Cosmic also implement gdm3 fix we just had in Disco
- Cosmic skip boot-smoke test on arm64 & amd64


I'm leaving this bug open to consider it on the next systemd upload as suggested.
For now I see no other way than to propose an MP to force-badtest.

** Affects: systemd (Ubuntu)
     Importance: Undecided
         Status: Fix Released

** Affects: systemd (Ubuntu Bionic)
     Importance: Undecided
         Status: New

** Affects: systemd (Ubuntu Cosmic)
     Importance: Undecided
         Status: New

** Also affects: systemd (Ubuntu Bionic)
   Importance: Undecided
       Status: New

** Also affects: systemd (Ubuntu Cosmic)
   Importance: Undecided
       Status: New

** Changed in: systemd (Ubuntu)
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1806347

Title:
  mask current unreliable tests in bionic and cosmic

Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Bionic:
  New
Status in systemd source package in Cosmic:
  New

Bug description:
  Hi,
  a recent SRU made me check the success rate of systemd autopkgtests.
  It looks grim atm:

  None of the fails look related to the actual uploads BTW

  Bionic:
  last 7 ppc64 - http://autopkgtest.ubuntu.com/packages/s/systemd/bionic/ppc64el
  last 7 s390x - http://autopkgtest.ubuntu.com/packages/s/systemd/bionic/s390x
  At least those with "last 7" of them is exactly since systemd/237-3ubuntu10.10 and only the "upstream" test is failing.

  last 26 amd64 - http://autopkgtest.ubuntu.com/packages/s/systemd/bionic/amd64
  last 12 i386 - http://autopkgtest.ubuntu.com/packages/s/systemd/bionic/i386
  If anything those fail since the last kernel update, but that isn't sure.

  last 80 armhf - http://autopkgtest.ubuntu.com/packages/s/systemd/bionic/armhf
  Here always "root-unittests" fails, maybe not container compatible?

  Cosmic:
  last ~40 ppc64 - http://autopkgtest.ubuntu.com/packages/s/systemd/cosmic/ppc64el
  In addition to the "upstream" fails seen in Bionic there are some gdm induced errors which we just fixed in https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3006fedda1d1ca3f04c5f593e8018bb6d1196025

  last 4 s390x - http://autopkgtest.ubuntu.com/packages/s/systemd/cosmic/s390x
  last 16 on amd64 - http://autopkgtest.ubuntu.com/packages/s/systemd/cosmic/amd64
  last 8 arm64 - http://autopkgtest.ubuntu.com/packages/s/systemd/cosmic/arm64
  last 4 i386 - http://autopkgtest.ubuntu.com/packages/s/systemd/cosmic/i386
  The tests that fail most of the time are "boot-smoke" and upstream"

  
  In general the test "upstream" fails most (amd64/s390x/ppc64).
  Maybe that one should be (partially) disabled on the next upload as the rest is at least semi reliable (doable with retries).

  The same applies for test "root-unittests" on armhf, please just skip
  that one and retain the rest.

  So maybe on the next systemd update those could be addressed or at
  least improved, but for now this is fake-blocking way too many SRUs -
  to lets add an all arch badtest for now.

  TL;DR on changes that I'd request for next systemd upload:
  - Bionic skip "upstream" test (unless you have the magic dust to make it good again)
  - Bionic skip root-unittest on armhf
  - Cosmic skip "upstream" test
  - Cosmic also implement gdm3 fix we just had in Disco
  - Cosmic skip boot-smoke test on arm64 & amd64

  
  I'm leaving this bug open to consider it on the next systemd upload as suggested.
  For now I see no other way than to propose an MP to force-badtest.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1806347/+subscriptions