[Bug 1892358] Re: autopkgtest success rate dropped inhibiting proposed migration

[Mathew Hodson]

** No longer affects: build-essential (Ubuntu)

** No longer affects: build-essential (Ubuntu Focal)

** No longer affects: glib2.0 (Ubuntu)

** No longer affects: iputils (Ubuntu)

** No longer affects: kbd (Ubuntu)

** No longer affects: linux-meta (Ubuntu)

** No longer affects: linux-meta (Ubuntu Bionic)

** No longer affects: linux-meta (Ubuntu Focal)

** No longer affects: ntpsec (Ubuntu)

** No longer affects: qemu (Ubuntu)

** No longer affects: qemu (Ubuntu Focal)

** No longer affects: util-linux (Ubuntu)

** No longer affects: util-linux (Ubuntu Focal)

** Changed in: systemd (Ubuntu)
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to build-essential in Ubuntu.
https://bugs.launchpad.net/bugs/1892358

Title:
  autopkgtest success rate dropped inhibiting proposed migration

Status in systemd package in Ubuntu:
  Fix Released
Status in systemd source package in Bionic:
  Fix Released
Status in systemd source package in Focal:
  Fix Released

Bug description:
  [impact]

  autopkgtests are failing/flaky and prevent other packages from
  migrating to -updates

  [test case]

  check autopkgtest history

  [regression potential]

  in regard to the changed test cases, any regression would likely
  result in either an incorrectly passed test, or an incorrectly failed
  test.

  [scope]

  for systemd, this is needed for x, b, and f.

  tests in g appear to be mostly stable, but I've opened MR (linked from
  this bug) to update the tests there as well.

  i don't plan to update x, as it's reaching ESM in ~6 months, and
  backporting the test fixes is more work than just a simple code copy,
  since there are additional differences/changes needed in the older
  version of systemd (and python3). the failing/flaky tests in x have
  been like that forever, and people have just retried them; we can keep
  retrying them until x moves into ESM next year.

  [original description]

  Hi,
  we had such cases in the past like bug 1817721 for bionic and maybe bug 1892130 is about the same as well. There were more but I didn't want to search for all of them - what I checked is that there are no open ones clearly pointing out the recent further drop in already flaky subtests.

  In particular the tests "tests-in-lxd" and "systemd-fsckd" were known
  to be flaky before, but got even worse.

  Here stats of the last 40 runs, it might be a coincidences that this
  is after 246-2ubuntu1 landed. Could as well be any other change

  groovy
    amd64
  tests-in-lxd                   (F 42% S  0% B 10% => P 45%/) FFFFBFFFFFFFB....FF.B.....F.....F...FBF
  build-login                    (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  unit-config                    (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  networkd-testpy                (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  boot-and-services              (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  boot-smoke                     (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  logind                         (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  storage                        (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  upstream                       (F 35% S  0% B 10% => P 52%/) ..FFB.FFF.FFB....FF.B.....F.F..F....FBF
  udev                           (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
  systemd-fsckd                  (F 37% S  0% B 10% => P 50%/) FFFFBFFFFFFFB.FF...FB.....F..........B.
  root-unittests                 (F  0% S  0% B 10% => P 87%/) ....B.......B.......B................B.
    ppc64el
  tests-in-lxd                   (F 25% S  0% B  0% => P 75%/) FFFF....FF............FFF.....F.........
  systemd-fsckd                  (F 35% S  0% B  0% => P 65%/) FFFFFFF...FF........F....FF.F..F........
  root-unittests                 (F  2% S  0% B  0% => P 97%/) ..............................F.........
    s390x
  tests-in-lxd                   (F 52% S  0% B  0% => P 47%/) FFFFFFF.FFFFFFF.FF.........FFFF...F.....
  timedated                      (F  2% S  0% B  0% => P 97%/) ...........F............................
  upstream                       (F 17% S  0% B  0% => P 82%/) .....F......F.F.............FFF...F.....
  systemd-fsckd                  (F 32% S  0% B  0% => P 67%/) FFFFFFF..FF..F.................FF..F....
  root-unittests                 (F 10% S  0% B  0% => P 90%/) ............................FFF...F.....
    arm64
  tests-in-lxd                   (F 40% S  0% B  2% => P 57%/) FFFFF.B...FFF.FF..F..F.........FFF.F....
  logind                         (F  2% S  0% B  2% => P 95%/) ......B...................F.............
  upstream                       (F 22% S  0% B  2% => P 75%/) ...F.FB.....F.F.............F..FFF.F....
  root-unittests                 (F 12% S  0% B  2% => P 85%/) ......B.F...........F.F........F...F....

  (I'm sure LP will make this unreadable, but is is nice in monospace)

  Whatever the root cause is - the success rate of these has reduced so
  much that the (even formerly questionable) practice of retry-until-
  success won't work anymore.

  I have run the two tests in a local VM and systemd-fsckd works there
  while tests-in-lxd seems to trip over the old flaky fellow being
  "boot-and-services".

  We had the discussion in the past, but I think I need to again bring
  up the suggestion to skip "tests-in-lxd" and "systemd-fsckd" until
  they are on reasonable success rates.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1892358/+subscriptions