[Bug 1915126] Re: autopkgtest times out (or fails before that) in hirsute

Tue Feb 9 13:53:54 UTC 2021

>From the log:
402209 Checking in progress on 1 disk (92.7% complete)fsckd:1:92.8:Checking in progress on 1 disk (92.8% complete)
402210 Checking in progress on 1 disk (92.8% complete)fsckd:1:92.9:Checking in progress on 1 disk (92.9% complete)
402211 Checking in progress on 1 disk (92.9% complete)[[0;1;31mFAILED[0m] Failed to start [0;1;39mFile Systeï¿½ï¿½ï¿½ck on /dev/disk/by-label/UEFI[0m.
402212 See 'systemctl status "systemd-fsck at devï¿½ï¿½ï¿½y\\x2dlabel-UEFI.service"' for details.
402213 [[0;1;38;5;185mDEPEND[0m] Dependency failed for [0;1;39m/boot/efi[0m.            
402214 [[0;1;38;5;185mDEPEND[0m] Dependency failed for [0;1;39mLocal File Systems[0m.   
402215 [[0;32m  OK  [0m] Stopped [0;1;39mForward Password Rï¿½ï¿½ï¿½uests to Wall Directory Watch[0m.

But on a try to locally recreate via
  $ sudo ~/work/autopkgtest/autopkgtest/runner/autopkgtest --no-built-binaries \
    --apt-upgrade --testname=systemd-fsckd --shell systemd_247.1-4ubuntu1.dsc \
    -- qemu --ram-size=1536 --cpus 1 ~/work/autopkgtest-hirsute-amd64.img 
It works just fine, so this is one of the lovely "only fails on infrastructure" cases

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1915126

Title:
  autopkgtest times out (or fails before that) in hirsute

Status in systemd package in Ubuntu:
  New

Bug description:
  Hi,
  I've asked yesterday on IRC but so far got no answer. I think it is right to file a bug about the current state of systemd autopkgtest to unite the efforts in regard to it.

  I was looking at the systemd tests for a no-change rebuild that really
  had no reason to now make it fail. While checking I found that as of
  this month (first bad test on 1st of February) most of the systemd
  test runs on amd64 will not pass/fail but instead in most cases time
  out.

  I've seen various people retry the case as it shows the typical
  symptoms of a "not the fault of the package, let us retry this" case.
  But it seems that won't help as the test history is rather clear.

  * Watch this in monospace to make more sense of it *

  $ check-autopkgtest-stats.sh -c 50 -p systemd -r "hirsute" -a "amd64"
  Check last 50 test results for src:systemd on releases 'hirsute' on architectures 'amd64'
  Of the 50 last tests, we had these subtest failing per release/arch:

  hirsute
    amd64
      tests-in-lxd                   (F  2% f  0% S  0% B 12% => P 52%/) .BTTTBTT.BTTTTTBTTTBBTTT..F......................
      hostnamed                      (F  0% f  0% S  0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT.........................
      build-login                    (F  0% f  0% S  0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT.........................
      unit-config                    (F  0% f  0% S  0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT.........................
      networkd-testpy                (F  0% f  0% S  0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT.........................
      localed-locale                 (F  0% f  0% S  0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT.........................
      boot-and-services              (F  4% f  0% S  0% B 12% => P 50%/) .BTTTBTT.BTTTTTBTTTBBTTT...............F.....F...
      timedated                      (F  0% f  0% S  0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT.........................
      boot-smoke                     (F  0% f  0% S  0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT.........................
      logind                         (F  0% f  0% S  0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT.........................
      storage                        (F  0% f  0% S  0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT.........................
      upstream                       (F 12% f  0% S  0% B 12% => P 42%/) .BTTTBTT.BTTTTTBTTTBBTTTF.....F.F..F........F..F.
      udev                           (F  0% f  0% S  0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT.........................
      systemd-fsckd                  (F  8% f  0% S  0% B 12% => P 46%/) FBTTTBTTFBTTTTTBTTTBBTTT.............F......F....
      root-unittests                 (F  0% f  0% S  0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT.........................
      localed-x11-keymap             (F  0% f  0% S  0% B 12% => P 54%/) .BTTTBTT.BTTTTTBTTTBBTTT.........................

  We see that formerly we had the known to be somewhat flaky, but
  otherwise working test (time goes from right to left). So we had the
  usual suspects of tests like upstream/systemd-fsck that failed a few
  times, but also working runs in between.

  But since February 22/24 runs failed very bad.
  6 (=B) of those cases are aborting mid execution
    autopkgtest [22:08:58]: ERROR: testbed failure: testbed auxverb failed with exit code 255
  And 16 of them timed out
    <VirtSubproc>: failure: Timed out on waiting for ssh connection

  Sadly the 2/24 that didn't hard fail by that where broken by the known
  flaky systemd-fsckd test.

  Something drives this test crazy that we have to find and resolve, at
  the current rate nothing depending on it is likely to migrate.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1915126/+subscriptions