[Bug 1895505] [NEW] autopkgtests failing in groovy

[Christian Ehrhardt ]

Public bug reported:

The three most recent tests all failed, but without a clear reason to do
so by package changes.


Two of the cases are actually guest kernels getting stuck on boot:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/amd64/o/open-iscsi/20200911_002132_17691@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/amd64/o/open-iscsi/20200910_111212_d54fc@/log.gz

[ 3461.006125] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u2:0:7]
[ 3489.006134] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u2:0:7]
[ 3517.006214] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u2:0:7]
[ 3545.006160] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u2:0:7]
[ 3573.006151] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u2:0:7]
qemu-system-x86_64: terminating on signal 15 from pid 10120 (timeout)
xkvm cleanup killing kvm pid '10146'
xkvm returned 124 in 3600s


One other case doesn't even get that far, it fails on modifying the qemu image:
CalledProcessError: Command '['patch-image', '/tmp/autopkgtest.z8gwmE/build.K76/src/debian/tests/groovy.d/disk.img', '--kernel=/tmp/autopkgtest.z8gwmE/build.K76/src/debian/tests/groovy.d/kernel', '--initrd=/tmp/autopkgtest.z8gwmE/build.K76/src/debian/tests/groovy.d/initrd']' returned non-zero exit status 1

We might consider the upper two flaky until we know more, but the latter
just happened with the new qemu which is suspicious. Yet that new qemu
had no changes to qemu-img (except being a new build with a potentially
new toolchain).


Yet OTOH the test runs on all architectures and they all worked:
https://autopkgtest.ubuntu.com/packages/o/open-iscsi/groovy/s390x
https://autopkgtest.ubuntu.com/packages/o/open-iscsi/groovy/arm64
https://autopkgtest.ubuntu.com/packages/o/open-iscsi/groovy/ppc64el

Running the same repro locally in autopkgtest with & without the new qemu works just fine even through multiple re-runs. So it might (tm) resolve on autopkgtest infra as well, not sure.
I'm triggering retries now and we will see later on.

** Affects: open-iscsi (Ubuntu)
     Importance: Undecided
         Status: Invalid


** Tags: update-excuse

** Changed in: open-iscsi (Ubuntu)
       Status: New => In Progress

** Tags added: update-excuse

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to open-iscsi in Ubuntu.
https://bugs.launchpad.net/bugs/1895505

Title:
  autopkgtests failing in groovy

Status in open-iscsi package in Ubuntu:
  Invalid

Bug description:
  The three most recent tests all failed, but without a clear reason to
  do so by package changes.

  
  Two of the cases are actually guest kernels getting stuck on boot:
  https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/amd64/o/open-iscsi/20200911_002132_17691@/log.gz
  https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/amd64/o/open-iscsi/20200910_111212_d54fc@/log.gz

  [ 3461.006125] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u2:0:7]
  [ 3489.006134] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u2:0:7]
  [ 3517.006214] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u2:0:7]
  [ 3545.006160] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u2:0:7]
  [ 3573.006151] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u2:0:7]
  qemu-system-x86_64: terminating on signal 15 from pid 10120 (timeout)
  xkvm cleanup killing kvm pid '10146'
  xkvm returned 124 in 3600s

  
  One other case doesn't even get that far, it fails on modifying the qemu image:
  CalledProcessError: Command '['patch-image', '/tmp/autopkgtest.z8gwmE/build.K76/src/debian/tests/groovy.d/disk.img', '--kernel=/tmp/autopkgtest.z8gwmE/build.K76/src/debian/tests/groovy.d/kernel', '--initrd=/tmp/autopkgtest.z8gwmE/build.K76/src/debian/tests/groovy.d/initrd']' returned non-zero exit status 1

  We might consider the upper two flaky until we know more, but the
  latter just happened with the new qemu which is suspicious. Yet that
  new qemu had no changes to qemu-img (except being a new build with a
  potentially new toolchain).

  
  Yet OTOH the test runs on all architectures and they all worked:
  https://autopkgtest.ubuntu.com/packages/o/open-iscsi/groovy/s390x
  https://autopkgtest.ubuntu.com/packages/o/open-iscsi/groovy/arm64
  https://autopkgtest.ubuntu.com/packages/o/open-iscsi/groovy/ppc64el

  Running the same repro locally in autopkgtest with & without the new qemu works just fine even through multiple re-runs. So it might (tm) resolve on autopkgtest infra as well, not sure.
  I'm triggering retries now and we will see later on.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1895505/+subscriptions