[Bug 1630578] Re: broken kernel causes eternal test retry loop

[Martin Pitt]

Thanks! I pushed your patch to the autopkgtest git. Not sure if there is
still an infra change to do (for the "Auto Package Testing" task).

** Also affects: autopkgtest (Ubuntu)
   Importance: Undecided
       Status: New

** Changed in: autopkgtest (Ubuntu)
       Status: New => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to autopkgtest in Ubuntu.
https://bugs.launchpad.net/bugs/1630578

Title:
  broken kernel causes eternal test retry loop

Status in Auto Package Testing:
  Triaged
Status in autopkgtest package in Ubuntu:
  Fix Committed

Bug description:
  The infra is currently testing linux-meta/4.4.0.41.43 which has some
  bug that it freezes the testbed (nothing actually runs any more and
  ssh hangs). The actual test timeout is being spotted correctly, but
  then it takes another two hours until it finally trips over some
  cleanup error/timeout -- this then gets counted as a tmpfail and the
  whole tests runs again:

  ⟫ tail -f /tmp/autopkgtest-work.s6p8ukov/out/log
  06:30:45 DEBUG| [stdout] shm-sysv PASSED
  06:30:55 DEBUG| [stdout] sigfd PASSED
  06:31:05 DEBUG| [stdout] sigfpe PASSED
  06:31:15 DEBUG| [stdout] sigpending PASSED
  06:31:25 DEBUG| [stdout] sigq PASSED
  06:31:35 DEBUG| [stdout] sigsegv PASSED
  06:31:45 DEBUG| [stdout] sigsuspend PASSED
  06:31:55 DEBUG| [stdout] sleep PASSED
  06:32:05 DEBUG| [stdout] sock PASSED
  06:32:16 DEBUG| [stdout] sockfd PASSED

  
  autopkgtest [10:43:18]: ERROR: timed out on command "su -s /bin/bash ubuntu -c set -e; export USER=`id -nu`; . /etc/profile >/dev/null 2>&1 || true;  . ~/.profile >/dev/null 2>&1 || true; buildtree="/tmp/autopkgtest.3eTi1L/build.zNX/linux-4.4.0"; mkdir -p -m 1777 -- "/tmp/autopkgtest.3eTi1L/ubuntu-regression-suite-artifacts"; export AUTOPKGTEST_ARTIFACTS="/tmp/autopkgtest.3eTi1L/ubuntu-regression-suite-artifacts"; export ADT_ARTIFACTS="$AUTOPKGTEST_ARTIFACTS"; mkdir -p -m 755 "/tmp/autopkgtest.3eTi1L/autopkgtest_tmp"; export AUTOPKGTEST_TMP="/tmp/autopkgtest.3eTi1L/autopkgtest_tmp"; export ADTTMP="$AUTOPKGTEST_TMP"; export DEBIAN_FRONTEND=noninteractive; export LANG=C.UTF-8; export DEB_BUILD_OPTIONS=parallel=4; unset LANGUAGE LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE   LC_MONETARY LC_MESSAGES LC_PAPER LC_NAME LC_ADDRESS   LC_TELEPHONE LC_MEASUREMENT LC_IDENTIFICATION LC_ALL;rm -f /tmp/autopkgtest_script_pid; set -C; echo $$ > /tmp/autopkgtest_script_pid; set +C; trap "rm -f /tmp/autopkgtest_script_pid" EXIT INT QUIT PIPE; cd "$buildtree"; export 'ADT_TEST_TRIGGERS=linux-meta/4.4.0.41.43'; chmod +x /tmp/autopkgtest.3eTi1L/build.zNX/linux-4.4.0/debian/tests/ubuntu-regression-suite; touch /tmp/autopkgtest.3eTi1L/ubuntu-regression-suite-stdout /tmp/autopkgtest.3eTi1L/ubuntu-regression-suite-stderr; /tmp/autopkgtest.3eTi1L/build.zNX/linux-4.4.0/debian/tests/ubuntu-regression-suite 2> >(tee -a /tmp/autopkgtest.3eTi1L/ubuntu-regression-suite-stderr >&2) > >(tee -a /tmp/autopkgtest.3eTi1L/ubuntu-regression-suite-stdout); " (kind: test)
  autopkgtest [10:43:18]: test ubuntu-regression-suite: -----------------------]
  ^[Unexpected cleanup error:
  Traceback (most recent call last):
    File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 708, in mainloop
      command()
    File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 646, in command
      r = f(c, ce)
    File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 584, in cmd_copyup
      copyupdown(c, ce, True)
    File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 469, in copyupdown
      copyupdown_internal(ce[0], c[1:], upp)
    File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 570, in copyupdown_internal
      (wh, ['source', 'destination'][sdn], status))
    File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 84, in bomb
      raise Quit(12, progname + ": failure: %s" % m)
  VirtSubproc.Quit: (12, '<VirtSubproc>: failure: copyup source failed, status 255')

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 680, in error_cleanup
      cleanup()
    File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 671, in cleanup
      caller.hook_cleanup()
    File "/home/ubuntu/autopkgtest/virt/autopkgtest-virt-ssh", line 464, in hook_cleanup
      VirtSubproc.downtmp_remove()
    File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 291, in downtmp_remove
      auxverb + ['rm', '-rf', '--', downtmp])
    File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 144, in execute_timeout
      (out, err) = sp.communicate(instr)
    File "/usr/lib/python3.5/subprocess.py", line 1064, in communicate
      self.wait()
    File "/usr/lib/python3.5/subprocess.py", line 1658, in wait
      (pid, sts) = self._try_wait(0)
    File "/usr/lib/python3.5/subprocess.py", line 1608, in _try_wait
      (pid, sts) = os.waitpid(self.pid, wait_flags)
    File "/home/ubuntu/autopkgtest/lib/VirtSubproc.py", line 64, in alarm_handler
      raise Timeout()
  VirtSubproc.Timeout

  while cleaning up because of another error:
  <VirtSubproc>: failure: copyup source failed, status 255
  autopkgtest [12:43:19]: ERROR: testbed failure: unexpected eof from the testbed

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/1630578/+subscriptions