[Bug 1807978] Re: vmtest: iscsi/LVM tests failing on cosmic/disco

Wed Dec 19 16:25:45 UTC 2018

With respect to the resource-agents autopkgtest failure in bionic,
I had commented on that in a previous SRU:
 https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1802354/comments/18

copied here for eaiser view:

It seems this test fails transiently. It was quite reliable from
 2017-11-27 to 2018-08-09
with only a single failure in 44 runs.
But 2018-08-22 and after there have been only 3 successful runs in 18.

The point of this is to show that there may well have been a regression
that caused resource-agents to fail, but it is not likely this change.

** Description changed:

  === Begin SRU Template ===
  [Impact]
  This bug results in lvm devices on top of iscsi block devices not being
  available to the system.  The journal will show messages like:
-   [ 17.666655] systemd[1]: dev-disk-by\x2duuid-a98f706b\x2db064\x2d4682\x2d8eb2\x2d6c2c1284060c.device: Dependency Before=network-online.target ignored (.device units cannot be delayed)
+   [ 17.666655] systemd[1]: dev-disk-by\x2duuid-a98f706b\x2db064\x2d4682\x2d8eb2\x2d6c2c1284060c.device: Dependency Before=network-online.target ignored (.device units cannot be delayed)

- And ultimately 
-  TIME [0m] Timed out waiting for device dev-di...device.
+ And ultimately
+  TIME [0m] Timed out waiting for device dev-di...device.

  [Test Case]
  This can be tested by running curtin's vmtest test harness.
  A run like this should suffice:

  env CURTIN_VMTEST_ADD_REPOS=proposed \
-     CURTIN_VMTEST_UPGRADE_PACKAGES=open-iscsi \
-     ./tools/jenkins-runner \
-        tests/vmtests/test_iscsi.py  \
-        tests/vmtests/test_lvm_iscsi.py  \
-        tests/vmtests/test_mdadm_iscsi.py \
-        tests/vmtests/test_basic.py
+     CURTIN_VMTEST_UPGRADE_PACKAGES=open-iscsi \
+     ./tools/jenkins-runner \
+        tests/vmtests/test_iscsi.py  \
+        tests/vmtests/test_lvm_iscsi.py  \
+        tests/vmtests/test_mdadm_iscsi.py \
+        tests/vmtests/test_basic.py

  That is all the iscsi based tests and a 'basic' install for good
  measure.

  [Regression Potential]
  Chance of regression is hopefully low, and hopefully lower now than before.
  Regression is almost certainly limited to iscsi devices.

  [Other Info]
  An attempt to fix bug 1802354 added a udev rule for open-iscsi that looked
  like:
-   SUBSYSTEM=="block", ACTION=="add", ENV{ID_PATH}=="*-iscsi-*", ENV{SYSTEMD_WANTS}="iscsid.service"
+   SUBSYSTEM=="block", ACTION=="add", ENV{ID_PATH}=="*-iscsi-*", ENV{SYSTEMD_WANTS}="iscsid.service"

  The problem was ENV{SYSTEMD_WANTS}="iscsid.service".

  The '=' operator would truncate the udev-added "wants".  lvm also utilized
  SYSTEMD_WANTS, and the two would conflict.

  This is the second regression fix to the open-iscsi package as a result
  of the fix for bug 1755858.

  The regression was caught by curtin's test harness (vmtest).  Specifically
  the following tests would fail:

-  tests/vmtests/test_lvm_iscsi.py:BionicTestIscsiLvm
-  tests/vmtests/test_lvm_iscsi.py:CosmicTestIscsiLvm
-  tests/vmtests/test_lvm_iscsi.py:DiscoTestIscsiLvm
+  tests/vmtests/test_lvm_iscsi.py:BionicTestIscsiLvm
+  tests/vmtests/test_lvm_iscsi.py:CosmicTestIscsiLvm
+  tests/vmtests/test_lvm_iscsi.py:DiscoTestIscsiLvm

  Related Bugs:
-  * bug 1755858: iscsid autostarts on all servers when it has nothing to do 
-  * bug 1802354: vmtest: iscsi tests failing on cosmic/disco 
+  * bug 1755858: iscsid autostarts on all servers when it has nothing to do
+  * bug 1802354: iscsid does not run if there are only initramfs initiated targets 
+ 
  === End SRU Template ===

  TestIscsiBasic and other Iscsi tests on Cosmic/Disco are failing in
  vmtest on jenkins.

    - looked some at curtin at iscsi failures cosmic, disco
      - 604 was first fail like this.
             cosmic 20181204
             disco  20181129
        https://jenkins.ubuntu.com/server/job/curtin-vmtest-devel-amd64/604/consoleFull
        602 PASSED and did run disco
             cosmic 20181114
             disco  20181109

      - seems likely tied to open-iscsi upload https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1755858 or https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1802354
    I want to revert iscsi and see if it fixes.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to open-iscsi in Ubuntu.
https://bugs.launchpad.net/bugs/1807978

Title:
  vmtest: iscsi/LVM tests failing on cosmic/disco

Status in curtin package in Ubuntu:
  Invalid
Status in open-iscsi package in Ubuntu:
  Fix Released
Status in curtin source package in Bionic:
  Invalid
Status in open-iscsi source package in Bionic:
  Fix Released
Status in curtin source package in Cosmic:
  Invalid
Status in open-iscsi source package in Cosmic:
  Fix Released

Bug description:
  === Begin SRU Template ===
  [Impact]
  This bug results in lvm devices on top of iscsi block devices not being
  available to the system.  The journal will show messages like:
    [ 17.666655] systemd[1]: dev-disk-by\x2duuid-a98f706b\x2db064\x2d4682\x2d8eb2\x2d6c2c1284060c.device: Dependency Before=network-online.target ignored (.device units cannot be delayed)

  And ultimately
   TIME [0m] Timed out waiting for device dev-di...device.