[Bug 1915678] Re: [SRU] iSCSI+Multipath: Volume attachment hungs if sessiong scanning fails

[Brett Milford]

** Description changed:

  [Impact]
  
-  * If some commands like "iscsiadm -m session" fail, the thread can abort immediately without updating any counters like failed_logins or stopped_threads properly, because there are no try-except block to catch exceptions.
-  * The main thread keeps waiting until these counters are updated, and this results in stuck of volume attachment process.
+  * If some commands like "iscsiadm -m session" fail, the thread can abort immediately without updating any counters like failed_logins or stopped_threads properly, because there are no try-except block to catch exceptions.
+  * The main thread keeps waiting until these counters are updated, and this results in stuck of volume attachment process.
  
  [Test Case]
  
-  * Deploy Cinder with an iSCSI driver and multipath enabled.
- 
-  * TBC.
+  * Deploy Cinder with a backend that uses an iSCSI driver and configure Multipath
+  * Attach a volume to an instance (first attachment for a period 
+  * See log line like:
+ 2021-12-01 00:23:24.044 2679 WARNING os_brick.initiator.connectors.iscsi [...] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.
+  * Volume attachment never completes
+  * Passing test: Log line appears but volume attachment succeeds.
  
  [Where problems could occur]
  
-  * Change primarily introduces error handling and doesn't change implementation details.
-    As such we may see an error condition logged.
+  * Change primarily introduces error handling and doesn't change implementation details.
+    As such we may see an error condition logged.
  
  [Other Info]
  
-  * -
+  * -
  
  --- Original Description ---
  
  Currently we execute login to iscsi portals and device discovery in
  multiple threads concurrently when multipath is enabled.
  
  However if some commands like "iscsiadm -m session" fail, the thread can abort immediately without updating any counters like failed_logins or stopped_threads properly, because there are no try-except block to catch  exceptions.
  However the main thread keeps waiting until these counters are updated, and this results in stuck of volume attachment process.
  
  This issue was initially reported in downstream bug https://bugzilla.redhat.com/show_bug.cgi?id=1923975 , and maybe is caused by a bug in iscsiadm command.
  However we should handle the error more properly because current behavior requires operators to restart services like cinder-volume to resolve the stuck.

** Description changed:

  [Impact]
  
   * If some commands like "iscsiadm -m session" fail, the thread can abort immediately without updating any counters like failed_logins or stopped_threads properly, because there are no try-except block to catch exceptions.
   * The main thread keeps waiting until these counters are updated, and this results in stuck of volume attachment process.
  
  [Test Case]
  
   * Deploy Cinder with a backend that uses an iSCSI driver and configure Multipath
-  * Attach a volume to an instance (first attachment for a period 
-  * See log line like:
+  * Attach a volume to an instance (first attachment for a period
+  * See log line like:
  2021-12-01 00:23:24.044 2679 WARNING os_brick.initiator.connectors.iscsi [...] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.
-  * Volume attachment never completes
-  * Passing test: Log line appears but volume attachment succeeds.
+  * Volume attachment never completes
+  * Passing test: Log line appears but volume attachment succeeds.
  
  [Where problems could occur]
  
   * Change primarily introduces error handling and doesn't change implementation details.
     As such we may see an error condition logged.
- 
- [Other Info]
- 
-  * -
  
  --- Original Description ---
  
  Currently we execute login to iscsi portals and device discovery in
  multiple threads concurrently when multipath is enabled.
  
  However if some commands like "iscsiadm -m session" fail, the thread can abort immediately without updating any counters like failed_logins or stopped_threads properly, because there are no try-except block to catch  exceptions.
  However the main thread keeps waiting until these counters are updated, and this results in stuck of volume attachment process.
  
  This issue was initially reported in downstream bug https://bugzilla.redhat.com/show_bug.cgi?id=1923975 , and maybe is caused by a bug in iscsiadm command.
  However we should handle the error more properly because current behavior requires operators to restart services like cinder-volume to resolve the stuck.

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1915678

Title:
  [SRU] iSCSI+Multipath: Volume attachment hungs if sessiong scanning
  fails

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive stein series:
  In Progress
Status in Ubuntu Cloud Archive train series:
  In Progress
Status in Ubuntu Cloud Archive ussuri series:
  In Progress
Status in Ubuntu Cloud Archive victoria series:
  In Progress
Status in os-brick:
  Fix Released
Status in python-os-brick package in Ubuntu:
  Fix Released
Status in python-os-brick source package in Bionic:
  In Progress
Status in python-os-brick source package in Focal:
  In Progress
Status in python-os-brick source package in Hirsute:
  Fix Released
Status in python-os-brick source package in Impish:
  Fix Released
Status in python-os-brick source package in Jammy:
  Fix Released

Bug description:
  [Impact]

   * If some commands like "iscsiadm -m session" fail, the thread can abort immediately without updating any counters like failed_logins or stopped_threads properly, because there are no try-except block to catch exceptions.
   * The main thread keeps waiting until these counters are updated, and this results in stuck of volume attachment process.

  [Test Case]

   * Deploy Cinder with a backend that uses an iSCSI driver and configure Multipath
   * Attach a volume to an instance (first attachment for a period
   * See log line like:
  2021-12-01 00:23:24.044 2679 WARNING os_brick.initiator.connectors.iscsi [...] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.
   * Volume attachment never completes
   * Passing test: Log line appears but volume attachment succeeds.

  [Where problems could occur]

   * Change primarily introduces error handling and doesn't change implementation details.
     As such we may see an error condition logged.

  --- Original Description ---

  Currently we execute login to iscsi portals and device discovery in
  multiple threads concurrently when multipath is enabled.

  However if some commands like "iscsiadm -m session" fail, the thread can abort immediately without updating any counters like failed_logins or stopped_threads properly, because there are no try-except block to catch  exceptions.
  However the main thread keeps waiting until these counters are updated, and this results in stuck of volume attachment process.

  This issue was initially reported in downstream bug https://bugzilla.redhat.com/show_bug.cgi?id=1923975 , and maybe is caused by a bug in iscsiadm command.
  However we should handle the error more properly because current behavior requires operators to restart services like cinder-volume to resolve the stuck.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1915678/+subscriptions