[Bug 1053306] Re: iscsiadm hangs in recvmsg()

[dann frazier]

** Description changed:

+ [Impact]
+ iscsid can hang or crash
+ 
+ [Test Case]
+ 1) This issue is exacerbated by increased logging output. To configure open-iscsi to emit the maximum amount of messages, edit /etc/init.d/open-iscsi and change the line:
+   start-stop-daemon --start --quiet --exec $DAEMON
+ to:
+   start-stop-daemon --start --quiet --exec $DAEMON -- -d 8
+ 
+ On your storage server, create multiple LUN targets and expose them to
+ your Ubuntu system. I recommend 3 or more.
+ 
+ Edit the attached repro.sh script to set the TARGET variable to point to
+ the correct IP of your storage server. Execute this script on your
+ Ubuntu system. When an error occurs, this script should hang or begin
+ emitting errors.
+ 
+ [Regression Potential] 
+ I would expect regressions here to have the same symptoms of the original bug - e.g. hangs, crashes.
+ 
+ Original report follows.
+ 
  We occasionally see iscsiadm hang indefinitely on a system. It is
  difficult to reproduce, there maybe months between occurrences. This
  happened again today, and we collected the following information.
  
  We captured an strace of a failing iscsiadm discover process, I'll
  attach it here. All iscsiadm processes that are hung are also in
  recvmsg() when we attach strace.
  
  It was observed that there was one iscsid process - where normally there
  is 2. The pids are normally consecutive, so I presume one forks the
  other before daemonizing.
  
  syslog contains the following messages from iscsid:
  
  Sep 20 15:55:57 cn5 iscsid: semop up failed 4
  Sep 20 15:56:05 cn5 iscsid: semop down failed
  Sep 20 15:56:23 cn5 iscsid: semop up failed 4
  Sep 20 16:02:23 cn5 iscsid: semop up failed 4
  Sep 20 16:02:26 cn5 iscsid: semop up failed 4
  Sep 20 16:02:26 cn5 iscsid: semop up failed 4
  Sep 20 16:02:26 cn5 iscsid: semop up failed 4
  Sep 20 16:02:32 cn5 iscsid: semop up failed 4
  Sep 20 18:01:49 cn5 iscsid: semop up failed 4
  Sep 20 18:03:56 cn5 iscsid: semop up failed 4
  Sep 20 18:05:03 cn5 iscsid: semop up failed 4
  
  This sounds similar to the issue described in
  https://bugzilla.redhat.com/show_bug.cgi?id=676804, but note the
  different errno.

** Attachment added: "sample reproduction script"
   https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1053306/+attachment/3336393/+files/repro.sh

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to open-iscsi in Ubuntu.
https://bugs.launchpad.net/bugs/1053306

Title:
  iscsiadm hangs in recvmsg()

Status in “open-iscsi” package in Ubuntu:
  New
Status in “open-iscsi” package in Fedora:
  Unknown

Bug description:
  [Impact]
  iscsid can hang or crash

  [Test Case]
  1) This issue is exacerbated by increased logging output. To configure open-iscsi to emit the maximum amount of messages, edit /etc/init.d/open-iscsi and change the line:
    start-stop-daemon --start --quiet --exec $DAEMON
  to:
    start-stop-daemon --start --quiet --exec $DAEMON -- -d 8

  On your storage server, create multiple LUN targets and expose them to
  your Ubuntu system. I recommend 3 or more.

  Edit the attached repro.sh script to set the TARGET variable to point
  to the correct IP of your storage server. Execute this script on your
  Ubuntu system. When an error occurs, this script should hang or begin
  emitting errors.

  [Regression Potential] 
  I would expect regressions here to have the same symptoms of the original bug - e.g. hangs, crashes.

  Original report follows.

  We occasionally see iscsiadm hang indefinitely on a system. It is
  difficult to reproduce, there maybe months between occurrences. This
  happened again today, and we collected the following information.

  We captured an strace of a failing iscsiadm discover process, I'll
  attach it here. All iscsiadm processes that are hung are also in
  recvmsg() when we attach strace.

  It was observed that there was one iscsid process - where normally
  there is 2. The pids are normally consecutive, so I presume one forks
  the other before daemonizing.

  syslog contains the following messages from iscsid:

  Sep 20 15:55:57 cn5 iscsid: semop up failed 4
  Sep 20 15:56:05 cn5 iscsid: semop down failed
  Sep 20 15:56:23 cn5 iscsid: semop up failed 4
  Sep 20 16:02:23 cn5 iscsid: semop up failed 4
  Sep 20 16:02:26 cn5 iscsid: semop up failed 4
  Sep 20 16:02:26 cn5 iscsid: semop up failed 4
  Sep 20 16:02:26 cn5 iscsid: semop up failed 4
  Sep 20 16:02:32 cn5 iscsid: semop up failed 4
  Sep 20 18:01:49 cn5 iscsid: semop up failed 4
  Sep 20 18:03:56 cn5 iscsid: semop up failed 4
  Sep 20 18:05:03 cn5 iscsid: semop up failed 4

  This sounds similar to the issue described in
  https://bugzilla.redhat.com/show_bug.cgi?id=676804, but note the
  different errno.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1053306/+subscriptions