[Bug 1569925] Re: Shutdown hang on 16.04 with iscsi targets

Thu Aug 24 02:32:27 UTC 2017

Okay, besides all crap I posted - which are definitely right and will
possibly be addressed in other bugs, and also show other problems that
could make systemd to hang because of iscsi disks - I was able to cause
the same symptom of this bug:

[  OK  ] Stopped LSB: ebtables ruleset management.
[  OK  ] Stopped Login to default iSCSI targets.
         Stopping iSCSI initiator daemon (iscsid)...
[  OK  ] Stopped iSCSI initiator daemon (iscsid).
[  OK  ] Stopped target Network is Online.
[  OK  ] Stopped target Network.
         Stopping Raise network interfaces...
         Stopping ifup for bond0.20...
[  OK  ] Stopped ifup for bond0.20.
         Stopping ifup for bond0.10...
[  OK  ] Stopped ifup for bond0.10.
         Stopping ifup for bond0...
[  OK  ] Stopped ifup for bond0.
         Stopping ifup for internal...
[  OK  ] Stopped ifup for internal.
[  OK  ] Stopped Raise network interfaces.
[  OK  ] Stopped target Network (Pre).
[  OK  ] Stopped Apply Kernel Variables.
[  OK  ] Stopped Load Kernel Modules.
[  OK  ] Stopped target Local File Systems.
         Unmounting /run/cgmanager/fs...
         Unmounting /run/user/1000...
         Unmounting /xfs...
         Unmounting /ext4...
[  OK  ] Unmounted /run/cgmanager/fs.
[  OK  ] Unmounted /run/user/1000.

[  522.972114]  connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295020535, last ping 4295021788, now 4295023040
[  522.974085]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295020535, last ping 4295021788, now 4295023040
[  522.975905]  connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295020535, last ping 4295021788, now 4295023040
[  522.976087]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295020535, last ping 4295021788, now 4295023040

[  720.156456] INFO: task umount:11860 blocked for more than 120 seconds.
[  720.157477]       Not tainted 4.4.0-93-generic #116-Ubuntu
[  720.158282] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  720.159695] INFO: task umount:11861 blocked for more than 120 seconds.
[  720.160680]       Not tainted 4.4.0-93-generic #116-Ubuntu
[  720.161587] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  733.180181] sd 2:0:0:1: timing out command, waited 180s
[  733.181191] blk_update_request: I/O error, dev sdc, sector 0
[  733.182164] sd 4:0:0:1: timing out command, waited 180s
[  733.183084] blk_update_request: I/O error, dev sdb, sector 0

[  OK  ] Unmounted /ext4.
[  OK  ] Unmounted /xfs.
[  OK  ] Reached target Unmount All Filesystems.
[  OK  ] Stopped target Local File Systems (Pre).
[  OK  ] Stopped Create Static Device Nodes in /dev.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Reached target Shutdown.
[  805.018886] systemd-shutdown[1]: Failed to finalize  DM devices, ignoring

<system hangs forever>

And that is likely because I added 2 paths to each of my nics - using
multipath, likely unrelated - and configured bonding + vlan within
"ifupdown" - likely what causes this. After that change, I was able to
make systemd to freeze in the same (likely) output as of this bug (and
the one I'm taking care of). I'll investigate further tomorrow, but this
can be related to ifupdown and its locking mechanism for configuring/de-
configuring bonds/vlans/slaves interfaces.

Matt,

If not too much trouble to you, could you share the
/etc/network/interfaces you were using ? You presented yourself and I
was so focused in working this case that I forgot to mention: I'm one of
the Sustaining Engineering Tech Leads @ Canonical. Usually those type of
"partnership" product enablements/fixes are done 4 hands in other
channels, that is why I felt lost in the beginning about your role here,
I'm sorry if that bothered you.

Now that I was to reproduce, I won't bother much, just want to compare
your "interfaces" file with the ones I have (facing similar problem).

Tks

-Rafael

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1569925

Title:
  Shutdown hang on 16.04 with iscsi targets

Status in systemd package in Ubuntu:
  Confirmed
Status in systemd source package in Xenial:
  In Progress

Bug description:
  I have 4 servers running the latest 16.04 updates from the development
  branch (as of right now).

  Each server is connected to NetApp storage using iscsi software
  initiator.  There are a total of 56 volumes spread across two NetApp
  arrays.  Each volume has 4 paths available to it which are being
  managed by device mapper.

  While logged into the iscsi sessions all I have to do is reboot the
  server and I get a hang.

  I see a message that says:

    "Reached target Shutdown"

  followed by

    "systemd-shutdown[1]: Failed to finalize DM devices, ignoring"

  and then I see 8 lines that say:

    "connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***"
    "connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***"
    "connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***"
    "connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***"
    "connection5:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***"
    "connection6:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***"
    "connection7:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***"
    "connection8:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4311815***, last ping 43118164**, now 4311817***"
    NOTE: the actual values of the *'s differ for each line above.

  This seems like a bug somewhere but I am unaware of any additional
  logging that I could turn on to pinpoint the problem.

  Note I also have similar setups that are not doing iscsi and they
  don't have this problem.

  Here is a screenshot of what I see on the shell when I try to reboot:

  (https://launchpadlibrarian.net/291303059/Screenshot.jpg)

  This is being tracked in NetApp bug tracker CQ number 860251.

  If I log out of all iscsi sessions before rebooting then I do not
  experience the hang:

  iscsiadm -m node -U all

  We are wondering if this could be some kind of shutdown ordering
  problem.  Like the network devices have already disappeared and then
  iscsi tries to perform some operation (hence the ping timeouts).

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions