[Bug 1435706] Re: DevLossTO, FastIoFailTO settings do not match multipath.conf expected values

Fri Jul 31 06:37:43 UTC 2015

To me fix doesn't actually appear to work. After upgrading to multipath-
tools 0.4.9-3ubuntu7.4on an amd64 trusty and rebooting, the
fast_io_fail_tmo and dev_loss_tmo values do not get written to sysfs:

$ grep . /sys/class/fc_remote_ports/*/*_tmo
/sys/class/fc_remote_ports/rport-2:0-0/dev_loss_tmo:30
/sys/class/fc_remote_ports/rport-2:0-0/fast_io_fail_tmo:off
/sys/class/fc_remote_ports/rport-2:0-1/dev_loss_tmo:30
/sys/class/fc_remote_ports/rport-2:0-1/fast_io_fail_tmo:off
/sys/class/fc_remote_ports/rport-3:0-0/dev_loss_tmo:30
/sys/class/fc_remote_ports/rport-3:0-0/fast_io_fail_tmo:off
/sys/class/fc_remote_ports/rport-3:0-1/dev_loss_tmo:30
/sys/class/fc_remote_ports/rport-3:0-1/fast_io_fail_tmo:off

The device stanza from multipath.conf contains the following:

	device {
		vendor			"DGC|EMC"
		product			"RAID [0-9]*|VRAID|SYMMETRIX.*"
		path_grouping_policy	group_by_prio
		getuid_callout		"/lib/udev/scsi_id --whitelisted --device=/dev/%n"
		path_selector		"round-robin 0"
		path_checker		emc_clariion
		features		"0"
		hardware_handler	"1 emc"
		prio			emc
		failback		immediate
		rr_weight		uniform
		no_path_retry		queue
		rr_min_io		100
		fast_io_fail_tmo	3
		dev_loss_tmo		2147483647
	}

FWIW, I can manually set the sysfs settings to the desired values:

$ echo 3 | sudo tee /sys/class/fc_remote_ports/rport-*/fast_io_fail_tmo
3
$ echo 2147483647 | sudo tee /sys/class/fc_remote_ports/rport-*/dev_loss_tmo
2147483647
$ grep . /sys/class/fc_remote_ports/*/*_tmo
/sys/class/fc_remote_ports/rport-2:0-0/dev_loss_tmo:2147483647
/sys/class/fc_remote_ports/rport-2:0-0/fast_io_fail_tmo:3
/sys/class/fc_remote_ports/rport-2:0-1/dev_loss_tmo:2147483647
/sys/class/fc_remote_ports/rport-2:0-1/fast_io_fail_tmo:3
/sys/class/fc_remote_ports/rport-3:0-0/dev_loss_tmo:2147483647
/sys/class/fc_remote_ports/rport-3:0-0/fast_io_fail_tmo:3
/sys/class/fc_remote_ports/rport-3:0-1/dev_loss_tmo:2147483647
/sys/class/fc_remote_ports/rport-3:0-1/fast_io_fail_tmo:3

Tore

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to multipath-tools in Ubuntu.
https://bugs.launchpad.net/bugs/1435706

Title:
  DevLossTO, FastIoFailTO settings do not match multipath.conf expected
  values

Status in multipath-tools package in Ubuntu:
  Fix Released
Status in multipath-tools source package in Trusty:
  Fix Released
Status in multipath-tools source package in Vivid:
  Fix Committed

Bug description:
  [Impact]
  This bug impacts multipath users who need to tweak timeout values for DevLoss and FastIoFail for performance reasons.

  [Test Case]
  On a multipath system, attempt to modify DevLossTO or FastIoFailTO, then verify that the values got applied with 'multipath -l'. See below.

  [Regression Potential]
  Users who have already modified these values but have not noticed they did not properly apply may notice a change in behavior on device failure.

  ---

  Problem Description
  =========================================
  DevLossTO, FastIoFailTO settings do not match multipath.conf expected values

  ---uname output---
  Linux ilp1fc85apA4.tuc.stglabs.ibm.com 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:09:21 UTC 2014 ppc64le ppc64le ppc64le GNU/Linuxuname -m

  Machine Type = p7 8247

  Steps to Reproduce
  ===================================
   Verify DevLossTO, FastIoFailTO setting match multipath.conf expected values

  == Comment: #31 - Thadeu Lima De Souza Cascardo <thadeul at br.ibm.com> - 2015-03-20 10:57:20 ==
  OK.

  From the point of view of multipathd, everything seems correct, by
  looking at the logs.

  I even parsed syslog and the output of getHBAInfo in order to find
  inconsistencies, and the inconsistency is between what multipathd
  logged as configured for a given target, and what its rport reports at
  getHBAInfo.

  So, either multipathd is not configuring the timeouts even though it
  has the right configuration, or something else is changing those
  timeouts.

  The other problem is that multipathd does not include the dev_loss_tmo
  configuration for 2145 as can be seen from list config. So, it could
  be not parsing the configuration correctly, or there could be a
  problem with the configuration.

  At this point, to move forward, I would like to take a look at your
  system, and try reconfigure and looking at some strace output of
  multipathd, to check for writes into sysfs.

  == Comment: #34 - Thadeu Lima De Souza Cascardo <thadeul at br.ibm.com> - 2015-03-20 15:56:46 ==
  OK, so I investigated in the system and read some of the code and checked changelog.

  It looks like Ubuntu is shipping a fairly old version of multipath-
  tools, which is understandable, since multipath-tools is not very good
  in doing frequent releases, so one needs to either ship a version
  closer to upstream git or include its own large set of patches.

  One of the patches missing is the one attached next. Without that, any
  devices included in the built-in hardware table will have some of its
  attributes from the config file ignored. That is the case with 2145.
  So, we lose the dev_loss_tmo setting for that device.

  Cascardo.

  == Comment: #38 - Thadeu Lima De Souza Cascardo <thadeul at br.ibm.com> - 2015-03-20 16:25:39 ==
  The bug this patch fixes would explain why fast_io_fail_tmo is not correctly set in some cases, but not dev_loss_tmo. So, probably, there is another missing patch here. I would like to experiment with the two patches I mentioned, however. Let's try to do this on Monday?

  Cascardo.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1435706/+subscriptions