[Bug 1409904] Re: Needed patches for InfiniBand Support: Flow Steering and Offload Support + Fixes
Rafael David Tinoco
rafael.tinoco at canonical.com
Thu Sep 3 02:07:07 UTC 2015
Verified TGT iSER support:
Linux iserserver.main 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Linux iserclient.main 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
### SERVER
root at iserserver:~# dd if=/dev/zero of=/disks/lunit1 bs=1M count=1024 oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 10.3035 s, 104 MB/s
root at iserserver:~# tgt-setup-lun -n lunit1 -d /disks/lunit1 -t iser
Using transport: iser
Creating new target (name=iqn.2001-04.com.iserserver-lunit1, tid=1)
Adding a logical unit (/disks/lunit1) to target, tid=1
Accepting connections from all initiators
root at iserserver:~# ifconfig ib0
ib0 Link encap:UNSPEC HWaddr 80-00-05-8A-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:172.16.0.21 Bcast:172.16.0.127 Mask:255.255.255.128
### CLIENT
inaddy at iserclient:~$ sudo apt-get install open-iscsi
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
open-iscsi
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 268 kB of archives.
After this operation, 2,221 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main open-iscsi amd64 2.0.873-3ubuntu9 [268 kB]
Fetched 268 kB in 0s (466 kB/s)
Selecting previously unselected package open-iscsi.
(Reading database ... 80686 files and directories currently installed.)
Preparing to unpack .../open-iscsi_2.0.873-3ubuntu9_amd64.deb ...
Unpacking open-iscsi (2.0.873-3ubuntu9) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up open-iscsi (2.0.873-3ubuntu9) ...
Processing triggers for ureadahead (0.100.0-16) ...
inaddy at iserclient:~$ sudo su -
root at iserclient:~# update-rc.d -f open-iscsi remove
Removing any system startup links for /etc/init.d/open-iscsi ...
/etc/rc0.d/K81open-iscsi
/etc/rc1.d/K81open-iscsi
/etc/rc6.d/K81open-iscsi
/etc/rcS.d/S45open-iscsi
root at iserclient:~# service open-iscsi stop
* Unmounting iscsi-backed filesystems [ OK ]
* Disconnecting iSCSI targets [ OK ]
* Stopping iSCSI initiator service [ OK ]
root at iserclient:~# service open-iscsi start
* Starting iSCSI initiator service iscsid [ OK ]
* Setting up iSCSI targets
iscsiadm: No records found
[ OK ]
* Mounting network filesystems [ OK ]
root at iserclient:~# ifconfig bi0
bi0: error fetching interface information: Device not found
root at iserclient:~# ifconfig ib0
ib0 Link encap:UNSPEC HWaddr 80-00-05-8B-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:172.16.0.22 Bcast:172.16.0.127 Mask:255.255.255.128
inet6 addr: fe80::214:500:0:85/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
RX packets:3960620 errors:0 dropped:0 overruns:0 frame:0
TX packets:1107686 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:7078431089 (7.0 GB) TX bytes:62030601 (62.0 MB)
root at iserclient:~# iscsiadm -m discovery --op=show --type sendtargets --portal 172.16.0.21 -I iser
172.16.0.21:3260,1 iqn.2001-04.com.iserserver-lunit1
root at iserclient:~# iscsiadm -m discovery --op=new --op=del --type sendtargets --portal 172.16.0.21 -I iser
172.16.0.21:3260,1 iqn.2001-04.com.iserserver-lunit1
root at iserclient:~# iscsiadm -m node -l
Logging in to [iface: iser, target: iqn.2001-04.com.iserserver-lunit1, portal: 172.16.0.21,3260] (multiple)
Login to [iface: iser, target: iqn.2001-04.com.iserserver-lunit1, portal: 172.16.0.21,3260] successful.
root at iserclient:~# dmesg | tail -3
[17385.747524] sd 4:0:0:1: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[17385.751051] sda: unknown partition table
[17385.752130] sd 4:0:0:1: [sda] Attached SCSI disk
root at iserclient:~# dd if=/dev/zero of=/dev/sda bs=1M count=1024 oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1.8081 s, 594 MB/s
root at iserclient:~# service open-iscsi stop
* Unmounting iscsi-backed filesystems [ OK ]
* Disconnecting iSCSI targets
Logging out of session [sid: 3, target: iqn.2001-04.com.iserserver-lunit1, portal: 172.16.0.21,3260]
Logout of [sid: 3, target: iqn.2001-04.com.iserserver-lunit1, portal: 172.16.0.21,3260] successful.
[ OK ]
* Stopping iSCSI initiator service [ OK ]
root at iserclient:~#
root at iserclient:~# service open-iscsi start
* Starting iSCSI initiator service iscsid [ OK ]
* Setting up iSCSI targets
iscsiadm: No records found
[ OK ]
* Mounting network filesystems [ OK ]
root at iserclient:~# ls -lah /dev/sd*
ls: cannot access /dev/sd*: No such file or directory
root at iserclient:~# iscsiadm -m node -l
Logging in to [iface: iser, target: iqn.2001-04.com.iserserver-lunit1, portal: 172.16.0.21,3260] (multiple)
Login to [iface: iser, target: iqn.2001-04.com.iserserver-lunit1, portal: 172.16.0.21,3260] successful.
root at iserclient:~# dd if=/dev/zero of=/dev/sda bs=1M count=1024 oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1.52628 s, 704 MB/s
--
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to a duplicate bug report (1445038).
https://bugs.launchpad.net/bugs/1409904
Title:
Needed patches for InfiniBand Support: Flow Steering and Offload
Support + Fixes
Status in corosync package in Ubuntu:
Invalid
Status in fio package in Ubuntu:
Invalid
Status in glusterfs package in Ubuntu:
Invalid
Status in libcxgb3 package in Ubuntu:
Invalid
Status in libibcm package in Ubuntu:
Invalid
Status in libibverbs package in Ubuntu:
Fix Released
Status in libipathverbs package in Ubuntu:
Invalid
Status in libmlx4 package in Ubuntu:
Fix Released
Status in libmlx5 package in Ubuntu:
Invalid
Status in libmthca package in Ubuntu:
Invalid
Status in libnes package in Ubuntu:
Invalid
Status in librdmacm package in Ubuntu:
Invalid
Status in openmpi package in Ubuntu:
Invalid
Status in perftest package in Ubuntu:
Invalid
Status in qpid-cpp package in Ubuntu:
Invalid
Status in srptools package in Ubuntu:
Invalid
Status in tgt package in Ubuntu:
Invalid
Status in corosync source package in Trusty:
Invalid
Status in fio source package in Trusty:
Invalid
Status in glusterfs source package in Trusty:
Invalid
Status in libcxgb3 source package in Trusty:
Invalid
Status in libibcm source package in Trusty:
Invalid
Status in libibverbs source package in Trusty:
Fix Committed
Status in libipathverbs source package in Trusty:
Invalid
Status in libmlx4 source package in Trusty:
Fix Committed
Status in libmlx5 source package in Trusty:
Invalid
Status in libmthca source package in Trusty:
Invalid
Status in libnes source package in Trusty:
Invalid
Status in librdmacm source package in Trusty:
Invalid
Status in openmpi source package in Trusty:
Invalid
Status in perftest source package in Trusty:
Invalid
Status in qpid-cpp source package in Trusty:
Invalid
Status in srptools source package in Trusty:
Invalid
Status in tgt source package in Trusty:
Fix Committed
Status in libibverbs source package in Vivid:
Fix Released
Status in libmlx4 source package in Vivid:
Fix Released
Status in tgt source package in Vivid:
Fix Released
Bug description:
[Impact]
== Trusty:
* Today's libibverbs is missing important code for arm64 moonshot project.
* Today's libmlx4 is missing important code for arm64 moonshot project.
- Flow steering control for Openstack enablement on IPoIB
- Offload support flags (for performance)
* Today's tgt is suffering from the lack of iSER support (LP: #1445038).
* qpid-cpp depends on ruby1.8 and should depend on ruby1.9 (LP: #1367255).
Other Cases to justify such SRU:
- LP: #1401575 (dup: #1445038) - iSER not working in Trusty
- LP: #1445038 - TGT does not support ISER discovery
- LP: #1443648 - iSER logouts cause kernel panic in 3.13
- LP: #1401141 - ISC DHCP server does not work with InfiniBand
- LP: #1367255 - qpid-cpp depends on ruby1.8 and should depend on ruby1.9
Collateral Impact:
- srptools
- rdmacm-utils
- perftest
- librdmawrap2
- libopenmpi1.6
- libnes1
- libmthca1
- libmlx5-1
- libipathverbs1
- libibcm1
- libcxgb3-1
- ibverbs-utils
- glusterfs-common
- fio
- libtotem-pg5
- librdmacm1
* libibverbs was re-factored and all rdepends should follow the ABI.
* ALL SRUs should be done AT ONCE together because of ABI dependency.
== Utopic:
* libibverbs had offload capability added.
* libmlx4 had offload capability added.
* tgt does not support iSER.
Other Cases to justify such SRU:
- LP: #1445038 - TGT does not support ISER discovery
- LP: #1401141 - ISC DHCP server does not work with InfiniBand
No Collateral Impact!!!
== Vivid:
* libibverbs had offload capability added.
* libmlx4 had offload capability added.
* tgt does not support iSER.
Other Cases to justify such SRU:
- LP: #1445038 - TGT does not support ISER discovery
- LP: #1401141 - ISC DHCP server does not work with InfiniBand
No Collateral Impact!!!
[ Test Case ]
* libibverbs and libmlx4 are being tested by Mellanox (who better ?)
* There is no support for flow steering for IB (needed) using libibverbs
- try to use IPoIB and flow steering will not work (openstack related)
* iSER is broken in Trusty when using tgt daemon with iSER.
* tgt test case can be seen in LP: #1445038
- you can check test case for tgt in case above.
* https://launchpad.net/~inaddy/+archive/ubuntu/lp1409904/
available for testing ALL proposed fixes.
- tests are being asked for Mellanox to accept these fixes.
[Regression Potential]
* Only for InfiniBand users who might rely on libibverbs and its rdepends.
* Today's libibverbs, libmlx4, isc-dhcp and tgt (iser) have issues with IB.
[Other Info]
* Trusty will be fully IB compatible and fixed per Mellanox request.
* Please do consider this as SRU and yes it will be big but needed.
* Any doubt please contact: Rafael Tinoco <rafael.tinoco at canonical.com>
ORIGINAL BUG DESCRIPTION:
Mellanox is requesting Canonical and Ubuntu Community for some patches
to be backported from upstream to libibverbs and libmlx4 in order to
have their HW proper supported in Trusty, Utopic and Vivid. There are
in charge of testing those packages and providing proper verification.
The following patch list is needed:
******
- libibverbs:
Description: Add checksum offload support capability flag
Origin: https://patchwork.kernel.org/patch/5572101/
Description: Add general definitions to support uverbs extensions
Origin: upstream, commit: cbf2a35162afcc9e97870b7b18d6477133a8dfa2
Description: Add receive flow steering support
Origin: upstream, commit: 389de6a6ef4ed90615629a9eed7f9f17bedf47e3
Description: Infrastructure to support verbs extensions
Origin: upstream, commit: 6be16586e081728a28fc41ce2fcc41867b7375ec
- libmlx4:
Description: Remove compatibility with libibverbs < 1.1.7-1ubuntu1
Origin: upstream, commit: bcc5d064268073cb996bdbac1ae36728db3b2c96
Description: Update to use libiverbs extension support
Origin: upstream, commit: d00cfeb15ef4516539725b2f6c7d992e7201e13d
Description: Add receive flow steering support
Origin: upstream, commit: 1b6875d4c2cba3b751ad0b4286cf52b6a22bff97
Description: Add support for IBV_SEND_IP_CSUM for posted send work-requests.
Origin: https://patchwork.kernel.org/patch/5572111/
******
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/corosync/+bug/1409904/+subscriptions
More information about the Ubuntu-sponsors
mailing list