[Bug 1886364] [NEW] initiator causes kernel crash when login lun/disk on Focal

xinliang 1886364 at bugs.launchpad.net
Mon Jul 6 03:14:11 UTC 2020 

Public bug reported:

Software version
linaro at j13-r120-t32-09:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal
linaro at j13-r120-t32-09:~$ uname -a
Linux j13-r120-t32-09 5.4.0-37-generic #41-Ubuntu SMP Wed Jun 3 17:57:16 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
linaro at j13-r120-t32-09:~$
stack at j13-r120-t32-09:~/devstack-plugin-ceph$ apt search ceph-iscsi
Sorting... Done
Full Text Search... Done
ceph-iscsi/focal,now 3.4-0ubuntu2 all [installed]
  common logic and CLI tools for creating and managing LIO gateways for Ceph

stack at j13-r120-t32-09:~/devstack-plugin-ceph$ apt search tcmu-runner
Sorting... Done
Full Text Search... Done
libtcmu2/focal,now 1.5.2-5build1 arm64 [installed,automatic]
  Library that handles the userspace side of the LIO TCM-User backstore

tcmu-runner/focal,now 1.5.2-5build1 arm64 [installed]
  Daemon that handles the userspace side of the LIO TCM-User backstore

Hardware
cavium thx1

root at j13-r120-t32-09:/home/linaro# lshw -c cpu
  *-cpu
       description: CPU
       product: ARM (CN88xx)
       vendor: CN8890-2000BG2601-CP-Y-G
       physical id: 2e
       bus info: cpu at 0
       version: 2.1
       serial: CPU Serial#
       slot: Socket
       size: 2GHz
       capacity: 2GHz
       clock: 156MHz
       capabilities: lm
       configuration: cores=48 enabledcores=48
root at j13-r120-t32-09:/home/linaro# lshw -c system
j13-r120-t32-09
    description: System
    product: R120-T32-00 (01234567890123456789AB)
    vendor: GIGABYTE
    version: 0100
    serial: GHG2N2912A0009
    width: 64 bits
    capabilities: smbios-3.0.0 dmi-3.0.0 smp cp15_barrier setend swp tagged_addr_disabled
    configuration: chassis=server family=Server sku=01234567890123456789AB uuid=00000000-0000-4000-8000-1C1B0D94A9DE

Reproduce
Setup ceph iscsi gate way and initiator in the same ubuntu focal all-in-one.
iSCSI targets setup
1, $ apt install ceph-iscsi targetcli-fb
2, ceph iscsi config
stack at j13-r120-t32-09:~/devstack-plugin-ceph$ sudo cat /etc/ceph/iscsi-gateway.cfg

[config]
api_port = 5002
api_password = openstack
api_user = openstack
api_secure = false
prometheus_host = 10.101.96.110
gateway_keyring = ceph.client.admin.keyring
cluster_name = ceph
trusted_ip_list = 10.101.96.110,localhost
minimum_gateways = 1
pool = volumes

3, target iqn, client iqn, disk/lun creation
https://docs.ceph.com/docs/master//rbd/iscsi-target-cli/

iSCSI initiator setup
1, $ apt install open-iscsi
https://www.server-world.info/en/note?os=Ubuntu_18.04&p=iscsi&f=3
2,
$ iscsiadm -m discovery -t sendtargets -p 10.101.96.110
$ sudo iscsiadm -m node -T iqn.1993-08.org.opendev:01:a9aa4032d2c1 -l
Login lun cause crash ceph iscsi gw node

[  122.112611] xfs filesystem being mounted at /var/lib/ceph supports timestamps until 2038 (0x7fffffff)
linaro at j13-r120-t32-09:~$ [ 1512.796815] Unable to handle kernel read from unreadable memory at virtual address 0000000001dc0040
[ 1512.805865] Mem abort info:
[ 1512.808647]   ESR = 0x96000004
[ 1512.811702]   EC = 0x25: DABT (current EL), IL = 32 bits
[ 1512.817023]   SET = 0, FnV = 0
[ 1512.820089]   EA = 0, S1PTW = 0
[ 1512.823238] Data abort info:
[ 1512.826128]   ISV = 0, ISS = 0x00000004
[ 1512.829972]   CM = 0, WnR = 0
[ 1512.832933] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000cd14a0000
[ 1512.839410] [0000000001dc0040] pgd=0000000000000000
[ 1512.844300] Internal error: Oops: 96000004 [#1] SMP
[ 1512.849169] Modules linked in: target_core_pscsi target_core_file target_core_iblock iscsi_target_mod xfs xt_REDIRECT xt_comment xt_nat xt_mark xt_connmark ip6table_raw iptable_raw xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_tables ip6table_filter ip6_tables iptable_filter bpfilter bridge stp llc target_core_user uio target_core_mod nf_conntrack_netlink binfmt_misc nls_iso8859_1 nfnetlink_cttimeout nfnetlink ipmi_ssif ipmi_devintf cavium_rng_vf joydev input_leds ipmi_msghandler cavium_rng thunderx_edac openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c sch_fq_codel ip_tables x_tables autofs4 crct10dif_ce ghash_ce nicvf ast cavium_ptp i2c_algo_bit drm_vram_helper sha2_ce ttm drm_kms_helper syscopyarea sha256_arm64 sysfillrect sysimgblt fb_sys_fops sha1_ce drm e1000e nicpf ahci thunder_bgx thunder_xcv i2c_thunderx mdio_thunder thunderx_mmc mdio_cavium hid_generic
[ 1512.849252]  usbhid hid aes_neon_bs aes_neon_blk aes_ce_blk crypto_simd cryptd aes_ce_cipher
[ 1512.945007] CPU: 0 PID: 11002 Comm: iscsi_trx Not tainted 5.4.0-37-generic #41-Ubuntu
[ 1512.952823] Hardware name: GIGABYTE R120-T32-00/MT30-GS1-00, BIOS T49 02/02/2018
[ 1512.960206] pstate: 80400005 (Nzcv daif +PAN -UAO)
[ 1512.964990] pc : flush_dcache_page+0x18/0x60
[ 1512.969258] lr : is_ring_space_avail+0x74/0x348 [target_core_user]
[ 1512.975424] sp : ffff80003b98bab0
[ 1512.978726] x29: ffff80003b98bab0 x28: ffff000c991241f0
[ 1512.984026] x27: 0000000000001000 x26: 0000000000001000
[ 1512.989326] x25: ffff80007f001000 x24: fffffdffffe00000
[ 1512.99462000000000000000
[ 1513.015825] x15: 0000000000000000 x14: 0000000000000004
[ 1513.021125] x13: 0000000000000001 x12: 0000000000000002
[ 1513.026425] x11: 0000000000000000 x10: ffff80001163af47
[ 1513.031725] x9 : 0000000000000000 x8 : ffff000cb02b0700
[ 1513.037025] x7 : 0000000000000000 x6 : 0000000000001390
[ 1513.042324] x5 : 0000000000000080 x4 : 0000000000800000
[ 1513.047629] x3 : 0000000000001000 x2 : 0000000000000078
[ 1513.052931] x1 : ffff000cd1dfc168 x0 : ffff800008fea95c
[ 1513.058232] Call trace:
[ 1513.060672]  flush_dcache_page+0x18/0x60
[ 1513.064589]  is_ring_space_avail+0x74/0x348 [target_core_user]
[ 1513.070411]  queue_cmd_ring+0x1fc/0x6b0 [target_core_user]
[ 1513.075886]  tcmu_queue_cmd+0xd8/0x31c [target_core_user]
[ 1513.081352]  __target_execute_cmd+0x30/0x118 [target_core_mod]
[ 1513.087205]  target_execute_cmd+0x19c/0x3e8 [target_core_mod]
[ 1513.092970]  transport_generic_new_cmd+0x200/0x3a0 [target_core_mod]
[ 1513.099344]  transport_handle_cdb_direct+0x50/0xb0 [target_core_mod]
[ 1513.105753]  iscsit_execute_cmd+0x2a8/0x328 [iscsi_target_mod]
[ 1513.111599]  iscsit_sequence_cmd+0xd8/0x1c8 [iscsi_target_mod]
[ 1513.117446]  iscsit_process_scsi_cmd+0xac/0xf8 [iscsi_target_mod]
[ 1513.123553]  iscsit_get_rx_pdu+0x42c/0xe28 [iscsi_target_mod]
[ 1513.129313]  iscsi_target_rx_thread+0xbc/0x148 [iscsi_target_mod]
[ 1513.135395]  kthread+0xec/0x118
[ 1513.138526]  ret_from_fork+0x10/0x18
[ 1513.142094] Code: f9000bf3 aa0003f3 aa1e03e0 d503201f (f9400260)
[ 1513.148175] ---[ end trace 3fb1a3e4d49f508d ]---
[ 1523.009525] Did not receive response to NOPIN on CID: 0, failing connection for I_T Nexus iqn.1994-05.com.redhat:rh7-client,i,0x00023d000002,iqn.1993-08.org.opendev:01:a9aa4032d2c1,t,0x01
[ 1523.026234] ------------[ cut here ]------------

** Affects: ceph-iscsi (Ubuntu)
     Importance: Undecided
         Status: New

** Description changed:

  Software version
  linaro at j13-r120-t32-09:~$ lsb_release -a
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 20.04 LTS
  Release:        20.04
  Codename:       focal
  linaro at j13-r120-t32-09:~$ uname -a
  Linux j13-r120-t32-09 5.4.0-37-generic #41-Ubuntu SMP Wed Jun 3 17:57:16 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
  linaro at j13-r120-t32-09:~$
+ stack at j13-r120-t32-09:~/devstack-plugin-ceph$ apt search ceph-iscsi
+ Sorting... Done
+ Full Text Search... Done
+ ceph-iscsi/focal,now 3.4-0ubuntu2 all [installed]
+   common logic and CLI tools for creating and managing LIO gateways for Ceph
+ 
+ stack at j13-r120-t32-09:~/devstack-plugin-ceph$ apt search tcmu-runner
+ Sorting... Done
+ Full Text Search... Done
+ libtcmu2/focal,now 1.5.2-5build1 arm64 [installed,automatic]
+   Library that handles the userspace side of the LIO TCM-User backstore
+ 
+ tcmu-runner/focal,now 1.5.2-5build1 arm64 [installed]
+   Daemon that handles the userspace side of the LIO TCM-User backstore
+ 
  
  Hardware
  root at j13-r120-t32-09:/home/linaro# lshw -c cpu
-   *-cpu
-        description: CPU
-        product: ARM (CN88xx)
-        vendor: CN8890-2000BG2601-CP-Y-G
-        physical id: 2e
-        bus info: cpu at 0
-        version: 2.1
-        serial: CPU Serial#
-        slot: Socket
-        size: 2GHz
-        capacity: 2GHz
-        clock: 156MHz
-        capabilities: lm
-        configuration: cores=48 enabledcores=48
+   *-cpu
+        description: CPU
+        product: ARM (CN88xx)
+        vendor: CN8890-2000BG2601-CP-Y-G
+        physical id: 2e
+        bus info: cpu at 0
+        version: 2.1
+        serial: CPU Serial#
+        slot: Socket
+        size: 2GHz
+        capacity: 2GHz
+        clock: 156MHz
+        capabilities: lm
+        configuration: cores=48 enabledcores=48
  root at j13-r120-t32-09:/home/linaro# lshw -c system
  j13-r120-t32-09
-     description: System
-     product: R120-T32-00 (01234567890123456789AB)
-     vendor: GIGABYTE
-     version: 0100
-     serial: GHG2N2912A0009
-     width: 64 bits
-     capabilities: smbios-3.0.0 dmi-3.0.0 smp cp15_barrier setend swp tagged_addr_disabled
-     configuration: chassis=server family=Server sku=01234567890123456789AB uuid=00000000-0000-4000-8000-1C1B0D94A9DE
- 
+     description: System
+     product: R120-T32-00 (01234567890123456789AB)
+     vendor: GIGABYTE
+     version: 0100
+     serial: GHG2N2912A0009
+     width: 64 bits
+     capabilities: smbios-3.0.0 dmi-3.0.0 smp cp15_barrier setend swp tagged_addr_disabled
+     configuration: chassis=server family=Server sku=01234567890123456789AB uuid=00000000-0000-4000-8000-1C1B0D94A9DE
  
  Reproduce
  Setup ceph iscsi gate way and initiator in the same ubuntu focal all-in-one.
  iSCSI targets setup
  1, $ apt install ceph-iscsi targetcli-fb
  2, ceph iscsi config
  stack at j13-r120-t32-09:~/devstack-plugin-ceph$ sudo cat /etc/ceph/iscsi-gateway.cfg
  
  [config]
  api_port = 5002
  api_password = openstack
  api_user = openstack
  api_secure = false
  prometheus_host = 10.101.96.110
  gateway_keyring = ceph.client.admin.keyring
  cluster_name = ceph
  trusted_ip_list = 10.101.96.110,localhost
  minimum_gateways = 1
  pool = volumes
  
  3, target iqn, client iqn, disk/lun creation
  https://docs.ceph.com/docs/master//rbd/iscsi-target-cli/
  
  iSCSI initiator setup
  1, $ apt install open-iscsi
  https://www.server-world.info/en/note?os=Ubuntu_18.04&p=iscsi&f=3
  2,
  $ iscsiadm -m discovery -t sendtargets -p 10.101.96.110
  $ sudo iscsiadm -m node -T iqn.1993-08.org.opendev:01:a9aa4032d2c1 -l
  Login lun cause crash ceph iscsi gw node
  
  [  122.112611] xfs filesystem being mounted at /var/lib/ceph supports timestamps until 2038 (0x7fffffff)
  linaro at j13-r120-t32-09:~$ [ 1512.796815] Unable to handle kernel read from unreadable memory at virtual address 0000000001dc0040
  [ 1512.805865] Mem abort info:
  [ 1512.808647]   ESR = 0x96000004
  [ 1512.811702]   EC = 0x25: DABT (current EL), IL = 32 bits
  [ 1512.817023]   SET = 0, FnV = 0
  [ 1512.820089]   EA = 0, S1PTW = 0
  [ 1512.823238] Data abort info:
  [ 1512.826128]   ISV = 0, ISS = 0x00000004
  [ 1512.829972]   CM = 0, WnR = 0
  [ 1512.832933] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000cd14a0000
  [ 1512.839410] [0000000001dc0040] pgd=0000000000000000
  [ 1512.844300] Internal error: Oops: 96000004 [#1] SMP
  [ 1512.849169] Modules linked in: target_core_pscsi target_core_file target_core_iblock iscsi_target_mod xfs xt_REDIRECT xt_comment xt_nat xt_mark xt_connmark ip6table_raw iptable_raw xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_tables ip6table_filter ip6_tables iptable_filter bpfilter bridge stp llc target_core_user uio target_core_mod nf_conntrack_netlink binfmt_misc nls_iso8859_1 nfnetlink_cttimeout nfnetlink ipmi_ssif ipmi_devintf cavium_rng_vf joydev input_leds ipmi_msghandler cavium_rng thunderx_edac openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c sch_fq_codel ip_tables x_tables autofs4 crct10dif_ce ghash_ce nicvf ast cavium_ptp i2c_algo_bit drm_vram_helper sha2_ce ttm drm_kms_helper syscopyarea sha256_arm64 sysfillrect sysimgblt fb_sys_fops sha1_ce drm e1000e nicpf ahci thunder_bgx thunder_xcv i2c_thunderx mdio_thunder thunderx_mmc mdio_cavium hid_generic
  [ 1512.849252]  usbhid hid aes_neon_bs aes_neon_blk aes_ce_blk crypto_simd cryptd aes_ce_cipher
  [ 1512.945007] CPU: 0 PID: 11002 Comm: iscsi_trx Not tainted 5.4.0-37-generic #41-Ubuntu
  [ 1512.952823] Hardware name: GIGABYTE R120-T32-00/MT30-GS1-00, BIOS T49 02/02/2018
  [ 1512.960206] pstate: 80400005 (Nzcv daif +PAN -UAO)
  [ 1512.964990] pc : flush_dcache_page+0x18/0x60
  [ 1512.969258] lr : is_ring_space_avail+0x74/0x348 [target_core_user]
  [ 1512.975424] sp : ffff80003b98bab0
  [ 1512.978726] x29: ffff80003b98bab0 x28: ffff000c991241f0
  [ 1512.984026] x27: 0000000000001000 x26: 0000000000001000
  [ 1512.989326] x25: ffff80007f001000 x24: fffffdffffe00000
  [ 1512.99462000000000000000
  [ 1513.015825] x15: 0000000000000000 x14: 0000000000000004
  [ 1513.021125] x13: 0000000000000001 x12: 0000000000000002
  [ 1513.026425] x11: 0000000000000000 x10: ffff80001163af47
  [ 1513.031725] x9 : 0000000000000000 x8 : ffff000cb02b0700
  [ 1513.037025] x7 : 0000000000000000 x6 : 0000000000001390
  [ 1513.042324] x5 : 0000000000000080 x4 : 0000000000800000
  [ 1513.047629] x3 : 0000000000001000 x2 : 0000000000000078
  [ 1513.052931] x1 : ffff000cd1dfc168 x0 : ffff800008fea95c
  [ 1513.058232] Call trace:
  [ 1513.060672]  flush_dcache_page+0x18/0x60
  [ 1513.064589]  is_ring_space_avail+0x74/0x348 [target_core_user]
  [ 1513.070411]  queue_cmd_ring+0x1fc/0x6b0 [target_core_user]
  [ 1513.075886]  tcmu_queue_cmd+0xd8/0x31c [target_core_user]
  [ 1513.081352]  __target_execute_cmd+0x30/0x118 [target_core_mod]
  [ 1513.087205]  target_execute_cmd+0x19c/0x3e8 [target_core_mod]
  [ 1513.092970]  transport_generic_new_cmd+0x200/0x3a0 [target_core_mod]
  [ 1513.099344]  transport_handle_cdb_direct+0x50/0xb0 [target_core_mod]
  [ 1513.105753]  iscsit_execute_cmd+0x2a8/0x328 [iscsi_target_mod]
  [ 1513.111599]  iscsit_sequence_cmd+0xd8/0x1c8 [iscsi_target_mod]
  [ 1513.117446]  iscsit_process_scsi_cmd+0xac/0xf8 [iscsi_target_mod]
  [ 1513.123553]  iscsit_get_rx_pdu+0x42c/0xe28 [iscsi_target_mod]
  [ 1513.129313]  iscsi_target_rx_thread+0xbc/0x148 [iscsi_target_mod]
  [ 1513.135395]  kthread+0xec/0x118
  [ 1513.138526]  ret_from_fork+0x10/0x18
  [ 1513.142094] Code: f9000bf3 aa0003f3 aa1e03e0 d503201f (f9400260)
  [ 1513.148175] ---[ end trace 3fb1a3e4d49f508d ]---
  [ 1523.009525] Did not receive response to NOPIN on CID: 0, failing connection for I_T Nexus iqn.1994-05.com.redhat:rh7-client,i,0x00023d000002,iqn.1993-08.org.opendev:01:a9aa4032d2c1,t,0x01
  [ 1523.026234] ------------[ cut here ]------------

** Package changed: open-iscsi (Ubuntu) => ceph-iscsi (Ubuntu)

** Description changed:

  Software version
  linaro at j13-r120-t32-09:~$ lsb_release -a
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 20.04 LTS
  Release:        20.04
  Codename:       focal
  linaro at j13-r120-t32-09:~$ uname -a
  Linux j13-r120-t32-09 5.4.0-37-generic #41-Ubuntu SMP Wed Jun 3 17:57:16 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
  linaro at j13-r120-t32-09:~$
  stack at j13-r120-t32-09:~/devstack-plugin-ceph$ apt search ceph-iscsi
  Sorting... Done
  Full Text Search... Done
  ceph-iscsi/focal,now 3.4-0ubuntu2 all [installed]
-   common logic and CLI tools for creating and managing LIO gateways for Ceph
+   common logic and CLI tools for creating and managing LIO gateways for Ceph
  
  stack at j13-r120-t32-09:~/devstack-plugin-ceph$ apt search tcmu-runner
  Sorting... Done
  Full Text Search... Done
  libtcmu2/focal,now 1.5.2-5build1 arm64 [installed,automatic]
-   Library that handles the userspace side of the LIO TCM-User backstore
+   Library that handles the userspace side of the LIO TCM-User backstore
  
  tcmu-runner/focal,now 1.5.2-5build1 arm64 [installed]
-   Daemon that handles the userspace side of the LIO TCM-User backstore
- 
+   Daemon that handles the userspace side of the LIO TCM-User backstore
  
  Hardware
+ cavium thx1
+ 
  root at j13-r120-t32-09:/home/linaro# lshw -c cpu
    *-cpu
         description: CPU
         product: ARM (CN88xx)
         vendor: CN8890-2000BG2601-CP-Y-G
         physical id: 2e
         bus info: cpu at 0
         version: 2.1
         serial: CPU Serial#
         slot: Socket
         size: 2GHz
         capacity: 2GHz
         clock: 156MHz
         capabilities: lm
         configuration: cores=48 enabledcores=48
  root at j13-r120-t32-09:/home/linaro# lshw -c system
  j13-r120-t32-09
      description: System
      product: R120-T32-00 (01234567890123456789AB)
      vendor: GIGABYTE
      version: 0100
      serial: GHG2N2912A0009
      width: 64 bits
      capabilities: smbios-3.0.0 dmi-3.0.0 smp cp15_barrier setend swp tagged_addr_disabled
      configuration: chassis=server family=Server sku=01234567890123456789AB uuid=00000000-0000-4000-8000-1C1B0D94A9DE
  
  Reproduce
  Setup ceph iscsi gate way and initiator in the same ubuntu focal all-in-one.
  iSCSI targets setup
  1, $ apt install ceph-iscsi targetcli-fb
  2, ceph iscsi config
  stack at j13-r120-t32-09:~/devstack-plugin-ceph$ sudo cat /etc/ceph/iscsi-gateway.cfg
  
  [config]
  api_port = 5002
  api_password = openstack
  api_user = openstack
  api_secure = false
  prometheus_host = 10.101.96.110
  gateway_keyring = ceph.client.admin.keyring
  cluster_name = ceph
  trusted_ip_list = 10.101.96.110,localhost
  minimum_gateways = 1
  pool = volumes
  
  3, target iqn, client iqn, disk/lun creation
  https://docs.ceph.com/docs/master//rbd/iscsi-target-cli/
  
  iSCSI initiator setup
  1, $ apt install open-iscsi
  https://www.server-world.info/en/note?os=Ubuntu_18.04&p=iscsi&f=3
  2,
  $ iscsiadm -m discovery -t sendtargets -p 10.101.96.110
  $ sudo iscsiadm -m node -T iqn.1993-08.org.opendev:01:a9aa4032d2c1 -l
  Login lun cause crash ceph iscsi gw node
  
  [  122.112611] xfs filesystem being mounted at /var/lib/ceph supports timestamps until 2038 (0x7fffffff)
  linaro at j13-r120-t32-09:~$ [ 1512.796815] Unable to handle kernel read from unreadable memory at virtual address 0000000001dc0040
  [ 1512.805865] Mem abort info:
  [ 1512.808647]   ESR = 0x96000004
  [ 1512.811702]   EC = 0x25: DABT (current EL), IL = 32 bits
  [ 1512.817023]   SET = 0, FnV = 0
  [ 1512.820089]   EA = 0, S1PTW = 0
  [ 1512.823238] Data abort info:
  [ 1512.826128]   ISV = 0, ISS = 0x00000004
  [ 1512.829972]   CM = 0, WnR = 0
  [ 1512.832933] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000cd14a0000
  [ 1512.839410] [0000000001dc0040] pgd=0000000000000000
  [ 1512.844300] Internal error: Oops: 96000004 [#1] SMP
  [ 1512.849169] Modules linked in: target_core_pscsi target_core_file target_core_iblock iscsi_target_mod xfs xt_REDIRECT xt_comment xt_nat xt_mark xt_connmark ip6table_raw iptable_raw xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_tables ip6table_filter ip6_tables iptable_filter bpfilter bridge stp llc target_core_user uio target_core_mod nf_conntrack_netlink binfmt_misc nls_iso8859_1 nfnetlink_cttimeout nfnetlink ipmi_ssif ipmi_devintf cavium_rng_vf joydev input_leds ipmi_msghandler cavium_rng thunderx_edac openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c sch_fq_codel ip_tables x_tables autofs4 crct10dif_ce ghash_ce nicvf ast cavium_ptp i2c_algo_bit drm_vram_helper sha2_ce ttm drm_kms_helper syscopyarea sha256_arm64 sysfillrect sysimgblt fb_sys_fops sha1_ce drm e1000e nicpf ahci thunder_bgx thunder_xcv i2c_thunderx mdio_thunder thunderx_mmc mdio_cavium hid_generic
  [ 1512.849252]  usbhid hid aes_neon_bs aes_neon_blk aes_ce_blk crypto_simd cryptd aes_ce_cipher
  [ 1512.945007] CPU: 0 PID: 11002 Comm: iscsi_trx Not tainted 5.4.0-37-generic #41-Ubuntu
  [ 1512.952823] Hardware name: GIGABYTE R120-T32-00/MT30-GS1-00, BIOS T49 02/02/2018
  [ 1512.960206] pstate: 80400005 (Nzcv daif +PAN -UAO)
  [ 1512.964990] pc : flush_dcache_page+0x18/0x60
  [ 1512.969258] lr : is_ring_space_avail+0x74/0x348 [target_core_user]
  [ 1512.975424] sp : ffff80003b98bab0
  [ 1512.978726] x29: ffff80003b98bab0 x28: ffff000c991241f0
  [ 1512.984026] x27: 0000000000001000 x26: 0000000000001000
  [ 1512.989326] x25: ffff80007f001000 x24: fffffdffffe00000
  [ 1512.99462000000000000000
  [ 1513.015825] x15: 0000000000000000 x14: 0000000000000004
  [ 1513.021125] x13: 0000000000000001 x12: 0000000000000002
  [ 1513.026425] x11: 0000000000000000 x10: ffff80001163af47
  [ 1513.031725] x9 : 0000000000000000 x8 : ffff000cb02b0700
  [ 1513.037025] x7 : 0000000000000000 x6 : 0000000000001390
  [ 1513.042324] x5 : 0000000000000080 x4 : 0000000000800000
  [ 1513.047629] x3 : 0000000000001000 x2 : 0000000000000078
  [ 1513.052931] x1 : ffff000cd1dfc168 x0 : ffff800008fea95c
  [ 1513.058232] Call trace:
  [ 1513.060672]  flush_dcache_page+0x18/0x60
  [ 1513.064589]  is_ring_space_avail+0x74/0x348 [target_core_user]
  [ 1513.070411]  queue_cmd_ring+0x1fc/0x6b0 [target_core_user]
  [ 1513.075886]  tcmu_queue_cmd+0xd8/0x31c [target_core_user]
  [ 1513.081352]  __target_execute_cmd+0x30/0x118 [target_core_mod]
  [ 1513.087205]  target_execute_cmd+0x19c/0x3e8 [target_core_mod]
  [ 1513.092970]  transport_generic_new_cmd+0x200/0x3a0 [target_core_mod]
  [ 1513.099344]  transport_handle_cdb_direct+0x50/0xb0 [target_core_mod]
  [ 1513.105753]  iscsit_execute_cmd+0x2a8/0x328 [iscsi_target_mod]
  [ 1513.111599]  iscsit_sequence_cmd+0xd8/0x1c8 [iscsi_target_mod]
  [ 1513.117446]  iscsit_process_scsi_cmd+0xac/0xf8 [iscsi_target_mod]
  [ 1513.123553]  iscsit_get_rx_pdu+0x42c/0xe28 [iscsi_target_mod]
  [ 1513.129313]  iscsi_target_rx_thread+0xbc/0x148 [iscsi_target_mod]
  [ 1513.135395]  kthread+0xec/0x118
  [ 1513.138526]  ret_from_fork+0x10/0x18
  [ 1513.142094] Code: f9000bf3 aa0003f3 aa1e03e0 d503201f (f9400260)
  [ 1513.148175] ---[ end trace 3fb1a3e4d49f508d ]---
  [ 1523.009525] Did not receive response to NOPIN on CID: 0, failing connection for I_T Nexus iqn.1994-05.com.redhat:rh7-client,i,0x00023d000002,iqn.1993-08.org.opendev:01:a9aa4032d2c1,t,0x01
  [ 1523.026234] ------------[ cut here ]------------

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceph-iscsi in Ubuntu.
https://bugs.launchpad.net/bugs/1886364

Title:
  initiator causes kernel crash when login lun/disk on Focal

Status in ceph-iscsi package in Ubuntu:
  New

Bug description:
  Software version
  linaro at j13-r120-t32-09:~$ lsb_release -a
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 20.04 LTS
  Release:        20.04
  Codename:       focal
  linaro at j13-r120-t32-09:~$ uname -a
  Linux j13-r120-t32-09 5.4.0-37-generic #41-Ubuntu SMP Wed Jun 3 17:57:16 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
  linaro at j13-r120-t32-09:~$
  stack at j13-r120-t32-09:~/devstack-plugin-ceph$ apt search ceph-iscsi
  Sorting... Done
  Full Text Search... Done
  ceph-iscsi/focal,now 3.4-0ubuntu2 all [installed]
    common logic and CLI tools for creating and managing LIO gateways for Ceph

  stack at j13-r120-t32-09:~/devstack-plugin-ceph$ apt search tcmu-runner
  Sorting... Done
  Full Text Search... Done
  libtcmu2/focal,now 1.5.2-5build1 arm64 [installed,automatic]
    Library that handles the userspace side of the LIO TCM-User backstore

  tcmu-runner/focal,now 1.5.2-5build1 arm64 [installed]
    Daemon that handles the userspace side of the LIO TCM-User backstore

  Hardware
  cavium thx1

  root at j13-r120-t32-09:/home/linaro# lshw -c cpu
    *-cpu
         description: CPU
         product: ARM (CN88xx)
         vendor: CN8890-2000BG2601-CP-Y-G
         physical id: 2e
         bus info: cpu at 0
         version: 2.1
         serial: CPU Serial#
         slot: Socket
         size: 2GHz
         capacity: 2GHz
         clock: 156MHz
         capabilities: lm
         configuration: cores=48 enabledcores=48
  root at j13-r120-t32-09:/home/linaro# lshw -c system
  j13-r120-t32-09
      description: System
      product: R120-T32-00 (01234567890123456789AB)
      vendor: GIGABYTE
      version: 0100
      serial: GHG2N2912A0009
      width: 64 bits
      capabilities: smbios-3.0.0 dmi-3.0.0 smp cp15_barrier setend swp tagged_addr_disabled
      configuration: chassis=server family=Server sku=01234567890123456789AB uuid=00000000-0000-4000-8000-1C1B0D94A9DE

  Reproduce
  Setup ceph iscsi gate way and initiator in the same ubuntu focal all-in-one.
  iSCSI targets setup
  1, $ apt install ceph-iscsi targetcli-fb
  2, ceph iscsi config
  stack at j13-r120-t32-09:~/devstack-plugin-ceph$ sudo cat /etc/ceph/iscsi-gateway.cfg

  [config]
  api_port = 5002
  api_password = openstack
  api_user = openstack
  api_secure = false
  prometheus_host = 10.101.96.110
  gateway_keyring = ceph.client.admin.keyring
  cluster_name = ceph
  trusted_ip_list = 10.101.96.110,localhost
  minimum_gateways = 1
  pool = volumes

  3, target iqn, client iqn, disk/lun creation
  https://docs.ceph.com/docs/master//rbd/iscsi-target-cli/

  iSCSI initiator setup
  1, $ apt install open-iscsi
  https://www.server-world.info/en/note?os=Ubuntu_18.04&p=iscsi&f=3
  2,
  $ iscsiadm -m discovery -t sendtargets -p 10.101.96.110
  $ sudo iscsiadm -m node -T iqn.1993-08.org.opendev:01:a9aa4032d2c1 -l
  Login lun cause crash ceph iscsi gw node

  [  122.112611] xfs filesystem being mounted at /var/lib/ceph supports timestamps until 2038 (0x7fffffff)
  linaro at j13-r120-t32-09:~$ [ 1512.796815] Unable to handle kernel read from unreadable memory at virtual address 0000000001dc0040
  [ 1512.805865] Mem abort info:
  [ 1512.808647]   ESR = 0x96000004
  [ 1512.811702]   EC = 0x25: DABT (current EL), IL = 32 bits
  [ 1512.817023]   SET = 0, FnV = 0
  [ 1512.820089]   EA = 0, S1PTW = 0
  [ 1512.823238] Data abort info:
  [ 1512.826128]   ISV = 0, ISS = 0x00000004
  [ 1512.829972]   CM = 0, WnR = 0
  [ 1512.832933] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000cd14a0000
  [ 1512.839410] [0000000001dc0040] pgd=0000000000000000
  [ 1512.844300] Internal error: Oops: 96000004 [#1] SMP
  [ 1512.849169] Modules linked in: target_core_pscsi target_core_file target_core_iblock iscsi_target_mod xfs xt_REDIRECT xt_comment xt_nat xt_mark xt_connmark ip6table_raw iptable_raw xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_tables ip6table_filter ip6_tables iptable_filter bpfilter bridge stp llc target_core_user uio target_core_mod nf_conntrack_netlink binfmt_misc nls_iso8859_1 nfnetlink_cttimeout nfnetlink ipmi_ssif ipmi_devintf cavium_rng_vf joydev input_leds ipmi_msghandler cavium_rng thunderx_edac openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c sch_fq_codel ip_tables x_tables autofs4 crct10dif_ce ghash_ce nicvf ast cavium_ptp i2c_algo_bit drm_vram_helper sha2_ce ttm drm_kms_helper syscopyarea sha256_arm64 sysfillrect sysimgblt fb_sys_fops sha1_ce drm e1000e nicpf ahci thunder_bgx thunder_xcv i2c_thunderx mdio_thunder thunderx_mmc mdio_cavium hid_generic
  [ 1512.849252]  usbhid hid aes_neon_bs aes_neon_blk aes_ce_blk crypto_simd cryptd aes_ce_cipher
  [ 1512.945007] CPU: 0 PID: 11002 Comm: iscsi_trx Not tainted 5.4.0-37-generic #41-Ubuntu
  [ 1512.952823] Hardware name: GIGABYTE R120-T32-00/MT30-GS1-00, BIOS T49 02/02/2018
  [ 1512.960206] pstate: 80400005 (Nzcv daif +PAN -UAO)
  [ 1512.964990] pc : flush_dcache_page+0x18/0x60
  [ 1512.969258] lr : is_ring_space_avail+0x74/0x348 [target_core_user]
  [ 1512.975424] sp : ffff80003b98bab0
  [ 1512.978726] x29: ffff80003b98bab0 x28: ffff000c991241f0
  [ 1512.984026] x27: 0000000000001000 x26: 0000000000001000
  [ 1512.989326] x25: ffff80007f001000 x24: fffffdffffe00000
  [ 1512.99462000000000000000
  [ 1513.015825] x15: 0000000000000000 x14: 0000000000000004
  [ 1513.021125] x13: 0000000000000001 x12: 0000000000000002
  [ 1513.026425] x11: 0000000000000000 x10: ffff80001163af47
  [ 1513.031725] x9 : 0000000000000000 x8 : ffff000cb02b0700
  [ 1513.037025] x7 : 0000000000000000 x6 : 0000000000001390
  [ 1513.042324] x5 : 0000000000000080 x4 : 0000000000800000
  [ 1513.047629] x3 : 0000000000001000 x2 : 0000000000000078
  [ 1513.052931] x1 : ffff000cd1dfc168 x0 : ffff800008fea95c
  [ 1513.058232] Call trace:
  [ 1513.060672]  flush_dcache_page+0x18/0x60
  [ 1513.064589]  is_ring_space_avail+0x74/0x348 [target_core_user]
  [ 1513.070411]  queue_cmd_ring+0x1fc/0x6b0 [target_core_user]
  [ 1513.075886]  tcmu_queue_cmd+0xd8/0x31c [target_core_user]
  [ 1513.081352]  __target_execute_cmd+0x30/0x118 [target_core_mod]
  [ 1513.087205]  target_execute_cmd+0x19c/0x3e8 [target_core_mod]
  [ 1513.092970]  transport_generic_new_cmd+0x200/0x3a0 [target_core_mod]
  [ 1513.099344]  transport_handle_cdb_direct+0x50/0xb0 [target_core_mod]
  [ 1513.105753]  iscsit_execute_cmd+0x2a8/0x328 [iscsi_target_mod]
  [ 1513.111599]  iscsit_sequence_cmd+0xd8/0x1c8 [iscsi_target_mod]
  [ 1513.117446]  iscsit_process_scsi_cmd+0xac/0xf8 [iscsi_target_mod]
  [ 1513.123553]  iscsit_get_rx_pdu+0x42c/0xe28 [iscsi_target_mod]
  [ 1513.129313]  iscsi_target_rx_thread+0xbc/0x148 [iscsi_target_mod]
  [ 1513.135395]  kthread+0xec/0x118
  [ 1513.138526]  ret_from_fork+0x10/0x18
  [ 1513.142094] Code: f9000bf3 aa0003f3 aa1e03e0 d503201f (f9400260)
  [ 1513.148175] ---[ end trace 3fb1a3e4d49f508d ]---
  [ 1523.009525] Did not receive response to NOPIN on CID: 0, failing connection for I_T Nexus iqn.1994-05.com.redhat:rh7-client,i,0x00023d000002,iqn.1993-08.org.opendev:01:a9aa4032d2c1,t,0x01
  [ 1523.026234] ------------[ cut here ]------------

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ceph-iscsi/+bug/1886364/+subscriptions

More information about the Ubuntu-openstack-bugs mailing list