[Bug 1466654] Re: kernel soft lockup on nfs server when using a kerberos mount

Cristiano Cumer 1466654 at bugs.launchpad.net
Tue Oct 2 08:53:12 UTC 2018 

Why not use gss-proxy instead of rpc.svcgssd for NFS also in Ubuntu,
like other distress do?

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/1466654

Title:
  kernel soft lockup on nfs server when using a kerberos mount

Status in nfs-utils package in Ubuntu:
  Confirmed

Bug description:
  The kernel seems to lock up when accessing a kerberos mounted nfs
  share with a user holding a kerberos ticket. This is in a virtualbox
  vm(run by vagrant), but it also occurs in a VMware vm from a standard
  ubuntu 14.04 server install disk.

  To reproduce:
  Join machine to windows active directory domain using sssd
  Install nfs-kernel server, enable NEED_SVCGSSD
  Enable NEED_GSSD for the client(can be same host as the server)
  $ sudo mount -t nfs4 <fqdn>:/ /mnt/nfs -o sec=krb5
  $ sudo ls -l /mnt/nfs # this works ok
  $ kinit <a_domain_user>
  $ ls /mnt/nfs
  Permission denied error # I don't recall the exact wording
  # wait a few moments, and the kernel starts reporting a soft lockup.

  I think there are a few other things that trigger this error as well.
  It basically makes nfs+kerberos unusable.

  I have the kernel crash dump, and the debugsyms installed so if there
  is any other information I can provide please let me know.

  lsb_release -rd:
  Description:	Ubuntu 14.04.2 LTS
  Release:	14.04

  $ apt-cache policy nfs-common
  nfs-common:
    Installed: 1:1.2.8-6ubuntu1.1

  Info from crash:
        KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-53-generic
      DUMPFILE: dump.201506181954  [PARTIAL DUMP]
          CPUS: 2
          DATE: Thu Jun 18 19:54:08 2015
        UPTIME: 00:05:48
  LOAD AVERAGE: 1.50, 0.45, 0.18
         TASKS: 120
      NODENAME: t-fileserver
       RELEASE: 3.13.0-53-generic
       VERSION: #89-Ubuntu SMP Wed May 20 10:34:39 UTC 2015
       MACHINE: x86_64  (3581 Mhz)
        MEMORY: 511.6 MB
         PANIC: "Kernel panic - not syncing: softlockup: hung tasks"
           PID: 1353
       COMMAND: "rpc.svcgssd"
          TASK: ffff880014dce000  [THREAD_INFO: ffff88001514e000]
           CPU: 0
         STATE: TASK_RUNNING (PANIC)

  Backtrace:
  PID: 1353   TASK: ffff880014dce000  CPU: 0   COMMAND: "rpc.svcgssd"
   #0 [ffff88001fc03d18] machine_kexec at ffffffff8104ace2
   #1 [ffff88001fc03d68] crash_kexec at ffffffff810e7423
   #2 [ffff88001fc03e30] panic at ffffffff8171bcc4
   #3 [ffff88001fc03ea8] watchdog_timer_fn at ffffffff8110dc85
   #4 [ffff88001fc03ed8] __run_hrtimer at ffffffff8108e8c7
   #5 [ffff88001fc03f18] hrtimer_interrupt at ffffffff8108f08f
   #6 [ffff88001fc03f80] local_apic_timer_interrupt at ffffffff81043617
   #7 [ffff88001fc03f98] smp_apic_timer_interrupt at ffffffff81735ccf
   #8 [ffff88001fc03fb0] apic_timer_interrupt at ffffffff8173465d
  --- <IRQ stack> ---
   #9 [ffff88001514fd58] apic_timer_interrupt at ffffffff8173465d
      [exception RIP: qword_addhex+176]
      RIP: ffffffffa01c2df0  RSP: ffff88001514fe08  RFLAGS: 00000206
      RAX: 0000000000000001  RBX: 0000000000000006  RCX: 00000000000001f6
      RDX: ffff880015e54678  RSI: ffff88001514fe84  RDI: ffff88001514fe88
      RBP: ffff88001514fe18   R8: ffff880015e57cf5   R9: 000000000000030b
      R10: 0000000000000039  R11: 0000000000000027  R12: 0000000000000006
      R13: ffffea0000422420  R14: ffffea00003e46e0  R15: ffff88001514fe98
      ORIG_RAX: ffffffffffffff10  CS: 0010  SS: 0018
  #10 [ffff88001514fe20] rsi_request at ffffffffa01f61bb [auth_rpcgss]
  #11 [ffff88001514fe48] cache_read at ffffffffa01c5045 [sunrpc]
  #12 [ffff88001514fec0] cache_read_procfs at ffffffffa01c51a1 [sunrpc]
  #13 [ffff88001514fee8] proc_reg_read at ffffffff81224a6d
  #14 [ffff88001514ff08] vfs_read at ffffffff811bdf55
  #15 [ffff88001514ff40] sys_read at ffffffff811bea69
  #16 [ffff88001514ff80] system_call_fastpath at ffffffff8173391d
      RIP: 00007f85cf9f3810  RSP: 00007ffdba7dfe78  RFLAGS: 00000206
      RAX: 0000000000000000  RBX: ffffffff8173391d  RCX: ffffffffffffffff
      RDX: 0000000000001000  RSI: 00000000008746f0  RDI: 0000000000000004
      RBP: 00000000006083f8   R8: 0000000000000000   R9: 0000000000878820
      R10: 00007f85cfcc67b8  R11: 0000000000000246  R12: 0000000000000000
      R13: 0000000000000004  R14: 0000000000608400  R15: 00000000008744b0
      ORIG_RAX: 0000000000000000  CS: 0033  SS: 002b

  Log:
  [  348.084011] BUG: soft lockup - CPU#0 stuck for 23s! [rpc.svcgssd:1353]
  [  348.084011] Modules linked in: cts vboxsf(OX) nfsv4 rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache dm_crypt ip6t_REJECT ppdev xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_comment xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter parport_pc ip6_tables parport nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter serio_raw ip_tables x_tables vboxguest(OX) psmouse ahci libahci e1000
  [  348.084011] CPU: 0 PID: 1353 Comm: rpc.svcgssd Tainted: G           OX 3.13.0-53-generic #89-Ubuntu
  [  348.084011] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
  [  348.084011] task: ffff880014dce000 ti: ffff88001514e000 task.ti: ffff88001514e000
  [  348.084011] RIP: 0010:[<ffffffffa01c2df0>]  [<ffffffffa01c2df0>] qword_addhex+0xb0/0x100 [sunrpc]
  [  348.084011] RSP: 0018:ffff88001514fe08  EFLAGS: 00000206
  [  348.084011] RAX: 0000000000000001 RBX: 0000000000000006 RCX: 00000000000001f6
  [  348.084011] RDX: ffff880015e54678 RSI: ffff88001514fe84 RDI: ffff88001514fe88
  [  348.084011] RBP: ffff88001514fe18 R08: ffff880015e57cf5 R09: 000000000000030b
  [  348.084011] R10: 0000000000000039 R11: 0000000000000027 R12: 0000000000000006
  [  348.084011] R13: ffffea0000422420 R14: ffffea00003e46e0 R15: ffff88001514fe98
  [  348.084011] FS:  00007f85d02f3740(0000) GS:ffff88001fc00000(0000) knlGS:0000000000000000
  [  348.084011] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  [  348.084011] CR2: 0000000000408000 CR3: 00000000152f7000 CR4: 00000000000006f0
  [  348.084011] Stack:
  [  348.084011]  ffff88001478d580 ffff88001514fe88 ffff88001514fe40 ffffffffa01f61bb
  [  348.084011]  ffff8800150b2840 0000000000001000 ffff88001f48f500 ffff88001514feb8
  [  348.084011]  ffffffffa01c5045 0002000000000001 ffff88000c084a60 ffff880015c2bb60
  [  348.084011] Call Trace:
  [  348.084011]  [<ffffffffa01f61bb>] rsi_request+0x3b/0x50 [auth_rpcgss]
  [  348.084011]  [<ffffffffa01c5045>] cache_read.isra.19+0x275/0x3a0 [sunrpc]
  [  348.084011]  [<ffffffffa01c51a1>] cache_read_procfs+0x31/0x40 [sunrpc]
  [  348.084011]  [<ffffffff81224a6d>] proc_reg_read+0x3d/0x80
  [  348.084011]  [<ffffffff811bdf55>] vfs_read+0x95/0x160
  [  348.084011]  [<ffffffff811bea69>] SyS_read+0x49/0xa0
  [  348.084011]  [<ffffffff811d35e5>] ? SyS_poll+0x65/0x100
  [  348.084011]  [<ffffffff8173391d>] system_call_fastpath+0x1a/0x1f
  [  348.084011] Code: e0 27 42 8d 44 20 30 41 88 40 fe 41 0f b6 c2 83 f8 0a 44 89 d8 0f 4c c3 41 83 e9 02 83 e9 01 46 8d 54 10 30 0f 95 c0 41 83 f9 01 <45> 88 50 ff 7f aa 45 85 c9 7f 1f 5b 41 5c 5d 41 b9 ff ff ff ff 
  [  348.084011] Kernel panic - not syncing: softlockup: hung tasks
  [  348.084011] CPU: 0 PID: 1353 Comm: rpc.svcgssd Tainted: G           OX 3.13.0-53-generic #89-Ubuntu
  [  348.084011] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
  [  348.084011]  000000000000012d ffff88001fc03e28 ffffffff81722e1e ffffffff81a62b16
  [  348.084011]  ffff88001fc03ea0 ffffffff8171bcbd 0000000000000008 ffff88001fc03eb0
  [  348.084011]  ffff88001fc03e50 0000000000000086 0000000000000046 0000000000000007
  [  348.084011] Call Trace:
  [  348.084011]  <IRQ>  [<ffffffff81722e1e>] dump_stack+0x45/0x56
  [  348.084011]  [<ffffffff8171bcbd>] panic+0xc8/0x1d7
  [  348.084011]  [<ffffffff8110dc85>] watchdog_timer_fn+0x165/0x170
  [  348.084011]  [<ffffffff8108e8c7>] __run_hrtimer+0x77/0x1d0
  [  348.084011]  [<ffffffff8110db20>] ? watchdog_cleanup+0x10/0x10
  [  348.084011]  [<ffffffff8108f08f>] hrtimer_interrupt+0xef/0x230
  [  348.084011]  [<ffffffff81043617>] local_apic_timer_interrupt+0x37/0x60
  [  348.084011]  [<ffffffff81735ccf>] smp_apic_timer_interrupt+0x3f/0x60
  [  348.084011]  [<ffffffff8173465d>] apic_timer_interrupt+0x6d/0x80
  [  348.084011]  <EOI>  [<ffffffffa01c2df0>] ? qword_addhex+0xb0/0x100 [sunrpc]
  [  348.084011]  [<ffffffffa01f61bb>] rsi_request+0x3b/0x50 [auth_rpcgss]
  [  348.084011]  [<ffffffffa01c5045>] cache_read.isra.19+0x275/0x3a0 [sunrpc]
  [  348.084011]  [<ffffffffa01c51a1>] cache_read_procfs+0x31/0x40 [sunrpc]
  [  348.084011]  [<ffffffff81224a6d>] proc_reg_read+0x3d/0x80
  [  348.084011]  [<ffffffff811bdf55>] vfs_read+0x95/0x160
  [  348.084011]  [<ffffffff811bea69>] SyS_read+0x49/0xa0
  [  348.084011]  [<ffffffff811d35e5>] ? SyS_poll+0x65/0x100
  [  348.084011]  [<ffffffff8173391d>] system_call_fastpath+0x1a/0x1f

  ProblemType: Bug
  DistroRelease: Ubuntu 14.04
  Package: nfs-common 1:1.2.8-6ubuntu1.1
  ProcVersionSignature: Ubuntu 3.13.0-53.89-generic 3.13.11-ckt19
  Uname: Linux 3.13.0-53-generic x86_64
  ApportVersion: 2.14.1-0ubuntu3.11
  Architecture: amd64
  Date: Thu Jun 18 20:46:05 2015
  ProcEnviron:
   TERM=xterm-256color
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  SourcePackage: nfs-utils
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1466654/+subscriptions

More information about the foundations-bugs mailing list