[Bug 1977745] Re: nfs-utils/nfs-kernel-server (rpc.svcgssd) ignored /etc/nfs.conf settings

[Andreas Hasenack]

** Description changed:

  [Impact]
  The "principal" option for svcgssd in /etc/nfs.conf is being ignored by the service. This breaks NFS setups that use kerberos and need a specific principal name.
  
  Another issue is that some svcgssd options are missing from the
  nfs.conf(5) manpage, notably verbosity, rpc-verbosity and idmap-
  verbosity, that on its own probably wouldn't warrant an SRU, but can go
  together with this one. I don't think a new bug is needed, since they
  are about svcgssd too.
  
  Finally, when using the "principal" option, it would be helpful if the
  error message actually used the given principal name, instead of a
  generic "nfs/<your.host>@<YOUR.REALM>" one. Again, on its own it does
  not warrant an SRU, but can go together with this one.
  
  The test case will cover all three issues.
  
  [Test Plan]
  
  Create a jammy vm. For example, with lxd:
  
  $ lxc launch ubuntu:jammy j-nfs-1977745 --vm
  
  Then login on the sytstem:
  
  $ lxc shell j-nfs-1977745
  
  Install a kerberos kdc, utilities, and nfs server packages:
  
  # apt update && apt install -y krb5-kdc krb5-admin-server nfs-kernel-
  server
  
  Above, when prompted for the realm and KDC/ADMIN servers, use SRU for
  the realm, and "localhost" for the kdc and admin servers.
  
  Now create the realm. Choose whatever password you like, it won't be
  needed again:
  
  # krb5_newrealm
  
  Create a service principal for the nfs server. To test this bug, we will
  create a principal with a non-standard name:
  
  # kadmin.local -q "addprinc -randkey someservice/somehost at SRU"
  
  Extract it to the system keytab:
  # kadmin.local -q "ktadd someservice/somehost at SRU"
  
  Stop nfs services, just to be sure they are not running for the
  subsequent tests:
  
  # systemctl stop nfs-utils.service nfs-server.service
  
  Now the tests begin.
  
  a) In the released version, the nfs.conf(5) manpage only lists the "principal" option under "svcgssd":
  """
  svcgssd
   Recognized values: principal.
   See rpc.svcgssd(8) for details.
  """
  
  The fixed version will add to that list verbosity, rpc-verbosity, and idmap-vervosity:
  """
  svcgssd
     Recognized values: principal, verbosity, rpc-verbosity, idmap-verbosity.
  """
  
  b) Set the specific principal we created for svcgssd in /etc/nfs.conf:
  
  # nfsconf --set svcgssd principal someservice/somehost at SRU
  
  Confirm it's there:
  # grep principal /etc/nfs.conf -B 1
  [svcgssd]
  principal = someservice/somehost at SRU
  
  Run the svcgssd binary, and confirm it fails and complains about missing
  credentials:
  
  # /usr/sbin/rpc.svcgssd -f
  ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_NO_CRED (No credentials were supplied, or the credentials were unavailable or inaccessible) - No key table entry found for @SRU
  unable to obtain root (machine) credentials
  do you have a keytab entry for nfs/<your.host>@<YOUR.REALM> in /etc/krb5.keytab?
  
  In the fixed version, it will start normally:
  
  # /usr/sbin/rpc.svcgssd -f
  (no further output)
  
  c) In the error case from (b), note that it suggests to check for a
  generic principal name in the keytab, without mentioning the specific
  principal we set in nfs.conf.
  
  For the fixed version, let's change the principal name to one we really
  don't have a keytab for, and see what the error message suggests this
  time:
  
  # nfsconf --set svcgssd principal anotherservice/anotherhost at SRU
  
  This time the error specifically mentions anotherservice/anotherhost at SRU
  instead of the generic nfs/<your.host>@<YOUR.REALM>:
  
  # /usr/sbin/rpc.svcgssd -f
  ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_NO_CRED (No credentials were supplied, or the credentials were unavailable or inaccessible) - No key table entry found for anotherservice/anotherhost at SRU
  unable to obtain root (machine) credentials
  do you have a keytab entry for anotherservice/anotherhost at SRU in/etc/krb5.keytab?
  
  [Where problems could occur]
  
-  * Think about what the upload changes in the software. Imagine the change is
-    wrong or breaks something else: how would this show up?
- 
-  * It is assumed that any SRU candidate patch is well-tested before
-    upload and has a low overall risk of regression, but it's important
-    to make the effort to think about what ''could'' happen in the
-    event of a regression.
- 
-  * This must '''never''' be "None" or "Low", or entirely an argument as to why
-    your upload is low risk.
- 
-  * This both shows the SRU team that the risks have been considered,
-    and provides guidance to testers in regression-testing the SRU.
+ An NFS server is actually comprised of multiple services, specially when
+ kerberos is involved. Restarting them in the correct order is the goal
+ of the packaging, but there are some cases where this doesn't yet work
+ correctly, like bug #1971935 shows.
  
  [Other Info]
- 
-  * Anything else you think is useful to include
-  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
-  * and address these questions in advance
+ Not at this time.
  
  [Original Description]
  
  Tested on:
   Ubuntu 22.04 (x86_64)
   Package: nfs-kernel-server 1:2.6.1-1ubuntu1
  
  Set config options in /etc/nfs.conf are ignored by rpc.svcgssd (required
  for krb5 NFSv4).
  
  I was trying to set the principal name like:
  
  [svcgssd]
  principal=nfs/myhost.mydomain.de at MYDOMAIN.DE
  
  but rpc.svcgssd refused to start. When specified on command line (using
  the -p option) things started working.
  
  After having a look at the code (nfs-utils-2.6.1/utils/gssd/svcgssd.c),
  the problem seems to be
  
          /* We don't need the config anymore */
          conf_cleanup();
  
  which is called too early. So at the point where gssd_acquire_cred() is
  called the variable "principal" does no longer contain the data read
  from the config file.
  
  Moving conf_cleanup() to the end of the code helps.
  
  I also tried to get into contact with the nfs-utils developer themselves
  - but I hope someone at Ubuntu has a better way to contact them.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nfs-utils in Ubuntu.
https://bugs.launchpad.net/bugs/1977745

Title:
  nfs-utils/nfs-kernel-server (rpc.svcgssd) ignored /etc/nfs.conf
  settings

Status in nfs-utils package in Ubuntu:
  Fix Released
Status in nfs-utils source package in Jammy:
  In Progress
Status in nfs-utils source package in Kinetic:
  Fix Released

Bug description:
  [Impact]
  The "principal" option for svcgssd in /etc/nfs.conf is being ignored by the service. This breaks NFS setups that use kerberos and need a specific principal name.

  Another issue is that some svcgssd options are missing from the
  nfs.conf(5) manpage, notably verbosity, rpc-verbosity and idmap-
  verbosity, that on its own probably wouldn't warrant an SRU, but can
  go together with this one. I don't think a new bug is needed, since
  they are about svcgssd too.

  Finally, when using the "principal" option, it would be helpful if the
  error message actually used the given principal name, instead of a
  generic "nfs/<your.host>@<YOUR.REALM>" one. Again, on its own it does
  not warrant an SRU, but can go together with this one.

  The test case will cover all three issues.

  [Test Plan]

  Create a jammy vm. For example, with lxd:

  $ lxc launch ubuntu:jammy j-nfs-1977745 --vm

  Then login on the sytstem:

  $ lxc shell j-nfs-1977745

  Install a kerberos kdc, utilities, and nfs server packages:

  # apt update && apt install -y krb5-kdc krb5-admin-server nfs-kernel-
  server

  Above, when prompted for the realm and KDC/ADMIN servers, use SRU for
  the realm, and "localhost" for the kdc and admin servers.

  Now create the realm. Choose whatever password you like, it won't be
  needed again:

  # krb5_newrealm

  Create a service principal for the nfs server. To test this bug, we
  will create a principal with a non-standard name:

  # kadmin.local -q "addprinc -randkey someservice/somehost at SRU"

  Extract it to the system keytab:
  # kadmin.local -q "ktadd someservice/somehost at SRU"

  Stop nfs services, just to be sure they are not running for the
  subsequent tests:

  # systemctl stop nfs-utils.service nfs-server.service

  Now the tests begin.

  a) In the released version, the nfs.conf(5) manpage only lists the "principal" option under "svcgssd":
  """
  svcgssd
   Recognized values: principal.
   See rpc.svcgssd(8) for details.
  """

  The fixed version will add to that list verbosity, rpc-verbosity, and idmap-vervosity:
  """
  svcgssd
     Recognized values: principal, verbosity, rpc-verbosity, idmap-verbosity.
  """

  b) Set the specific principal we created for svcgssd in /etc/nfs.conf:

  # nfsconf --set svcgssd principal someservice/somehost at SRU

  Confirm it's there:
  # grep principal /etc/nfs.conf -B 1
  [svcgssd]
  principal = someservice/somehost at SRU

  Run the svcgssd binary, and confirm it fails and complains about
  missing credentials:

  # /usr/sbin/rpc.svcgssd -f
  ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_NO_CRED (No credentials were supplied, or the credentials were unavailable or inaccessible) - No key table entry found for @SRU
  unable to obtain root (machine) credentials
  do you have a keytab entry for nfs/<your.host>@<YOUR.REALM> in /etc/krb5.keytab?

  In the fixed version, it will start normally:

  # /usr/sbin/rpc.svcgssd -f
  (no further output)

  c) In the error case from (b), note that it suggests to check for a
  generic principal name in the keytab, without mentioning the specific
  principal we set in nfs.conf.

  For the fixed version, let's change the principal name to one we
  really don't have a keytab for, and see what the error message
  suggests this time:

  # nfsconf --set svcgssd principal anotherservice/anotherhost at SRU

  This time the error specifically mentions
  anotherservice/anotherhost at SRU instead of the generic
  nfs/<your.host>@<YOUR.REALM>:

  # /usr/sbin/rpc.svcgssd -f
  ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_NO_CRED (No credentials were supplied, or the credentials were unavailable or inaccessible) - No key table entry found for anotherservice/anotherhost at SRU
  unable to obtain root (machine) credentials
  do you have a keytab entry for anotherservice/anotherhost at SRU in/etc/krb5.keytab?

  [Where problems could occur]

  An NFS server is actually comprised of multiple services, specially
  when kerberos is involved. Restarting them in the correct order is the
  goal of the packaging, but there are some cases where this doesn't yet
  work correctly, like bug #1971935 shows.

  [Other Info]
  Not at this time.

  [Original Description]

  Tested on:
   Ubuntu 22.04 (x86_64)
   Package: nfs-kernel-server 1:2.6.1-1ubuntu1

  Set config options in /etc/nfs.conf are ignored by rpc.svcgssd
  (required for krb5 NFSv4).

  I was trying to set the principal name like:

  [svcgssd]
  principal=nfs/myhost.mydomain.de at MYDOMAIN.DE

  but rpc.svcgssd refused to start. When specified on command line
  (using the -p option) things started working.

  After having a look at the code (nfs-
  utils-2.6.1/utils/gssd/svcgssd.c), the problem seems to be

          /* We don't need the config anymore */
          conf_cleanup();

  which is called too early. So at the point where gssd_acquire_cred()
  is called the variable "principal" does no longer contain the data
  read from the config file.

  Moving conf_cleanup() to the end of the code helps.

  I also tried to get into contact with the nfs-utils developer
  themselves - but I hope someone at Ubuntu has a better way to contact
  them.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1977745/+subscriptions