[Bug 1723350] Re: sssd offline on boot, stays offline forever

Andreas Hasenack andreas at canonical.com
Mon May 18 17:49:48 UTC 2020


** Description changed:

  [Impact]
  sssd can switch to an offline mode of operation when it cannot reach the authentication or id backend. It uses several methods to assess the situation, and one of them is monitoring the /etc/resolv.conf file for changes.
  
  In ubuntu that file is a symlink to /run/systemd/resolve/stub-
  resolv.conf, but the target doesn't exist at all times during boot. It's
  expected that symlink to be broken for a while during boot.
  
  Turns out that the monitoring that sssd was doing on /etc/resolv.conf
  didn't take into consideration that what could change was the *target*
  of the symlink. it completely ignored that fact, and didn't notice when
  the resolv.conf contents actually changed in this scenario, which
  resulted in sssd staying in the offline mode when it shouldn't.
  
  There are two fixes being pulled in for this SRU:
  a) fix the monitoring of the target of the /etc/resolv.conf symlink
  b) change the fallback polling code to keep trying, instead of giving up right away
  
  [Test Case]
  It's recommended to test this in a lxd container, or a vm.
  
- Preparation steps:
+ Preparation steps. When prompted for an openldap/slapd password, chose any password you want. It won't be needed again:
  $ sudo apt update && sudo apt dist-upgrade
  $ sudo apt install sssd-ldap sssd-tools sssd-dbus slapd dnsmasq
  
  Become root:
  $ sudo su -
  
  Detect your ip:
  # export interface=$(ip route | grep default | sed -r 's,^default via .* dev ([a-z0-9]+) .*,\1,')
  # export ip=$(ip addr show dev $interface | grep "inet [0-9]" | awk '{print $2}' | cut -d / -f 1)
  
  Confirm the $ip variable is correct for your case:
  # echo $ip
  
  Create /etc/dnsmasq.d/sssd-test.conf using your real ip:
  # cat > /etc/dnsmasq.d/sssd-test.conf <<EOF
  host-record=ldap01.example.com,$ip
  listen-address=$ip
  EOF
  
  restart dnsmasq
  # systemctl restart dnsmasq
  
  a) inotify test
  Create /etc/sssd/sssd.conf:
  # cat > /etc/sssd/sssd.conf <<EOF
  [sssd]
  config_file_version = 2
  services = nss, pam, ifp
  domains = LDAP
  debug_level = 6
  
  [domain/LDAP]
  id_provider = ldap
  ldap_uri = ldap://ldap01.example.com
  cache_credentials = True
  ldap_search_base = dc=example,dc=com
  EOF
  
  # chmod 0600 /etc/sssd/sssd.conf
  
  # rm /etc/resolv.conf
  # ln -s /etc/resolv.conf.target /etc/resolv.conf
  
  create good resolv.conf:
  # echo "nameserver $ip" > /etc/resolv.conf.good
  
  Confirm /etc/resolv.conf is a broken symlink:
  # ll /etc/resolv.conf*
  lrwxrwxrwx 1 root root 23 May 13 20:48 /etc/resolv.conf -> /etc/resolv.conf.target
  -rw-r--r-- 1 root root 24 May 13 20:48 /etc/resolv.conf.good
  
  Open another terminal/screen and tail the sssd logs with a grep:
  # tail -f /var/log/sssd/sssd.log | grep resolv
  
  Start sssd
  # systemctl restart sssd
  
  The tail output from that other terminal should say sssd is monitoring /etc/resolv.conf (that's the broken symlink):
  (Mon May 18 17:32:34 2020) [sssd] [_snotify_create] (0x0400): Added a watch for /etc/resolv.conf with inotify flags 0x8D88 internal flags 0x1 using function resolv_conf_inotify_cb after delay 1.0
  
- 
  Repeat this sssctl call until it shows the offline mode persistently:
  # sssctl domain-status LDAP
  Online status: Offline
  
  Active servers:
  LDAP: not connected
  
  Discovered LDAP servers:
  - ldap01.example.com
  
  "Unbreak" the symlink:
  # cp /etc/resolv.conf.good /etc/resolv.conf.target
  
  The log should now say:
  (Mon May 18 17:33:30 2020) [sssd] [process_dir_event] (0x0400): Not interested in resolv.conf.target
  
  This shows that sssd didn't pick up that resolv.conf changed, via the
  target of the symlink.
  
  Run sssctl again, and the online status will remain offline.
  
- 
  With the fixed packages, the sssd startup log will say:
  (Mon May 18 17:17:06 2020) [sssd] [_snotify_create] (0x0400): Added a watch for /etc/resolv.conf.target with inotify flags 0x8D88 internal flags 0x1 using function resolv_conf_inotify_cb after delay 1.0
  
  Showing that it's monitoring the symlink target.
  
  And after fixing the broken symlink, it will say:
  (Mon May 18 17:18:06 2020) [sssd] [process_dir_event] (0x0400): received notification for watched file [resolv.conf.target] under /etc
  
  Run sssctl again, it should almost immediately switch to online:
  # sssctl domain-status LDAP
  Online status: Online
  
  Active servers:
  LDAP: ldap01.example.com
  
  Discovered LDAP servers:
  - ldap01.example.com
- 
  
  b) polling test
  Repeat the previous test, but with "try_inotify = false" in sssd.conf, like this:
  # cat > /etc/sssd/sssd.conf <<EOF
  [sssd]
  config_file_version = 2
  services = nss, pam, ifp
  domains = LDAP
  #debug_level = 6
  try_inotify = false
  
  [domain/LDAP]
  id_provider = ldap
  ldap_uri = ldap://ldap01.example.com
  cache_credentials = True
  ldap_search_base = dc=example,dc=com
  EOF
  
  After unbreaking the symbolic link, in a few seconds (5s at most),
  sssctl should show the service as being online, if using the fixed
  packages.
  
  [Regression Potential]
  TBD
  
  [Other Info]
  Not at this time.
  
  [Original Description]
  
  SSSD 1.15.3-2ubuntu1 on 17.10/artful (previous versions on artful were
  also affected) is offline on boot and seems to stay offline forever (I
  waited over 20 minutes).
  
  sssd_nss.log:
  (Fri Oct 13 09:49:50 2017) [sssd[nss]] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.sssd.Error.DataProvider.Offline]
  (Fri Oct 13 09:49:51 2017) [sssd[nss]] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.sssd.Error.DataProvider.Offline]
  (Fri Oct 13 09:49:51 2017) [sssd[nss]] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.sssd.Error.DataProvider.Offline]
  (Fri Oct 13 09:49:51 2017) [sssd[nss]] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.sssd.Error.DataProvider.Offline]
  ...
  
  SSSD immediately returns to normal operation after restarting it or
  after sending SIGUSR2.
  
  A workaround for the problem is creating the file /etc/systemd/system/sssd.service.d/override.conf with contents
  [Unit]
  Requires=network-online.target
  After=network-online.target

-- 
You received this bug notification because you are a member of Ubuntu
Server, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1723350

Title:
  sssd offline on boot, stays offline forever

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1723350/+subscriptions



More information about the Ubuntu-server-bugs mailing list