[Bug 1811580] Re: systemd fails to start sshd at reboot

Tue Jan 15 17:47:27 UTC 2019

@Dimitri John Ledkov (xnox)

nvidia at tegra-ubuntu:~$ mount
/dev/mmcblk0p1 on / type ext4 (rw,relatime,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=7976720k,nr_inodes=1994180,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/debug type cgroup (rw,nosuid,nodev,noexec,relatime,debug)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/sda1 on /media type ext4 (rw,relatime,data=ordered)
tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=804352k,mode=700,uid=1001,gid=1001)
/dev/sda1 on /media/docker/overlay2 type ext4 (rw,relatime,data=ordered)

nvidia at tegra-ubuntu:/$ ls -la
total 108
drwxrwxr-x  23 ubuntu ubuntu  4096 Oct 30 17:36 .
drwxrwxr-x  23 ubuntu ubuntu  4096 Oct 30 17:36 ..
drwxr-xr-x   2 root   root    4096 Jan 12 00:17 bin
drwxr-xr-x   5 root   root    4096 Oct 13 00:40 boot
drwxr-xr-x   4 root   root    4096 Jan  3 23:00 data
drwxr-xr-x  14 root   root    5480 Jan 15 01:14 dev
drwxr-xr-x 141 root   root   12288 Jan 15 17:41 etc
drwxr-xr-x   4 root   root    4096 Jan  6  2017 home
drwxrwxr-x  22 ubuntu ubuntu  4096 Oct 29 20:31 lib
drwx------   2 root   root   16384 Oct 12 20:30 lost+found
drwxr-xr-x   5 nvidia nvidia  4096 Oct 29 20:36 media
drwxr-xr-x   2 root   root    4096 Apr 20  2016 mnt
drwxr-xr-x   3 root   root    4096 Oct 12 20:28 opt
dr-xr-xr-x 313 root   root       0 Jan  1  1970 proc
-rw-r--r--   1 ubuntu ubuntu    62 May 17  2018 README.txt
drwx------   4 root   root    4096 Jan  2 23:47 root
drwxr-xr-x  23 root   root     820 Jan 15 17:41 run
drwxr-xr-x   2 root   root   12288 Jan 12 00:17 sbin
drwxr-xr-x   2 root   root    4096 Apr 19  2016 snap
drwxr-xr-x   2 root   root    4096 Apr 20  2016 srv
dr-xr-xr-x  12 root   root       0 Jan  1  2000 sys
drwxrwxrwt   7 root   root    4096 Jan 15 17:41 tmp
drwxr-xr-x  12 root   root    4096 Oct 12 21:45 usr
drwxr-xr-x  19 root   root    4096 Oct 12 20:51 var

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1811580

Title:
  systemd fails to start sshd at reboot

Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  Ubuntu 16.04.5, systemd 229-4ubuntu21.15

  The latest systemd update has somehow changed the method it uses to
  start 'ssh.service' i.e. 'sshd'. systemd fails to start sshd if
  /etc/ssh/sshd_config contains "UsePrivilegeSeparation yes" and
  /var/run/sshd/ does not already exist. Being as this is the default,
  virtually EVERY Ubuntu 16.04 server in the world has
  UsePrivilegeSeparation set to yes. Furthermore, at the time when the
  user performs 'apt upgrade' and receives the newest version of
  systemd, /var/run/sshd/ already exists, so sshd successfully reloads
  for as long as the server doesn't get rebooted. BUT, as soon as the
  server is rebooted for any reason, /var/run/sshd/ gets cleaned away,
  and sshd fails to start, causing the remote user to be completely
  locked out of his system. This is a MAJOR issue for millions of VPS
  servers worldwide, as they are all about to get locked out of their
  servers and potentially lose data. The next reboot is a ticking time
  bomb waiting to spring. The bomb can be defused by implicitly setting
  'UsePrivilegeSeparation no' in /etc/ssh/sshd_config, however
  unsuspecting administrators are bound to be caught out by the
  millions. I got caught by it in the middle of setting up a new server
  yesterday, and it took a whole day to find the source.

  The appropriate fix would be to ensure that systemd can successfully
  'start ssh.service' even when 'UsePrivilegeSeparation yes' is set.
  systemd needs to test that /var/run/sshd/ exists before starting sshd,
  just as the init.d script for sshd does. openssl could also be patched
  so that UsePrivilegeSeparation is no longer enabled by default,
  however that is not going to solve the problem for millions of pre-
  existing config files. Only an update to openssl to force-override
  that flag to 'no' would solve the problem. Thus systemd still needs to
  be responsible for ensuring that it inits sshd properly by ensuring
  that /var/run/sshd/ exists before it sends the 'start' command.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1811580/+subscriptions