[Bug 1811580] Re: systemd fails to start sshd at reboot

[Dimitri John Ledkov]

On Tue, 26 Feb 2019 at 18:05, Matt P <1811580 at bugs.launchpad.net> wrote:
>
> Okay.  I guess I would have expected that if there was a dependency on a
> specific kernel version, that I wouldn't be able to install a package
> that wasn't compatible and breaks the system by installing a security
> update.  It would be preferable to be informed there is a security

The kernel you are running is not an Ubuntu one. There is no package
for it known to either apt nor dpkg, thus there is no possible
dependency we could express.
How can we express a dependency, on something that is unknown to us?
After all, one can prepare installs using chroots, to then later run
the system on an incompatible kernel.

> update but that I can't install it because I am running an out of date
> kernel...then I know I am insecure and that the kernel is the issue.

Escalate to your provider. Who is your provider? Maybe Canonical can
get in touch with them?

> But I guess that is a topic for the package management guys.  The error
> message from systemd-tmpfiles about too many symlinks isn't particularly
> helpful either since in this case the problem (apparently) has nothing
> to do with symlinks but rather filesystem apis in the old kernel (I
> guess?).
>
> Yes of course I can contact the hosting provider and ask them to provide
> an updated kernel and the likely result may be that I just have to use
> an alternate provider if I want this to work.  Perhaps I should anyway
> since the hosting provider having such old kernels isn't a good sign.
>
> I also saw this comment:
> https://github.com/systemd/systemd/commit/6a89d671dfdd92c0b1b703d7fcb5b0551cafb570
>
> For now I have worked around this issue by just updating the paths to
> point to /run instead of /var/run so systemd-tmpfiles doesn't barf on
> the symlinks.
>
>     sed -i -e 's;/var/run;/run;g' /usr/lib/tmpfiles.d/*.conf

I wonder if we should SRU that. Which might not help for all instances
of this issue, but maybe at least some.

-- 
Regards,

Dimitri.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1811580

Title:
  systemd fails to start sshd at reboot

Status in systemd package in Ubuntu:
  Incomplete

Bug description:
  So far reported issues turned out to be:
  - obsolete/buggy/vulnerable 3rd party provided kernels
  - bad permissions on /

  Please ensure / is owned by root:root.
  Please ensure you are running up to date kernels.

  
  ===

  Ubuntu 16.04.5, systemd 229-4ubuntu21.15

  The latest systemd update has somehow changed the method it uses to
  start 'ssh.service' i.e. 'sshd'. systemd fails to start sshd if
  /etc/ssh/sshd_config contains "UsePrivilegeSeparation yes" and
  /var/run/sshd/ does not already exist. Being as this is the default,
  virtually EVERY Ubuntu 16.04 server in the world has
  UsePrivilegeSeparation set to yes. Furthermore, at the time when the
  user performs 'apt upgrade' and receives the newest version of
  systemd, /var/run/sshd/ already exists, so sshd successfully reloads
  for as long as the server doesn't get rebooted. BUT, as soon as the
  server is rebooted for any reason, /var/run/sshd/ gets cleaned away,
  and sshd fails to start, causing the remote user to be completely
  locked out of his system. This is a MAJOR issue for millions of VPS
  servers worldwide, as they are all about to get locked out of their
  servers and potentially lose data. The next reboot is a ticking time
  bomb waiting to spring. The bomb can be defused by implicitly setting
  'UsePrivilegeSeparation no' in /etc/ssh/sshd_config, however
  unsuspecting administrators are bound to be caught out by the
  millions. I got caught by it in the middle of setting up a new server
  yesterday, and it took a whole day to find the source.

  The appropriate fix would be to ensure that systemd can successfully
  'start ssh.service' even when 'UsePrivilegeSeparation yes' is set.
  systemd needs to test that /var/run/sshd/ exists before starting sshd,
  just as the init.d script for sshd does. openssl could also be patched
  so that UsePrivilegeSeparation is no longer enabled by default,
  however that is not going to solve the problem for millions of pre-
  existing config files. Only an update to openssl to force-override
  that flag to 'no' would solve the problem. Thus systemd still needs to
  be responsible for ensuring that it inits sshd properly by ensuring
  that /var/run/sshd/ exists before it sends the 'start' command.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1811580/+subscriptions