[Bug 1905285] Re: socket-activated sshd breaks on concurrent connections

Thu Nov 26 21:56:05 UTC 2020

Thanks for the comment, Marcin.  Yes, you're right, the correct file to
edit was ssh at .service indeed.  That was a thinko on my part.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/1905285

Title:
  socket-activated sshd breaks on concurrent connections

Status in openssh package in Ubuntu:
  Triaged

Bug description:
  This is mostly the same issue as https://bugs.debian.org/cgi-
  bin/bugreport.cgi?bug=934663.

  With the default configuration of openssh-server and systemd, sshd
  will complain and crash when multiple connections are made and
  terminated in a quick succession, e.g. with `ssh-keyscan`. It results
  in the following errors in /var/log/auth.log:

  ```
  Nov 22 20:53:34 {host} sshd[14567]: Unable to negotiate with {client} port 41460: no matching host key type found. Their offer: sk-ecdsa-sha2-nistp256 at openssh.com [preauth]
  Nov 22 20:53:34 {host} sshd[14570]: fatal: chroot("/run/sshd"): No such file or directory [preauth]
  Nov 22 20:53:34 {host} sshd[14569]: fatal: chroot("/run/sshd"): No such file or directory [preauth]
  Nov 22 20:53:34 {host} sshd[14568]: fatal: chroot("/run/sshd"): No such file or directory [preauth]
  Nov 22 20:53:34 {host} sshd[14566]: fatal: chroot("/run/sshd"): No such file or directory [preauth]
  Nov 22 20:53:47 {host} sshd[14584]: Connection closed by {client} port 59312 [preauth]
  Nov 22 20:53:47 {host} sshd[14586]: fatal: chroot("/run/sshd"): No such file or directory [preauth]
  Nov 22 20:53:48 {host} sshd[14585]: fatal: chroot("/run/sshd"): No such file or directory [preauth]
  ```

  as well as e.g. missing responses in ssh-keyscan:

  ```
  $ ssh-keyscan -vvv {host}
  debug2: fd 3 setting O_NONBLOCK
  debug3: conalloc: oname {host} kt 2
  debug2: fd 4 setting O_NONBLOCK
  debug3: conalloc: oname {host} kt 4
  debug2: fd 5 setting O_NONBLOCK
  debug3: conalloc: oname {host} kt 8
  debug2: fd 6 setting O_NONBLOCK
  debug3: conalloc: oname {host} kt 32
  debug2: fd 7 setting O_NONBLOCK
  debug3: conalloc: oname {host} kt 64
  debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
  # {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
  debug3: send packet: type 20
  debug1: SSH2_MSG_KEXINIT sent
  debug3: receive packet: type 20
  debug1: SSH2_MSG_KEXINIT received
  debug2: local client KEXINIT proposal
  debug2: KEX algorithms: curve25519-sha256,curve25519-sha256 at libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
  debug2: host key algorithms: sk-ecdsa-sha2-nistp256 at openssh.com
  debug2: ciphers ctos: chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com
  debug2: ciphers stoc: chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com
  debug2: MACs ctos: umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
  debug2: MACs stoc: umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
  debug2: compression ctos: none,zlib at openssh.com
  debug2: compression stoc: none,zlib at openssh.com
  debug2: languages ctos:
  debug2: languages stoc:
  debug2: first_kex_follows 0
  debug2: reserved 0
  debug2: peer server KEXINIT proposal
  debug2: KEX algorithms: curve25519-sha256,curve25519-sha256 at libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
  debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
  debug2: ciphers ctos: chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com
  debug2: ciphers stoc: chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com
  debug2: MACs ctos: umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
  debug2: MACs stoc: umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
  debug2: compression ctos: none,zlib at openssh.com
  debug2: compression stoc: none,zlib at openssh.com
  debug2: languages ctos:
  debug2: languages stoc:
  debug2: first_kex_follows 0
  debug2: reserved 0
  debug1: kex: algorithm: curve25519-sha256
  debug1: kex: host key algorithm: (no match)
  debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
  # {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
  debug3: send packet: type 20
  debug1: SSH2_MSG_KEXINIT sent
  debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
  # {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
  debug3: send packet: type 20
  debug1: SSH2_MSG_KEXINIT sent
  debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
  # {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
  debug3: send packet: type 20
  debug1: SSH2_MSG_KEXINIT sent
  debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
  # {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
  debug3: send packet: type 20
  debug1: SSH2_MSG_KEXINIT sent
  ```

  The error is most likely caused by a race condition on removing /run/sshd, which is easily reproducible by ssh-keyscan.
  I noticed that depeding on client, I'd sometimes miss all keys, sometimes get one, sometimes more.

  Modifying the following files (they should me marked as modified in the bug report) seems to solve the issue, at least temporarily:
  /usr/lib/systemd/system/ssh.service
  /usr/lib/systemd/system/ssh at .service
  In both cases, I added `RuntimeDirectoryPreserve=yes` to the `[Service]` section, after `RuntimeDirectory=sshd`.
  This is the same solution mentioned in the Debian bug, although their bug report doesn't mention which service files are affected.

  This doesn't seem to be a proper long-term solution though, as it seems apt doesn't respect configuration files in /usr (or they are unlisted somewhere),
  because after upgrading system just before filing this bug report, the files got overwritten and reverted to their original form.
  I only got asked about the /etc/ssh/sshd_config, which I have chosen to keep.

  ProblemType: Bug
  DistroRelease: Ubuntu 20.04
  Package: openssh-server 1:8.2p1-4ubuntu0.1 [modified: lib/systemd/system/ssh.service lib/systemd/system/ssh at .service]
  ProcVersionSignature: Ubuntu 5.4.0-54.60-generic 5.4.65
  Uname: Linux 5.4.0-54-generic x86_64
  ApportVersion: 2.20.11-0ubuntu27.12
  Architecture: amd64
  CasperMD5CheckResult: skip
  Date: Mon Nov 23 15:09:32 2020
  SourcePackage: openssh
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1905285/+subscriptions