[Bug 1905285] Re: socket-activated sshd breaks on concurrent connections
Brian Murray
1905285 at bugs.launchpad.net
Fri Jul 30 20:42:19 UTC 2021
Hello Marcin, or anyone else affected,
Accepted openssh into focal-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/openssh/1:8.2p1-4ubuntu0.3 in a few
hours, and then in the -proposed repository.
Please help us by testing this new package. See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed. Your feedback will aid us getting this
update out to other Ubuntu users.
If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
focal to verification-done-focal. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-focal. In either case, without details of your testing we will
not be able to proceed.
Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
advance for helping!
N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.
** Changed in: openssh (Ubuntu Focal)
Status: In Progress => Fix Committed
** Tags added: verification-needed-focal
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/1905285
Title:
socket-activated sshd breaks on concurrent connections
Status in openssh package in Ubuntu:
Fix Released
Status in openssh source package in Focal:
Fix Committed
Status in openssh source package in Hirsute:
Fix Committed
Bug description:
[Impact]
Users of the systemd socket activated ssh service may experience a
race condition that may lead an ssh instance to fail.
The race condition happens when, for a running socket activated ssh
service,
an instance A is started, creating the RuntimeDirectory for the
service; then
an instance B is started, relying on the RuntimeDirectory created for
instance A; then
instance A halts, causing the RuntimeDirectory to be deleted.
If, at this point, instance B has not chrooted into RuntimeDirectory
yet, then instance B will fail.
The proposed patch fixes the issue by preserving the RuntimeDirectory
after an instance A of the socket activated ssh service halts.
[Test Plan]
1) Stop any running instances of ssh.
`systemctl stop ssh`
2) Start the socket activated ssh service.
`systemctl start ssh.socket`
3) Verify that no errors related to ssh were logged in /var/log/auth.log
`cat /var/log/auth.log | grep 'sshd.*fatal.*chroot.*No such file or directory'`
4) perform several ssh connections to the running server in a short time span. ssh-keyscan may help here.
`ssh-keyscan localhost`
5) Verify that errors related to ssh were logged in /var/log/auth.log
`cat /var/log/auth.log | grep 'sshd.*fatal.*chroot.*No such file or directory'`
6) Apply the proposed fix (make sure the socket activated service is
restarted)
7) repead step (4), then verify that no new entries were appended to
the step (5) output
[Where problems could occur]
If the changes to the socket activated unit file are wrong, the socket
activated service may fail to start after the package upgrade. In this
case, we would need to instruct users to perform local changes to the
unit file with possible additional fixes while a new version of the
patch lands.
[racb] There might be cases where users are inadvertently depending on
the cleanup that will now be disabled - for example by a bug or
misconfiguration that would result in /run filling up otherwise. By
disabling systemd cleanup and relying solely on openssh for cleanup,
such a bug or misconfiguration may be exposed and cause problems on
such systems.
[Other Info]
This fix has been forwarded to Debian and accepted in
https://salsa.debian.org/ssh-team/openssh/-/merge_requests/12
[Original message]
This is mostly the same issue as https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=934663.
With the default configuration of openssh-server and systemd, sshd
will complain and crash when multiple connections are made and
terminated in a quick succession, e.g. with `ssh-keyscan`. It results
in the following errors in /var/log/auth.log:
```
Nov 22 20:53:34 {host} sshd[14567]: Unable to negotiate with {client} port 41460: no matching host key type found. Their offer: sk-ecdsa-sha2-nistp256 at openssh.com [preauth]
Nov 22 20:53:34 {host} sshd[14570]: fatal: chroot("/run/sshd"): No such file or directory [preauth]
Nov 22 20:53:34 {host} sshd[14569]: fatal: chroot("/run/sshd"): No such file or directory [preauth]
Nov 22 20:53:34 {host} sshd[14568]: fatal: chroot("/run/sshd"): No such file or directory [preauth]
Nov 22 20:53:34 {host} sshd[14566]: fatal: chroot("/run/sshd"): No such file or directory [preauth]
Nov 22 20:53:47 {host} sshd[14584]: Connection closed by {client} port 59312 [preauth]
Nov 22 20:53:47 {host} sshd[14586]: fatal: chroot("/run/sshd"): No such file or directory [preauth]
Nov 22 20:53:48 {host} sshd[14585]: fatal: chroot("/run/sshd"): No such file or directory [preauth]
```
as well as e.g. missing responses in ssh-keyscan:
```
$ ssh-keyscan -vvv {host}
debug2: fd 3 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 2
debug2: fd 4 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 4
debug2: fd 5 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 8
debug2: fd 6 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 32
debug2: fd 7 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 64
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256 at libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: sk-ecdsa-sha2-nistp256 at openssh.com
debug2: ciphers ctos: chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com
debug2: ciphers stoc: chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com
debug2: MACs ctos: umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib at openssh.com
debug2: compression stoc: none,zlib at openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256 at libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com
debug2: ciphers stoc: chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com
debug2: MACs ctos: umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib at openssh.com
debug2: compression stoc: none,zlib at openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: (no match)
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
```
The error is most likely caused by a race condition on removing /run/sshd, which is easily reproducible by ssh-keyscan.
I noticed that depeding on client, I'd sometimes miss all keys, sometimes get one, sometimes more.
Modifying the following files (they should me marked as modified in the bug report) seems to solve the issue, at least temporarily:
/usr/lib/systemd/system/ssh.service
/usr/lib/systemd/system/ssh at .service
In both cases, I added `RuntimeDirectoryPreserve=yes` to the `[Service]` section, after `RuntimeDirectory=sshd`.
This is the same solution mentioned in the Debian bug, although their bug report doesn't mention which service files are affected.
This doesn't seem to be a proper long-term solution though, as it seems apt doesn't respect configuration files in /usr (or they are unlisted somewhere),
because after upgrading system just before filing this bug report, the files got overwritten and reverted to their original form.
I only got asked about the /etc/ssh/sshd_config, which I have chosen to keep.
ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: openssh-server 1:8.2p1-4ubuntu0.1 [modified: lib/systemd/system/ssh.service lib/systemd/system/ssh at .service]
ProcVersionSignature: Ubuntu 5.4.0-54.60-generic 5.4.65
Uname: Linux 5.4.0-54-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.12
Architecture: amd64
CasperMD5CheckResult: skip
Date: Mon Nov 23 15:09:32 2020
SourcePackage: openssh
UpgradeStatus: No upgrade log present (probably fresh install)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1905285/+subscriptions
More information about the foundations-bugs
mailing list