[Bug 1686618] Re: ssh connection attempts fail if hw crypto support on s390x is enabled on 17.04
Frank Heimes
1686618 at bugs.launchpad.net
Fri May 5 09:18:34 UTC 2017
ubuntu at zlin42:~$ sudo sh -c "echo 'deb http://ports.ubuntu.com/ubuntu-ports $(lsb_release -sc)-proposed restricted main multiverse universe' >> /etc/apt/sources.list.d/proposed-repositories.list"
ubuntu at zlin42:~$ sudo apt -y update -qq
12 packages can be upgraded. Run 'apt list --upgradable' to see them.
ubuntu at zlin42:~$ apt list --upgradable
Listing... Done
linux-firmware/zesty-proposed 1.164.1 all [upgradable from: 1.164]
linux-generic/zesty-proposed 4.10.0.21.23 s390x [upgradable from: 4.10.0.20.22]
linux-headers-generic/zesty-proposed 4.10.0.21.23 s390x [upgradable from: 4.10.0.20.22]
linux-image-generic/zesty-proposed 4.10.0.21.23 s390x [upgradable from: 4.10.0.20.22]
linux-libc-dev/zesty-proposed 4.10.0-21.23 s390x [upgradable from: 4.10.0-20.22]
openssh-client/zesty-proposed 1:7.4p1-10ubuntu0.1 s390x [upgradable from: 1:7.4p1-10]
openssh-server/zesty-proposed 1:7.4p1-10ubuntu0.1 s390x [upgradable from: 1:7.4p1-10]
openssh-sftp-server/zesty-proposed 1:7.4p1-10ubuntu0.1 s390x [upgradable from: 1:7.4p1-10]
snap-confine/zesty-proposed 2.25+17.04 s390x [upgradable from: 2.24.1+17.04]
snapd/zesty-proposed 2.25+17.04 s390x [upgradable from: 2.24.1+17.04]
sosreport/zesty-proposed 3.4-1~ubuntu17.04.1 s390x [upgradable from: 3.3+git50-g3c0349b-2]
unattended-upgrades/zesty-proposed 0.93.1ubuntu2.1 all [upgradable from: 0.93.1ubuntu2]
ubuntu at zlin42:~$
###
ubuntu at zlin42:~$ sudo vi /etc/ssh/sshd_config
ubuntu at zlin42:~$ sudo systemctl restart sshd
ubuntu at zlin42:~$ apt-cache policy openssh-server
openssh-server:
Installed: 1:7.4p1-10
Candidate: 1:7.4p1-10ubuntu0.1
Version table:
1:7.4p1-10ubuntu0.1 500
500 http://ports.ubuntu.com/ubuntu-ports zesty-proposed/main s390x Packages
*** 1:7.4p1-10 500
500 http://us.ports.ubuntu.com/ubuntu-ports zesty/main s390x Packages
100 /var/lib/dpkg/status
ubuntu at zlin42:~$
me at WS:~$ ssh ubuntu at zlin42
ubuntu at zlin42's password:
Welcome to Ubuntu 17.04 (GNU/Linux 4.10.0-20-generic s390x)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
0 packages can be updated.
0 updates are security updates.
Last login: Fri May 5 03:22:00 2017 from 10.172.66.66
ubuntu at zlin42:~$ exit
logout
Connection to zlin42 closed.
me at WS:~$
### activate hw crypto for ssl / ibmca engine
ubuntu at zlin42:~$ sudo vi /etc/ssl/openssl.cnf
# set: openssl_conf = openssl_def
ubuntu at zlin42:~$ openssl engine
(dynamic) Dynamic engine loading support
(ibmca) Ibmca hardware engine support
ubuntu at zlin42:~$
### negative test - expecting the problem to occur
me at WS:~$ ssh ubuntu at zlin42
ubuntu at zlin42's password:
Connection to zlin42 closed by remote host.
Connection to zlin42 closed.
me at WS:~$
ubuntu at zlin42:~$ sudo apt install openssh-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
openssh-client openssh-sftp-server
Suggested packages:
keychain libpam-ssh monkeysphere ssh-askpass molly-guard rssh
The following packages will be upgraded:
openssh-client openssh-server openssh-sftp-server
3 upgraded, 0 newly installed, 0 to remove and 9 not upgraded.
Need to get 928 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://ports.ubuntu.com/ubuntu-ports zesty-proposed/main s390x openssh-sftp-server s390x 1:7.4p1-10ubuntu0.1 [38.0 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports zesty-proposed/main s390x openssh-server s390x 1:7.4p1-10ubuntu0.1 [316 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports zesty-proposed/main s390x openssh-client s390x 1:7.4p1-10ubuntu0.1 [574 kB]
Fetched 928 kB in 1s (722 kB/s)
Preconfiguring packages ...
(Reading database ... 134327 files and directories currently installed.)
Preparing to unpack .../openssh-sftp-server_1%3a7.4p1-10ubuntu0.1_s390x.deb ...
Unpacking openssh-sftp-server (1:7.4p1-10ubuntu0.1) over (1:7.4p1-10) ...
Preparing to unpack .../openssh-server_1%3a7.4p1-10ubuntu0.1_s390x.deb ...
Unpacking openssh-server (1:7.4p1-10ubuntu0.1) over (1:7.4p1-10) ...
Preparing to unpack .../openssh-client_1%3a7.4p1-10ubuntu0.1_s390x.deb ...
Unpacking openssh-client (1:7.4p1-10ubuntu0.1) over (1:7.4p1-10) ...
Processing triggers for ufw (0.35-4) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (232-21ubuntu3) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up openssh-client (1:7.4p1-10ubuntu0.1) ...
Setting up openssh-sftp-server (1:7.4p1-10ubuntu0.1) ...
Setting up openssh-server (1:7.4p1-10ubuntu0.1) ...
ubuntu at zlin42:~$
ubuntu at zlin42:~$ exit
logout
Connection to zlin42 closed.
me at WS:~$ ssh ubuntu at zlin42
ubuntu at zlin42's password:
Connection to zlin42 closed by remote host.
Connection to zlin42 closed.
me at WS:~$ ssh ubuntu at zlin42
ubuntu at zlin42's password:
Connection to zlin42 closed by remote host.
Connection to zlin42 closed.
me at WS:~$
ubuntu at zlin42:~$ sudo systemctl restart sshd
### positive test, expecting the problem to be solved
me at WS:~$ ssh ubuntu at zlin42
ubuntu at zlin42's password:
Connection to zlin42 closed by remote host.
Connection to zlin42 closed.
me at WS:~$
### test/verification failed! - problem still exists
ubuntu at zlin42:~$ sudo apt list openssh-server
Listing... Done
openssh-server/zesty-proposed,now 1:7.4p1-10ubuntu0.1 s390x [installed]
N: There is 1 additional version. Please use the '-a' switch to see it
ubuntu at zlin42:~$
ubuntu at zlin42:~$ sudo apt -a list openssh-server
Listing... Done
openssh-server/zesty-proposed,now 1:7.4p1-10ubuntu0.1 s390x [installed]
openssh-server/zesty 1:7.4p1-10 s390x
ubuntu at zlin42:~$
---
https://launchpad.net/ubuntu/+source/openssh/1:7.4p1-10ubuntu0.1
---
The workaround with:
ubuntu at s1lp15:~$ cat /etc/ssh/sshd_config | grep -i ^UsePrivilegeSeparation
UsePrivilegeSeparation yes
still works ...
Some further investigations needed ...
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/1686618
Title:
ssh connection attempts fail if hw crypto support on s390x is enabled
on 17.04
Status in Ubuntu on IBM z Systems:
Fix Committed
Status in openssh package in Ubuntu:
Fix Released
Status in openssh source package in Zesty:
Fix Committed
Status in openssh source package in Artful:
Fix Released
Bug description:
[ Impact ]
* Unable to ssh into Ubuntu, using default sshd configuration, when hw
acceleration is enabled in openssl.
[ Proposed solution ]
* Cherrypick upstream fixes for:
- sandboxing code on big endian
- allowing hw accel iocls in the sandbox
short:
after investigations the following commits are needed by openssh-server version 7.4p1 that is part of 17.04:
- 5f1596e11d55539678c41f68aed358628d33d86f
- 9e96b41682aed793fadbea5ccd472f862179fb02
on master branch in https://github.com/openssh/openssh-portable
that belong to openssh 7.5 release notes statement: "sshd(8): Avoid sandbox errors for Linux S390 systems using an ICA crypto coprocessor."
__________
[Test case]
long:
enable z hw crypto support for openssh on an Ubuntu host (zlin42) on s390x like this:
sudo apt-get install openssl-ibmca libica-utils libica2
sudo tee -a /etc/ssl/openssl.cnf < /usr/share/doc/openssl-ibmca/examples/openssl.cnf.sample
sudo sed -i 's/^\(openssl_conf = openssl_def.*$\)/# \1/g' /etc/ssl/openssl.cnf
sudo sed -i '10i openssl_cnf = openssl_def' /etc/ssl/openssl.cnf
afterwards ssh login attempts fail:
$ ssh ubuntu at zlin42
ubuntu at zlin42's password:
Connection to zlin42 closed by remote host.
Connection to zlin42 closed.
the normal logs don't provide any interesting details:
mit log:
Apr 24 12:37:52 zlin42 kernel: [933567.994312] audit: type=1326 audit(1493051872.112:29): auid=4294967295 uid=107 gid=65534 ses=4294967295 pid=25105 comm="sshd" exe="/usr/sbin/sshd" sig=31 arch=80000016 syscall=201 compat=0 ip=0x3ffb8a3fb32 code=0x0
Verbose:
OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /home/fheimes/.ssh/config
debug1: /home/fheimes/.ssh/config line 6: Deprecated option "useroaming"
debug1: /home/fheimes/.ssh/config line 7: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 10.245.208.7 [10.245.208.7] port 22.
debug1: Connection established.
debug1: identity file /home/fheimes/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fheimes/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fheimes/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fheimes/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fheimes/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fheimes/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fheimes/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/fheimes/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Ubuntu-10
debug1: match: OpenSSH_7.4p1 Ubuntu-10 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 10.245.208.7:22 as 'ubuntu'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256 at libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305 at openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305 at openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:ss9j12+jMMKL9u2vxNeb3XjOeH0E9lw24IG5LxUeJXk
debug1: Host '10.245.208.7' is known and matches the ECDSA host key.
debug1: Found key in /home/fheimes/.ssh/known_hosts:87
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/fheimes/.ssh/id_rsa
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/fheimes/.ssh/id_dsa
debug1: Trying private key: /home/fheimes/.ssh/id_ecdsa
debug1: Trying private key: /home/fheimes/.ssh/id_ed25519
debug1: Next authentication method: password
ubuntu at 10.245.208.7's password:
debug1: Authentication succeeded (password).
Authenticated to 10.245.208.7 ([10.245.208.7]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions at openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: channel 0: free: client-session, nchannels 1
Connection to 10.245.208.7 closed by remote host.
Connection to 10.245.208.7 closed.
Transferred: sent 2084, received 1596 bytes, in 0.0 seconds
Bytes per second: sent 10518567.4, received 8055486.4
debug1: Exit status -1
but loglevel verbose points to this issue:
"fatal: privsep_preauth: preauth child terminated by signal 31"
syslog:
Apr 26 12:39:18 s1lp15 kernel: [12676.655977] audit: type=1326 audit(1493224758.414:99): auid=4294967295 uid=107 gid=65534 ses=4294967295 pid=12380 comm="sshd" exe="/usr/sbin/sshd" sig=31 arch=80000016 syscall=201 compat=0 ip=0x3ff850bfb32 code=0x0
authlog:
Apr 26 12:38:40 s1lp15 sshd[12323]: Connection from 10.172.194.66 port 51512 on 10.245.236.15 port 22
Apr 26 12:38:40 s1lp15 sshd[12323]: Failed publickey for ubuntu from 10.172.194.66 port 51512 ssh2: RSA SHA256:joGsdfW7NbJRkg17sRyXaegoR0iZEdDWdR9Hpbc2KIw
Apr 26 12:38:43 s1lp15 sshd[12323]: Accepted password for ubuntu from 10.172.194.66 port 51512 ssh2
Apr 26 12:38:43 s1lp15 sshd[12323]: fatal: privsep_preauth: preauth child terminated by signal 31
Apr 26 12:39:15 s1lp15 sshd[12379]: Connection from 10.172.194.66 port 51534 on 10.245.236.15 port 22
Apr 26 12:39:16 s1lp15 sshd[12379]: Failed publickey for ubuntu from 10.172.194.66 port 51534 ssh2: RSA SHA256:joGsdfW7NbJRkg17sRyXaegoR0iZEdDWdR9Hpbc2KIw
Apr 26 12:39:18 s1lp15 sshd[12379]: Accepted password for ubuntu from 10.172.194.66 port 51534 ssh2
Apr 26 12:39:18 s1lp15 sshd[12379]: fatal: privsep_preauth: preauth child terminated by signal 31
compared to a system with hw cryto disabled (means ssh working):
syslog:
Apr 26 12:42:04 s1lp15 systemd[1]: Started Session 30 of user ubuntu.
authlog:
Apr 26 12:42:01 s1lp15 sshd[12542]: Connection from 10.172.194.66 port 51658 on 10.245.236.15 port 22
Apr 26 12:42:02 s1lp15 sshd[12542]: Failed publickey for ubuntu from 10.172.194.66 port 51658 ssh2: RSA SHA256:joGsdfW7NbJRkg17sRyXaegoR0iZEdDWdR9Hpbc2KIw
Apr 26 12:42:04 s1lp15 sshd[12542]: Accepted password for ubuntu from 10.172.194.66 port 51658 ssh2
Apr 26 12:42:04 s1lp15 sshd[12542]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)
Apr 26 12:42:04 s1lp15 systemd-logind[1167]: New session 30 of user ubuntu.
Apr 26 12:42:09 s1lp15 sshd[12542]: User child is on pid 12605
Apr 26 12:42:09 s1lp15 sshd[12605]: Starting session: shell on pts/5 for ubuntu from 10.172.194.66 port 51658 id 0
Workaround:
in /etc/ssh/sshd_config
change:
#UsePrivilegeSeparation sandbox
to:
UsePrivilegeSeparation yes
So it's an issue with the sandbox / seccomp
that got fixed in openssh 7.5
release notes: "sshd(8): Avoid sandbox errors for Linux S390 systems using an ICA crypto coprocessor."
corresponding patches/commits:
master branch https://github.com/openssh/openssh-portable
- 5f1596e11d55539678c41f68aed358628d33d86f
- 9e96b41682aed793fadbea5ccd472f862179fb02
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-z-systems/+bug/1686618/+subscriptions
More information about the foundations-bugs
mailing list