[Bug 1874455] Re: ssh daemon doesn't report successful start state to systemd after upgrade to 1:7.2p2-4ubuntu2.9

Thu Apr 23 15:15:22 UTC 2020

Very strange. Surprisingly, the simple re-installation did do the trick.
Luckily, `openssh-server' package doesn't have any hard dependencies to
the essential distro meta packages.

Well, originally, I was going to lock the version from `security' repo
but first I would have to remove the package anyway, so:

$ sudo apt-get remove openssh-server

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libssl1.0.0:i386 openssh-sftp-server
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  openssh-server
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 904 kB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 357805 files and directories currently installed.)
Removing openssh-server (1:7.2p2-4ubuntu2.9) ...
Processing triggers for man-db (2.7.5-1) ...

$ sudo apt-get -y autoremove

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  libssl1.0.0:i386 openssh-sftp-server
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.
After this operation, 3,223 kB disk space will be freed.
(Reading database ... 357792 files and directories currently installed.)
Removing libssl1.0.0:i386 (1.0.2g-1ubuntu4.15) ...
Removing openssh-sftp-server (1:7.2p2-4ubuntu2.9) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Processing triggers for man-db (2.7.5-1) ...

$ sudo apt-get install openssh-server

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  openssh-sftp-server
Suggested packages:
  ssh-askpass rssh molly-guard monkeysphere
Recommended packages:
  ncurses-term ssh-import-id
The following NEW packages will be installed:
  openssh-server openssh-sftp-server
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/374 kB of archives.
After this operation, 1,013 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Preconfiguring packages ...
Selecting previously unselected package openssh-sftp-server.
(Reading database ... 357773 files and directories currently installed.)
Preparing to unpack .../openssh-sftp-server_1%3a7.2p2-4ubuntu2.9_amd64.deb ...
Unpacking openssh-sftp-server (1:7.2p2-4ubuntu2.9) ...
Selecting previously unselected package openssh-server.
Preparing to unpack .../openssh-server_1%3a7.2p2-4ubuntu2.9_amd64.deb ...
Unpacking openssh-server (1:7.2p2-4ubuntu2.9) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for ureadahead (0.100.0-19.1) ...
ureadahead will be reprofiled on next reboot
Processing triggers for ufw (0.35-0ubuntu2) ...
Processing triggers for systemd (229-4ubuntu21.27) ...
Setting up openssh-sftp-server (1:7.2p2-4ubuntu2.9) ...
Setting up openssh-server (1:7.2p2-4ubuntu2.9) ...

$    sudo systemctl status sshd.service 

● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-04-23 18:07:13 MSK; 13s ago
  Process: 27871 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
 Main PID: 27874 (sshd)
   CGroup: /system.slice/ssh.service
           ├─ 9759 sshd: ia [priv]
           ├─ 9771 sshd: ia at pts/18
           ├─ 9772 -zsh
           └─27874 /usr/sbin/sshd -D

Apr 23 18:07:13 asrockZ97e systemd[1]: Starting OpenBSD Secure Shell server...
Apr 23 18:07:13 asrockZ97e sshd[27874]: Server listening on 0.0.0.0 port 22.
Apr 23 18:07:13 asrockZ97e sshd[27874]: Server listening on :: port 22.
Apr 23 18:07:13 asrockZ97e systemd[1]: Started OpenBSD Secure Shell server.

$

** Changed in: openssh (Ubuntu)
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/1874455

Title:
  ssh daemon doesn't report successful start state to systemd after
  upgrade to 1:7.2p2-4ubuntu2.9

Status in openssh package in Ubuntu:
  Invalid

Bug description:
  System info:

  $ lsb_release -rd

  Description:	Ubuntu 16.04.6 LTS
  Release:	16.04

  $ apt-cache policy openssh-server

  openssh-server:
    Installed: 1:7.2p2-4ubuntu2.9
    Candidate: 1:7.2p2-4ubuntu2.9
    Version table:
   *** 1:7.2p2-4ubuntu2.9 500
          500 https://mirror.one.com/ubuntu xenial-proposed/main amd64 Packages
          100 /var/lib/dpkg/status
       1:7.2p2-4ubuntu2.8 500
          500 https://mirror.one.com/ubuntu xenial-security/main amd64 Packages
          500 https://mirror.one.com/ubuntu xenial-updates/main amd64 Packages
          500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
       1:7.2p2-4 500
          500 https://mirror.one.com/ubuntu xenial/main amd64 Packages
          500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

   - what should happen?

  After regular update ssh daemon should restart successfully.

   - what actually happens:

  $ sudo apt-get -y update

  [skip]

  Fetched 2,473 B in 1s (1,702 B/s)
  Reading package lists... Done

  $ sudo apt-get -y dist-upgrade

  Reading package lists... Done
  Building dependency tree       
  Reading state information... Done
  Calculating upgrade... Done
  The following packages were automatically installed and are no longer required:
    linux-headers-4.4.0-164 linux-headers-4.4.0-164-generic linux-image-4.4.0-164-generic linux-modules-4.4.0-164-generic linux-modules-extra-4.4.0-164-generic
  Use 'sudo apt autoremove' to remove them.
  0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
  1 not fully installed or removed.
  After this operation, 0 B of additional disk space will be used.
  Setting up openssh-server (1:7.2p2-4ubuntu2.9) ...
  Job for ssh.service failed because a timeout was exceeded. See "systemctl status ssh.service" and "journalctl -xe" for details.
  invoke-rc.d: initscript ssh, action "restart" failed.
  ● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: timeout) since Thu 2020-04-23 17:21:42 MSK; 3ms ago
    Process: 9000 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=0/SUCCESS)
    Process: 8996 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 9000 (code=exited, status=0/SUCCESS)

  Apr 23 17:21:42 asrockZ97e systemd[1]: Failed to start OpenBSD Secure Shell server.
  Apr 23 17:21:42 asrockZ97e systemd[1]: ssh.service: Unit entered failed state.
  Apr 23 17:21:42 asrockZ97e systemd[1]: ssh.service: Failed with result 'timeout'.
  dpkg: error processing package openssh-server (--configure):
   subprocess installed post-installation script returned error exit status 1
  Errors were encountered while processing:
   openssh-server
  E: Sub-process /usr/bin/dpkg returned an error code (1)

  However, if I check manually then everything seems fine:

  $ ps aux | grep sshd | grep -v grep

  root      9481  0.0  0.0  26212  3728 ?        Ss   17:23   0:00
  /usr/sbin/sshd -D

  $ sudo systemctl status sshd.service

  ● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: activating (start) since Thu 2020-04-23 17:29:15 MSK; 1min 19s ago
    Process: 10454 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 10457 (sshd)
     CGroup: /system.slice/ssh.service
             ├─ 9759 sshd: ia [priv]
             ├─ 9771 sshd: ia at pts/18
             ├─ 9772 -zsh
             ├─10457 /usr/sbin/sshd -D
             ├─10750 sudo systemctl status sshd.service
             └─10760 systemctl status sshd.service

  Apr 23 17:29:15 asrockZ97e systemd[1]: Starting OpenBSD Secure Shell server...
  Apr 23 17:29:15 asrockZ97e sshd[10457]: Set /proc/self/oom_score_adj from 0 to -1000
  Apr 23 17:29:15 asrockZ97e sshd[10457]: Server listening on 0.0.0.0 port 22.
  Apr 23 17:29:15 asrockZ97e sshd[10457]: Server listening on :: port 22.
  Apr 23 17:30:34 asrockZ97e sudo[10750]:       ia : TTY=pts/18 ; PWD=/home/ia ; USER=root ; COMMAND=/bin/systemctl status sshd.service
  Apr 23 17:30:34 asrockZ97e sudo[10750]: pam_unix(sudo:session): session opened for user root by ia(uid=0)

  $ ssh localhost

  Last login: Thu Apr 23 17:01:34 2020

  $

  After some google fu, I found some recommendations about removing `-D' from `ExecStart' and replacing `Type=notify' by `Type=forking' in `/etc/systemd/system/sshd.service' but I'm not sure how accurate those hints are and what the real root cause behind such sudden and unexpected change in behavior.

  P.S. I do realize that I use the package from `proposed' repo but I
  don't think that it's the excuse for a such strange behavior after
  upgrade.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1874455/+subscriptions