[Bug 2004551] Re: upgrade to lunar fails due to rescue-ssh.target or port 22 takeover

[Martin Pitt]

We see this in our CI infra as well, it's not just about jammy → lunar
dist-upgrades, but also happens with regular updates within lunar as
well. The cloud image from a few days ago had an older openssh-server,
tried to update to 1:9.0p1-1ubuntu8.4, and failed:

Setting up openssh-server (1:9.0p1-1ubuntu8.4) ...
Creating SSH2 ECDSA key; this may take some time ...
256 SHA256:r08SEQICcnkgu0I1PQwdYTqHeuYVRkz4oPF9VJeC5xQ root at ubuntu (ECDSA)
Creating SSH2 ED25519 key; this may take some time ...
256 SHA256:Qaa3l3IsQuQpKwtESOnfyg5STi64/5pVX4WfiEGX5CI root at ubuntu (ED25519)
rescue-ssh.target is a disabled or a static unit not running, not starting it.
Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.
dpkg: error processing package openssh-server (--configure):
 installed openssh-server package post-installation script subprocess returned error exit status 1

Today's cloud image avoids this problem as it has the current openssh-
server package on it already, so it's not part of the dist-upgrade. But
this will likely come back.


[1] https://cockpit-logs.us-east-1.linodeobjects.com/image-refresh-logs/ubuntu-stable-20230727-131010.log

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2004551

Title:
  upgrade to lunar fails due to rescue-ssh.target or port 22 takeover

Status in openssh package in Ubuntu:
  New

Bug description:
  Hi,
  I just upgraded a system from Jammy to Lunar and openssh-server refuses to upgrade well.

  Setting up openssh-server (1:9.0p1-1ubuntu8) ...
  Replacing config file /etc/ssh/sshd_config with new version
  Replacing config file /etc/ssh/sshd_config with new version
  Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
  Executing: /lib/systemd/systemd-sysv-install disable ssh
  rescue-ssh.target is a disabled or a static unit not running, not starting it.
  Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.
  dpkg: error processing package openssh-server (--configure):
   installed openssh-server package post-installation script subprocess returned error exit status 1
  Processing triggers for man-db (2.11.2-1) ...
  Processing triggers for libc-bin (2.36-0ubuntu4) ...
  Errors were encountered while processing:
   openssh-server
  Error: Timeout was reached
  needrestart is being skipped since dpkg has failed
  E: Sub-process /usr/bin/dpkg returned an error code (1)

  I'm not sure what exactly it is.
  This output complains about rescue-ssh.target and indeed that can not be started even directly.

  $ sudo systemctl start rescue-ssh.target
  A dependency job for rescue-ssh.target failed. See 'journalctl -xe' for details.

  And in postinst is a try to start it:
  $  grep rescue /var/lib/dpkg/info/openssh-server.postinst 
  		deb-systemd-invoke $_dh_action 'rescue-ssh.target' >/dev/null || true

  
  But I think the underlying issue is that ssh is already on, and I'm logged in via it.
  And that makes the service restart of the ssh socket which was added break.

  Feb 02 10:40:56 node-horsea systemd[104560]: ssh.socket: Failed to create listening socket ([::]:22): Address already in use
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to receive listening socket ([::]:22): Input/output error
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to listen on sockets: Input/output error
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed with result 'resources'.

  
  Now, whichever it is, it is hard to resolve.
  The only way to get the socket to own it would be rebooting so that sshd lets go and systemd can take over.
  I could reboot, but that is not the point.
  What if I'd want to get the service and upgrade completed before reboot.
  Because as of now dpkg considers the system unhappy, and that would usually be a sign for "better not reboot before being resolved" to me.

  One thing though, I have not upgraded with do-release-upgrade - would
  we / do we have magic there to make the ssh socket activation
  transition smoother?

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2004551/+subscriptions