[Bug 2004551] Re: upgrade to lunar fails due to rescue-ssh.target or port 22 takeover

Mon Jul 24 11:48:59 UTC 2023

In horsea's syslog.1, some errors happen at the same time that do-dist-
upgrade is running and fails:

Jul 20 10:20:23 node-horsea fwupd[214065]: 10:20:23:0245 FuEngine             failed to get releases for UEFI dbx: No releases found: Not compatible with org.freedesktop.fwupd version 1.7.9, requires >= 1.9.1
Jul 20 10:20:23 node-horsea fwupd[214065]: 10:20:23:0252 FuEngine             failed to get releases for UEFI dbx: No releases found: Not compatible with org.freedesktop.fwupd version 1.7.9, requires >= 1.9.1
[...]
Jul 20 10:28:26 node-horsea kernel: [610494.600470] systemd[1]: Failed to set up automount Arbitrary Executable File Formats File System Automount Point.
[...]
Jul 20 10:28:26 node-horsea kernel: [610494.648822] systemd[1]: multipathd.socket: Failed to create listening socket (@/org/kernel/linux/storage/multipathd): Address already in use
Jul 20 10:28:26 node-horsea kernel: [610494.707090] systemd[1]: multipathd.socket: Failed to listen on sockets: Address already in use
Jul 20 10:28:26 node-horsea kernel: [610494.707135] systemd[1]: multipathd.socket: Failed with result 'resources'.
Jul 20 10:28:26 node-horsea kernel: [610494.707179] systemd[1]: Failed to listen on multipathd control socket.
[... The next appears several times:]
Jul 20 10:28:26 node-horsea kernel: [610494.788337] systemd-journald[529]: Failed to send stream file descriptor to service manager: Transport endpoint is not connected
[...]
Jul 20 10:28:27 node-horsea systemd[1]: multipathd.service: Found left-over process 576 (multipathd) in control group while starting unit. Ignoring.
Jul 20 10:28:27 node-horsea systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 20 10:28:27 node-horsea systemd[1]: Finished Load/Save Random Seed.
Jul 20 10:28:27 node-horsea systemd[1]: Finished Flush Journal to Persistent Storage.
Jul 20 10:28:27 node-horsea systemd[1]: Condition check resulted in First Boot Complete being skipped.
Jul 20 10:28:27 node-horsea systemd[1]: Finished Create System Users.
Jul 20 10:28:27 node-horsea multipathd[217042]: process is already running
Jul 20 10:28:27 node-horsea multipathd[217042]: failed to create pidfile
Jul 20 10:28:27 node-horsea systemd[1]: Starting Create Static Device Nodes in /dev...
Jul 20 10:28:27 node-horsea systemd[1]: multipathd.service: Main process exited, code=exited, status=1/FAILURE
Jul 20 10:28:27 node-horsea systemd[1]: multipathd.service: Got notification message from PID 576, but reception only permitted for main PID which is currently not known
Jul 20 10:28:27 node-horsea systemd[1]: message repeated 2 times: [ multipathd.service: Got notification message from PID 576, but reception only permitted for main PID which is currently not known]
Jul 20 10:28:27 node-horsea multipathd[576]: exit (signal)
Jul 20 10:28:27 node-horsea multipathd[576]: --------shut down-------

[... A lot of the following: Processes 592, 216916 and then 216963 to
217007, 217010 to 217011]

Jul 20 10:28:27 node-horsea systemd[1]: Finished Create Static Device Nodes in /dev.
Jul 20 10:28:27 node-horsea systemd[1]: systemd-udevd.service: Found left-over process 592 (systemd-udevd) in control group while starting unit. Ignoring.
[...]
Jul 20 10:28:27 node-horsea systemd[1]: Starting Rule-based Manager for Device Events and Files...
Jul 20 10:28:27 node-horsea systemd[1]: multipathd.service: Failed with result 'exit-code'.
Jul 20 10:28:27 node-horsea systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
[... Some of this:]
Jul 20 10:28:28 node-horsea systemd-udevd[217552]: nvme0n1p1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/nvme0n1p1' failed with exit code 1.
[... A couple of: ]
Jul 20 10:28:28 node-horsea libvirtd[87522]: internal error: The keyword is not comprised only of uppercase ASCII letters or digits
Jul 20 10:28:28 node-horsea libvirtd[87522]: internal error: A field data length violates the resource length boundary.
Jul 20 10:28:28 node-horsea libvirtd[87522]: internal error: The keyword is not comprised only of uppercase ASCII letters or digits
Jul 20 10:28:28 node-horsea libvirtd[87522]: internal error: A field data length violates the resource length boundary.
[...]
Jul 20 10:28:28 node-horsea systemd-udevd[217550]: sdb: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdb' failed with exit code 1.
[...]
Jul 20 10:28:28 node-horsea cloud-init[217927]: 2023-07-20 10:28:28,415 - stages.py[WARNING]: Failed to rename devices: Failed to apply network config names: [nic not present] Cannot rename mac=8c:dc:d4:b3:6d:e8 to eno49, not available.
[... More left over processes, but this seems due to the previous dpdk tests:]
Jul 20 10:28:28 node-horsea systemd[1]: ovsdb-server.service: Found left-over process 214295 (ovsdb-server) in control group while starting unit. Ignoring.
Jul 20 10:28:28 node-horsea systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 20 10:28:28 node-horsea systemd-udevd[217076]: Failed to unlink /run/udev/queue: No such file or directory
[...]
Jul 20 10:28:28 node-horsea systemd[1]: systemd-resolved.service: Found left-over process 830 (systemd-resolve) in control group while starting unit. Ignoring.
Jul 20 10:28:28 node-horsea systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
[...]
Jul 20 10:28:29 node-horsea fwupd[214065]: 10:28:29:0161 FuEngine             failed to add device /sys/devices/pci0000:00/0000:00:1f.0: failed to add device using on pci_bcr: could not read BCR: 5ca35f261be27f4d2786162a4e2dd1494697e678 [C610/X99 series chipset LPC Controller] has not been opened
Jul 20 10:28:29 node-horsea fwupd[214065]: 10:28:29:0250 FuEngine             failed to add device /sys/devices/pci0000:00/0000:00:1c.4/0000:02:00.0: EEPROM size invalid, got 0x80000, expected 0x40000
Jul 20 10:28:30 node-horsea ModemManager[1150]: <info>  [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:02.2/0000:04:00.1': not supported by any plugin
Jul 20 10:28:30 node-horsea ModemManager[1150]: <info>  [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:03.0/0000:08:00.0': not supported by any plugin
Jul 20 10:28:30 node-horsea ModemManager[1150]: <info>  [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:1c.4/0000:02:00.2': not supported by any plugin
Jul 20 10:28:30 node-horsea ModemManager[1150]: <info>  [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:1c.4/0000:02:00.3': not supported by any plugin
Jul 20 10:28:30 node-horsea ModemManager[1150]: <info>  [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:1c.4/0000:02:00.0': not supported by any plugin
Jul 20 10:28:30 node-horsea ModemManager[1150]: <info>  [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:1c.4/0000:02:00.1': not supported by any plugin
[...]

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2004551

Title:
  upgrade to lunar fails due to rescue-ssh.target or port 22 takeover

Status in openssh package in Ubuntu:
  New

Bug description:
  Hi,
  I just upgraded a system from Jammy to Lunar and openssh-server refuses to upgrade well.

  Setting up openssh-server (1:9.0p1-1ubuntu8) ...
  Replacing config file /etc/ssh/sshd_config with new version
  Replacing config file /etc/ssh/sshd_config with new version
  Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
  Executing: /lib/systemd/systemd-sysv-install disable ssh
  rescue-ssh.target is a disabled or a static unit not running, not starting it.
  Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.
  dpkg: error processing package openssh-server (--configure):
   installed openssh-server package post-installation script subprocess returned error exit status 1
  Processing triggers for man-db (2.11.2-1) ...
  Processing triggers for libc-bin (2.36-0ubuntu4) ...
  Errors were encountered while processing:
   openssh-server
  Error: Timeout was reached
  needrestart is being skipped since dpkg has failed
  E: Sub-process /usr/bin/dpkg returned an error code (1)

  I'm not sure what exactly it is.
  This output complains about rescue-ssh.target and indeed that can not be started even directly.

  $ sudo systemctl start rescue-ssh.target
  A dependency job for rescue-ssh.target failed. See 'journalctl -xe' for details.

  And in postinst is a try to start it:
  $  grep rescue /var/lib/dpkg/info/openssh-server.postinst 
  		deb-systemd-invoke $_dh_action 'rescue-ssh.target' >/dev/null || true

  But I think the underlying issue is that ssh is already on, and I'm logged in via it.
  And that makes the service restart of the ssh socket which was added break.

  Feb 02 10:40:56 node-horsea systemd[104560]: ssh.socket: Failed to create listening socket ([::]:22): Address already in use
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to receive listening socket ([::]:22): Input/output error
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to listen on sockets: Input/output error
  Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed with result 'resources'.

  Now, whichever it is, it is hard to resolve.
  The only way to get the socket to own it would be rebooting so that sshd lets go and systemd can take over.
  I could reboot, but that is not the point.
  What if I'd want to get the service and upgrade completed before reboot.
  Because as of now dpkg considers the system unhappy, and that would usually be a sign for "better not reboot before being resolved" to me.

  One thing though, I have not upgraded with do-release-upgrade - would
  we / do we have magic there to make the ssh socket activation
  transition smoother?

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2004551/+subscriptions