[Bug 2004551] Re: upgrade to lunar fails due to rescue-ssh.target or port 22 takeover
Miriam España Acebal
2004551 at bugs.launchpad.net
Thu Jul 27 11:40:54 UTC 2023
I'm starting to think that there is something wrong with the horsea
machine indeed: still not able to upgrade to lunar. Starting from jammy
or from kinetic failed.
#Jammy
Deploying again jammy and doing do-release-upgrade again failed with
ssh, with differents things after rebooting and trying the do the
release upgrade (I have outputs, but I don't remember exactly at what
point I did the reboot of the machine, i.e: "Missing privilege
separation directory: /run/sshd" could happen after reboot, but not
sure).
#Kinetic
So, ok... let's think is something wrong with the jammy image that is being deployed (a remote possibility, I think, but ...).... trying the upgrading from Kinetic: failed. I checked the system status before starting the upgrade and it was not degraded.
It was stucked at 97% on:
Setting up snapd (2.58+22.10.1) ...
Installing new version of config file /etc/apt/apt.conf.d/20snapd.conf ...
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
snapd.failure.service is a disabled or a static unit not running, not starting it.
snapd.snap-repair.service is a disabled or a static unit not running, not starting it.
Failed to restart snapd.mounts-pre.target: Operation refused, unit snapd.mounts-pre.target may be requested by dependenc
y only (it is configured to refuse manual start/stop).
See system logs and 'systemctl status snapd.mounts-pre.target' for details.
system status was:
│ └─unattended-upgrades.service
│ ├─ 1227 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
│ └─11628 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
└─user.slice
└─user-1000.slice
└─session-1.scope
├─ 1856 "sshd: ubuntu [priv]"
├─ 1937 "sshd: ubuntu at pts/0"
├─ 1941 -bash
├─ 2302 sudo apt upgrade -y
├─ 2303 sudo apt upgrade -y
├─ 2304 apt upgrade -y
├─ 5750 /usr/bin/dpkg --status-fd 39 --configure --pending
├─ 5751 sh -c "(test -x /usr/lib/needrestart/dpkg-status && /usr/lib/needrestart/dpkg-status || cat > /dev/null>
├─ 5752 sh -c "(test -x /usr/lib/needrestart/dpkg-status && /usr/lib/needrestart/dpkg-status || cat > /dev/null>
├─ 5753 /bin/sh /usr/lib/needrestart/dpkg-status
├─11336 /bin/sh /var/lib/dpkg/info/snapd.postinst configure 2.57.5+22.10ubuntu0.1
├─11501 /usr/bin/perl /usr/bin/deb-systemd-invoke restart snapd.aa-prompt-listener.service snapd.apparmor.servi>
├─11563 systemctl --quiet --system restart snapd.aa-prompt-listener.service snapd.apparmor.service snapd.autoim>
└─11564 /bin/systemd-tty-ask-password-agent --watch
After that, it failed with:
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
Setting up openssh-sftp-server (1:9.0p1-1ubuntu7.3) ...
Setting up needrestart (3.6-1ubuntu0.1) ...
Setting up openssh-server (1:9.0p1-1ubuntu7.3) ...
rescue-ssh.target is a disabled or a static unit not running, not starting it.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
dpkg: error processing package openssh-server (--configure):
installed openssh-server package post-installation script subprocess returned error exit status 1
Setting up bind9-dnsutils (1:9.18.12-0ubuntu0.22.10.2) ...
Processing triggers for libc-bin (2.36-0ubuntu4) ...
Processing triggers for rsyslog (8.2208.0-1ubuntu2) ...
Processing triggers for ufw (0.36.1-4build1) ...
Processing triggers for man-db (2.10.2-2) ...
Processing triggers for dbus (1.14.0-2ubuntu3) ...
Processing triggers for initramfs-tools (0.140ubuntu17) ...
update-initramfs: Generating /boot/initrd.img-5.19.0-46-generic
Processing triggers for ca-certificates (20230311ubuntu0.22.10.1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Errors were encountered while processing:
openssh-server
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
System status after that was:
ubuntu at node-horsea:~$ systemctl status -l
Failed to read server status: Connection timed out
ubuntu at node-horsea:~$ systemctl status -l
● node-horsea
State: degraded
Units: 425 loaded (incl. loaded aliases)
Jobs: 1 queued
Failed: 7 units
Since: Thu 2023-07-27 10:48:27 UTC; 18min ago
systemd: 251.4-1ubuntu7.3
CGroup: /
├─init.scope
│ └─1 /lib/systemd/systemd --- --system --deserialize 22
├─system.slice
│ ├─cron.service
│ │ └─1138 /usr/sbin/cron -f -P
│ ├─dbus.service
│ │ ├─1139 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
│ │ └─6714 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
│ ├─irqbalance.service
│ │ ├─1145 /usr/sbin/irqbalance --foreground
│ │ └─6719 /usr/sbin/irqbalance --foreground
│ ├─multipathd.service
│ │ └─6654 /sbin/multipathd -d -s
│ ├─packagekit.service
│ │ └─2204 /usr/libexec/packagekitd
│ ├─polkit.service
│ │ └─2208 /usr/libexec/polkitd --no-debug
│ ├─rsyslog.service
│ │ └─12184 /usr/sbin/rsyslogd -n -iNONE
│ ├─snapd.service
│ │ ├─ 1150 /usr/lib/snapd/snapd
│ │ └─23845 /usr/lib/snapd/snapd
│ ├─ssh.service
│ │ ├─ 1849 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
│ │ ├─11762 "sshd: ubuntu [priv]"
│ │ ├─11841 "sshd: ubuntu at pts/3"
│ │ ├─11842 -bash
│ │ ├─23869 systemctl status -l
│ │ └─23870 less
│ ├─system-getty.slice
│ │ └─getty at tty1.service
│ │ └─6840 /sbin/agetty -o "-p -- \\u" --noclear - linux
│ ├─system-serial\x2dgetty.slice
│ │ └─serial-getty at ttyS1.service
│ │ └─6834 /sbin/agetty -o "-p -- \\u" --keep-baud 115200,57600,38400,9600 - vt220
│ ├─systemd-journald.service
│ │ └─3834 /lib/systemd/systemd-journald
│ ├─systemd-logind.service
│ │ └─11626 /lib/systemd/systemd-logind
│ ├─systemd-networkd.service
│ │ ├─1086 /lib/systemd/systemd-networkd
│ │ └─4522 /lib/systemd/systemd-networkd
│ ├─systemd-resolved.service
│ │ └─6942 /lib/systemd/systemd-resolved
│ ├─systemd-timesyncd.service
│ │ └─5877 /lib/systemd/systemd-timesyncd
│ ├─systemd-udevd.service
│ │ └─udev
│ │ └─5921 /lib/systemd/systemd-udevd
│ └─unattended-upgrades.service
│ ├─ 1227 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
│ └─11628 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
└─user.slice
└─user-1000.slice
└─session-1.scope
├─ 1856 "sshd: ubuntu [priv]"
├─ 1937 "sshd: ubuntu at pts/0"
├─ 1941 -bash
├─23835 sudo su
├─23836 sudo su
├─23837 su
└─23838 bash
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2004551
Title:
upgrade to lunar fails due to rescue-ssh.target or port 22 takeover
Status in openssh package in Ubuntu:
New
Bug description:
Hi,
I just upgraded a system from Jammy to Lunar and openssh-server refuses to upgrade well.
Setting up openssh-server (1:9.0p1-1ubuntu8) ...
Replacing config file /etc/ssh/sshd_config with new version
Replacing config file /etc/ssh/sshd_config with new version
Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable ssh
rescue-ssh.target is a disabled or a static unit not running, not starting it.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
dpkg: error processing package openssh-server (--configure):
installed openssh-server package post-installation script subprocess returned error exit status 1
Processing triggers for man-db (2.11.2-1) ...
Processing triggers for libc-bin (2.36-0ubuntu4) ...
Errors were encountered while processing:
openssh-server
Error: Timeout was reached
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
I'm not sure what exactly it is.
This output complains about rescue-ssh.target and indeed that can not be started even directly.
$ sudo systemctl start rescue-ssh.target
A dependency job for rescue-ssh.target failed. See 'journalctl -xe' for details.
And in postinst is a try to start it:
$ grep rescue /var/lib/dpkg/info/openssh-server.postinst
deb-systemd-invoke $_dh_action 'rescue-ssh.target' >/dev/null || true
But I think the underlying issue is that ssh is already on, and I'm logged in via it.
And that makes the service restart of the ssh socket which was added break.
Feb 02 10:40:56 node-horsea systemd[104560]: ssh.socket: Failed to create listening socket ([::]:22): Address already in use
Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to receive listening socket ([::]:22): Input/output error
Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed to listen on sockets: Input/output error
Feb 02 10:40:56 node-horsea systemd[1]: ssh.socket: Failed with result 'resources'.
Now, whichever it is, it is hard to resolve.
The only way to get the socket to own it would be rebooting so that sshd lets go and systemd can take over.
I could reboot, but that is not the point.
What if I'd want to get the service and upgrade completed before reboot.
Because as of now dpkg considers the system unhappy, and that would usually be a sign for "better not reboot before being resolved" to me.
One thing though, I have not upgraded with do-release-upgrade - would
we / do we have magic there to make the ssh socket activation
transition smoother?
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2004551/+subscriptions
More information about the foundations-bugs
mailing list