Is it possible to remotely force a reboot on Ubuntu 20.04.3?

[Karl Auer]

On Mon, 2022-02-21 at 15:44 +0100, Bo Berglund wrote:
> So it seems like it is in real need of a restart...
> But is it possible to force such when the situation is as described?

Addressing this question,but not the apparent ssh issue:

You can't force a restart of any system without some preparation.

   "If they think you're crude, go technical; if
    they think you're technical, go crude. I'm a
    very technical boy. So I decided to get as
    crude as possible. These days, though, you
    have to be pretty technical before you can
    even aspire to crudeness."
        - William Gibson, "Johnny Mnemonic"

You gave one lo-tech suggestion yourself, namely a timer on the power
supply, but this is a bit brutal and is inviting filesystem corruption.

You can buy remote-controlled switches, allowing you to turn the power
off and on any time you want. Obviously these depend on a functional
network and access to the Internet. It's also important that these
switches are themselves secure! You don't want random Internet users
switching off your server...

Another system is the "trained monkey" method, a rather rude phrase
meaning that you provide physical access to a trusted third party such
as a neighbour. If you need something done like a reboot you can call
them and ask them to do it for you. Not useful for a system in an
isolated area though, or without trustworthy neighbours.

One fairly simple mechanism is a "dead man switch" on the system itself
- a process which looks for something, and reboots the system (cleanly)
if it doesn't find it. For example, it could check every 24 hours for
the existence of /tmp/dont_kill_me, and if it finds it, deletes it. If
it doesn't find it, it reboots the system. It is up to you to make sure
that file is always there, for example by having a cronjob on your
local system that uses scp to put the file there twice a day (to avoid
race conditions).

Obviously you can be as complicated as you like about what specifically
your dead man switch looks for and how frequently it looks etc. I often
use a dead man switch if I have to change firewall rules on a remote
system. I set things up so that my dead man switch puts known-working
firewall rules in place. That way, if I do something silly that cuts me
off from the system, I need only wait a little while and the old rules
will be restored.

You can combine the dead man switch idea and the remote-controlled
power switch by having a separate watchdog system that automatically
cycles power to a system that does not respond.

Server-class systems, designed for operation in datacentres, usually
have some out-of-band support built in that allows some inspection and
control of the hardware remotely, bypassing the operating system
completely, but these are not common on consumer-class systems.

Not as reliable, but you can also run two access methods on the same
system. The commonest is to run two ssh servers on two different ports.
The second one is for use if the first one fails for some reason. 

If you are willing to allow remote access into your local system, you
can set things up so that the remote system makes an ssh connection to
you and sets up a port forwarding back to itself. This does not require
a functional ssh server on the remote machine, just a functional ssh
client. It does require root access on your local machine though. This
is actually a nice way to secure some kinds of systems, where access
from multiple locations is not required. You could combine this with a
dead man switch so the connection only comes up when there is a
problem.

Hope some of these ideas help you for next time.

Regards, K.

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Karl Auer (kauer at biplane.com.au)
http://www.biplane.com.au/kauer

GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58
Old fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170