Automatic restart of daemons on snap installation

[Danilo Šegan]

Hi all,

We are using snaps to deliver software to juju deployed units, and we
are hitting a bit of a blocker: upon snap installation, daemons
provided by a snap get unconditionally enabled and (re)started.

I've filed a bug already:

  https://bugs.launchpad.net/bugs/1664163

But I wonder what do people think about making this optional?

The code in question that does this seems to be:

https://github.com/snapcore/snapd/blob/master/interfaces/systemd/backend.go#L115

This is a problem for us for multiple reasons:

 1. Upon original installation, service will attempt to start before
we've even written out a configuration file, which means that it will
fail to start.

    In a charm world, getting details to finish the configuration can
take a while (think waiting for DB to become available), and while we
could delay snap installation up to a point where we've got all the
data available to write the configuration out, it would still result in
at least one restart without the configuration present: we can only
configure the snap after it is installed.

    This would also feel a bit unnatural to me: I'd like to keep flows
in the charm as simple as possible because that makes it easier to
guarantee and prove correctness: to *install* a snap, the only pre-
condition is that the snap is available, and not that everything is
there for it to be *configured* (it also makes it faster: waiting for
everything to settle would make nodes sit idle while other nodes get
brough up and installed).

    I understand the desire behind the explicit restart and service
enablement: installing a snap should let users actually use the
software they installed right away, and we could have snap provide an
installation that runs without any configuration for the initial
install, but that's where we get to our next issue.


2. We are deploying multiple units of the same service and for HA
and/or no-downtime upgrades, we do something along these lines:

  a) unit-0: stop daemon, upgrade snap, regenerate config
 
  b) unit-1: stop daemon

  c) unit-0: run DB schema upgrade, restart daemon

  d) unit-1: restart daemon

   This is a bit simplified, but allows us to back off if any of the
steps in a) fail, and the downtime is there only after b) the c) step.

   However, to achieve this, we need snapd not to attempt to restart
services after snap is installed: new version might need configuration
or DB schema changes.

It would be great if this would be made an option for "daemon" commands
in snaps, eg. "restart-daemon-on-snap-install" that I suggest in the
bug above.  Or, it is probably even better to optionally not
explicitely "enable" the service post installation.

Daemons are a bit of a special case already, and their installation is
even more so (as evidenced by all the options that apply to them, some
of them 1-to-1 mapping of systemd internal behaviour).


Cheers,
Danilo