[Bug 1575572] Re: apache2 fails to start if installed via cloud config (on Xenial)

[Martin Pitt]

For the record, even after fixing this, installing packages from current
cloud-init will still not be 100% reliable in its current form.

The design problem here is that cloud-init runs *in* the  boot sequence,
i. e. in the transaction where default.target and its dependencies get
started. You can't place/start new services into the boot dependency
tree (i. e. dependencies of default.target) while default.target is
still being started. Any new service which wants to start needs to wait
until after the initial boot happens, and will then start afterwards.

invoke-rc.d has  some code to deal with that situation, to avoid
deadlocks: You can't run the blocking "systemctl start" in a postinst
that you run within the boot sequence -- you can at most try to start it
without any dependencies and in non-blocking mode, which is why invoke-
rc.d has this code:

                # avoid deadlocks during bootup and shutdown from units/hooks
                # which call "invoke-rc.d service reload" and similar, since
                # the synchronous wait plus systemd's normal behaviour of
                # transactionally processing all dependencies first easily
                # causes dependency loops
                if ! OUT=$(systemctl is-system-running 2>/dev/null) && [ "$OUT" != "degraded" ]; then
                    sctl_args="--job-mode=ignore-dependencies"
                fi

This was mostly added for things like /etc/network/ifup.d/ scripts that
start stuff, to avoid getting deadlocks on boot. But it would apply here
too.

For the most part this should work fine, unless apache expects that any
of its dependencies actually get started in that situation (they can't).
But if you *do* install a package with an init.d script or service that
has dependencies that are not already running, they will *not* be
started.

This behaviour of installing packages or configuring your system while
the system isn't booted yet might also bite you in other cases (it
certainly bit me a number of cases). Maybe we should discuss how to move
that after the boot? I do that in https://git.launchpad.net/~ubuntu-
release/+git/autopkgtest-cloud/tree/tools/armf-lxd-slave.userdata but
this doesn't look very pretty.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1575572

Title:
  apache2 fails to start if installed via cloud config (on Xenial)

Status in cloud-init package in Ubuntu:
  Confirmed
Status in init-system-helpers package in Ubuntu:
  In Progress
Status in systemd package in Ubuntu:
  Won't Fix

Bug description:
  Using the following cloud config, apache2 will fail to start on
  installation on Xenial:

  #cloud-config
  packages:
  - apache2

  See for example:

  $ gcloud compute instances create xenial-$(date +%y%m%d-%H%M) --image ubuntu-1604-xenial-v20160420c --image-project ubuntu-os-cloud --metadata-from-file user-data=cloud-config
  NAME                ZONE            MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
  xenial-160427-1050  europe-west1-d  n1-standard-1               10.240.0.7   104.155.86.94  RUNNING

  $ ssh ubuntu at 104.155.86.94 systemctl status apache2.service
  ● apache2.service - LSB: Apache2 web server
     Loaded: loaded (/etc/init.d/apache2; bad; vendor preset: enabled)
    Drop-In: /lib/systemd/system/apache2.service.d
             └─apache2-systemd.conf
     Active: inactive (dead)
       Docs: man:systemd-sysv-generator(8)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1575572/+subscriptions