[DRAFT] Co-located services

[Benjamin Saller]

[ This is an effort to collect the output of a number of discussions
regarding services that are co-located within a single container. If
you see something that isn't what we talked about or disagree with
something please respond. We intend to use this as the basis for a
specification moving forward. ]

Ensemble Co-located Services
=========================

Services execute in one or more service units. Service Units provide
both a conceptual and a practical containment of a service's runtime.
Until the implementation of this specification service units represent
the smallest execution unit of a service at the concrete level and
map. With the changes in this specification the container of a service
unit will be able to nest additional unit agents in cases where it
makes sense.

Motivations
-----------------

Services such as logging, monitoring and storage often require some
access to the runtime of the service they apply to. Under the current
modeling of services its possible to relate things like monitoring to
services by expressing in the monitored service that it supports a
relationship with the monitoring service. This has possible issues,
monitoring solutions might require direct access to log and state
files as well as machine/container specific stats for load and IO. In
addition this requires that every formula author that wishes to
support logging provide additional interfaces in their formula
specifications. It is desirable to allow for services to have complete
access to the runtime state of another service with little to no
change on the part of the formula author who may never know services
are being co-located with their service.

The following changes are designed to address both these issues and
allow a class of formula that can nest within an existing container
while still taking advantage of the existing relationship machinery.


Terms
-----------

Co-located service/formula: A service designed for and deployed to the
running container of another service.
Parent service: The running service in whose service units we will be
executing units of the co-located service.

Interface
---------------

Ensembles deploy command will take a few options designed to easily
express how services co-locate.

ensemble deploy <formula> [[--with <colo-formual [<service_name>]] ...]

Each `with` stanza takes the name of a formula and an optional service
name under which to deploy it as. The --with argument can be used
multiple times in a single deploy.  All possible relation and
co-location restrictions will validated before any services are
deployed.


ensemble deploy <formula> <service_name> [--in <service_name>]

This format  indicates we are deploying a new formula co-located to an
existing service.

Relations
---------------

When service support co-location (see metadata section) the following
pattern is used. In the event of

deploy <x> --with <y> --with <z>

<y> and <z> will have their install and start hooks fired before <x>.
This would ease services like storage into being properly setup within
a container before the parent service is initialized. In the event of
a

deploy <y> --in <x>

<x> must already be deployed and thus its install hook has already
fired. In this event <y> will have its install and start hooks
triggered before a relationship is established with the parent service
unit. This essentially mimics the behavior of traditional relations.

While the co-located services have relationships with the parent
service (and the parent service may supply them with general
information) its is generally considered a usage error for the parent
service to ever query the co-located service's relation-data.

Service shutdown and unit removal follow a similar pattern. If the
unit of the parent container is removed it will first transition to a
stopped state (when possible) and then the co-located services will
transition to stopped as well. The significant changes here are that
the co-located service will undergo the same state transition to
stopped as the parent and the container will be kept in place unit
such time as the co-located services have finished their hook
execution.

Implications for Status
-------------------------------------

The current model of reporting services should under-go some
modifications. While evolving status is an evolved topic these options
should be evaluated for the short term to address this. Currently I
suggest we show the services and mark them as collocated with a
reference to the parent service but omit co-located units from the
default output. In the future status will need to be much more query
oriented.

Implications for hooks
------------------------------------

The socket nested in the formula directory should allow each unit
agent in the container the ability to communicate with the proper
agent. Some additional assurance might be needed to prevent malicious
code from talking to the wrong agent.

Metadata Changes
------------------------------

Formula's YAML metadata file will grow new constructs. The first is a
`co-located` top level flag which can be 'allow', 'always', 'never'
(defaulting to never when omitted). 'allow' indicates if a formula is
allowed to co-locate with another service (but might be useful through
a normal relationship), 'always' means the service can only deploy in
a co-located fashion and 'never' indicates that it will only support
traditional single unit deployments.

The other important change is the addition of the 'consumes' interface
type. Previously we had 'provides' and 'requires', and these are
mandates, 'consumes' indicates that the formula will create a
relationship with the parent formula if its present but can otherwise
attempt to function without it. This allows some parent formula to
provide additional relationship information to their co-located
services.

Risks
---------

Many of the advantages of containment are given away, we allow for the
introduction of services who can now have conflicting packages,
dependencies, make incompatible file system modifications, and so on.
In many cases the parent formula doesn't know that it will have
co-located services nested in its container and can do nothing in
response to changes made with in its environment. In practice this
should rarely be a serious issue *if* this system is used as intended.
Orthogonal services and container level aspects used for generic
infrastructure modification should be acceptable.

The second risk is that this system is abused in the wild to escape
possible limitations of the current system with regard to machine
reuse. This could inadvertently allow for a new class of bad practice
to become status quo. Its felt that through documentation and the
forth coming changes to the system to support LXC we can navigate this
risk.