Dependency engine in the machine agent

[Menno Smits]

Hi everyone,

One of the pieces of the puzzle required for model migrations was to get
all the workers in the machine agent running under the new dependency
engine framework. It will allow us to cleanly shut down workers so that a
model to migrated can settle to a stable, "read-only" mode. This has been a
major focus for team Onyx and Will.

If you're not sure what the dependency engine is all about, a good place to
start is Will's excellent documentation in the source tree:

https://github.com/juju/juju/blob/master/worker/dependency/doc.go

The tl;dr is that the dependency engine provides a framework where workers
are wrapped in a "manifold" which describes the resources the worker
depends on, a function to start the worker and (optionally) a function
which makes resources available to other manifolds/workers. In the context
of the dependency engine framework, a "resource" is any value that is
exported by a manifold for use by other manifolds.

All the manifolds for an agent are registered with a dependency engine
which then starts, stops and restarts workers based on the availability of
their configured inputs (i.e. dependencies) and worker behaviour (i.e. when
they exit and the errors they exit with).

One nice property of the dependency engine framework is that it makes it
very clear what a worker depends on and how it starts. The machine agent
has become a complex hierarchy of runners and workers with sometimes
non-obvious interactions between them. When the machine agent was younger
and simpler the "nested runners" architecture was elegant and appropriate,
but as more demands have been put on the machine agent it's become almost
unmanageable. Fixing this wasn't the main driver for moving the machine
agent to the dependency engine framework but it's been a side benefit.

The machine agent dependency engine work has also meant that code to set up
and start specific workers has been moved out of the machine agent itself
to worker-specific packages, meaning everything to do with a worker is
located together, and the machine agent's implementation is becoming
clearer and more concise. Testing of workers and the agent has also become
more straightforward and more robust.

So where are things now? Before Onyx started doing this work on the machine
agent, the unit agent was already completely converted to use the
dependency engine for its workers - a unit agent runs far fewer workers so
it was a good place to start. Much of the conversion work for the machine
agent has already landed in master and there's several feature branches at
varying levels of completion that cover most of the remaining work.
Specifically the following areas have been converted:

   - a manifold which manages the API connection
   - most of the (many!) workers which depend on the API connection
   - the base workers which manage a State instance (state servers only)
   - some of the workers which depend on State

What does this mean for Juju developers? If you're adding a new worker to
the machine (or unit) agent you'll need to create a manifold for it and
integrate it with the agent's dependency engine. Similarly if you're making
changes to an existing worker, it's likely to have already been converted
so you'll need to understand how manifolds and the dependency engine work.

If you're not sure about something dependency engine related, please feel
free to ask Will, Jesse or me. There's an initial learning curve but it's
great once you get basic concepts straight. Aside from the docs (linked
above), there's also lots of examples to study. Any worker package you find
in the source with a manifold.go file has been converted.

- Menno
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20160303/10115c02/attachment.html>