Juju architecture questions

[Clint Byrum]

Excerpts from Thomas Leonard's message of 2012-09-07 08:05:16 -0700:
> Is there a diagram somewhere showing how all the different bits of Juju fit 
> together?
> 
> I found it quite confusing when I read the tutorial for the first time, and 
> there are still bits that I'm not sure about. I tried making a diagram to 
> show the example (wordpress + mysql). It's probably wrong, but maybe someone 
> could correct it?
> 
> https://docs.google.com/drawings/d/1LVrBjoMivIQZf6TNaquA5dFrZcd2x2Cj_XNB2GBPaA0/edit
> 
> 

Its pretty close. You missed the provisioning agent though.

https://docs.google.com/drawings/d/1Az4cx4w6J3dvJfQmg8ZbrOIx8vSuBFhYcaP0nHYUHEQ/edit

Its existence helps understand a lot of the things you're confused about.

> Here's my understanding so far (please correct):
> 
> The Juju admin has the environment.yaml and the data directory. Together, 
> these give access to the system and could be shared with other admins, 
> allowing them to manage the system together.
> 
> [ which bits of the data directory need to be shared, and which are just log 
> files and temporary files? do the admins need to sync anything manually 
> between their systems? ]
> 

Only environments.yaml must be shared if you are using the official charm store.
If you are using local: charms, then you need to share the charm repository too.

> For EC2, bootstrap deploys a new machine and runs Zookeeper on it to store 
> all the configuration settings. Charms and other files are uploaded to an S3 
> bucket.
> 

The only "other files" is one item which records the address of machine
0. This is done so that we don't have to make several API calls to EC2
to connect to ZooKeeper from the client.

> For a local deployment, Zookeeper and a Python web-server are run on 
> localhost. No new machines or containers are created at this point.
> 
> When a new service unit is deployed to EC2, a new machine is created.
> For local deployments, a new LXC container is created on localhost. Juju 
> considers local deployments to consist of a single machine, but with 
> multiple containers (container != machine). EC2 machines do not use 
> containers, although there are plans to change this ("LXC everywhere").
> 
> For every "machine", a "machine agent" process runs on that machine. It 
> monitors the Zookeeper service looking for changes to the set of units it is 
> supposed to be running.
> 
> Deploying a service writes the configuration for the new service and for a 
> single initial unit of it to Zookeeper. The machine agent notices the change 
> and deploys the actual charm (downloading it from S3 and running its install 
> hook).
> 

What you missed in this is where the new machines come from. On machine 0,
the provisioning agent watches for service units which are not assigned
to any machines.  If there are unused machines, it assigns those services
to those machines. If there are no unused machines, it provisions a new
machine for the unit.

> For each running unit, there is a "unit agent". The unit agent watches the 
> Zookeeper configuration for configuration changes and runs the charm's hooks 
> as needed. The unit agent runs within the LXC container (if any) along with 
> the actual running service.
> 
> Destroying a service or unit removes it from the Zookeeper configuration. 
> The machine agent then destroys the LXC container. [ what happens for EC2? ]
> 

For EC2 the unit agent is just shut down, but nothing else happens
on the machines. You can at that point use the 'terminate-machine'
command to remove it.  This separation is done in part to make sure we
don't accidentally blow away somebody's data.. it gives them a chance
to re-deploy or to backup their data.

> Adding a relation is done (as always) by updating the Zookeeper 
> configuration. The units for the services at each end of the relation are 
> invoked immediately and simultaneously. Typically, one end will need to run 
> first. e.g. when connecting wordpress to mysql, the mysql hook needs to run 
> first to create the database. If the wordpress hook runs before mysql has 
> updated the configuration it will detect this and exit(0). It will be 
> invoked again once mysql updates the settings. This can lead to race 
> conditions in charms if the case of running in the wrong order is not 
> considered.
> 

Its really easy to avoid these races though, because you simply exit if
the data you need has not been set yet. The guarantees around the hooks
allow for you to make the assumption that your hook will be invoked again
when the data has changed.

> 
> What needs access to what?
> 
> - The admin needs complete access to everything. Deploying new machines is 
> done directly by the admin's machine, but everything else is done indirectly 
> by updating Zookeeper.
> 

There are really two kinds of admins. There is the admin that does
the bootstrap and/or destroy-environment. Those calls do bypass the
provisioning agent and make direct calls to create or terminate machines
from an admin machine.

Then there are all other admins who only need access to read/write S3
and ZooKeeper. In theory, one could leave out the ec2 access creds from
environments.yaml after bootstrap, and give admins IAM limited credentials
which only have access to the S3 bucket for reading/writing. However,
since they can access ZooKeeper, and there are no ACLs, they could just
read the provisioning-capable EC2 credentials anyway. Perhaps still a
slightly better strategy as a backup of the machine's hard drive will
at least not contain the fully-capable credentials.

All admins also only needs SSH access to machine 0, which in turn allows
them to access ZooKeeper through a tunnel.

> - Machine agents need access to all units, so they can search for the ones 
> they want. They also need read-only access to S3 to download charms.
> 

Machine agents only need access to zookeeper really.

> - Units need read-only access to their service configuration, and read-write 
> access to their relations, whether providing or requiring.
> 

Service configs and relation data, yes.

> - The EC2 credentials are stored in Zookeeper too. [ why? ]
> 

I think I covered that. ;)

Its worth noting that right now, all agents and clients have full
unfettered access to all of ZooKeeper. This is recorded in these bugs:

https://bugs.launchpad.net/juju/+bug/813773
https://bugs.launchpad.net/juju/+bug/821074

Which have a branch nearing completion to implement full ACLs.

> 
> More questions:
> 
> I don't understand why the machine agents monitor the central Zookeeper. Why 
> doesn't the admin just contact a machine directly to request new units?
> 

The idea is that the admin's only job is to express intent for the
services and relationships, and then juju's agents work to make that
happen, or report errors. This makes it easier to build a highly
parallel system because all the parts work mostly independent of
one another.

> How are relations destroyed? Presumably the client end needs to stop using 
> it first somehow.
> 

The unit agent sees the deletion and calls the broken hook. This is why
the broken hook does not have access to 'relation-get' or 'relation-list',
because the relation is already gone. One has to use $JUJU_RELATION_ID
to clean up resources.

> What happens when multiple units are deployed for a service? For example, 
> connecting wordpress to mysql requires creating a new database, username and 
> password (once), plus e.g. updating the firewall rules (on each mysql unit). 
> What ensures we don't have every mysql unit creating a new user?
> 

Hooks are guaranteed to be executed in serial on a single unit. So on
the mysql server, only one joined/changed/etc. event will be handled at
one time. Thusly, this works fine:

# Determine if we need to create a new database
if slave and slave_configured and not broken:
  print "%s exists, assuming configuration done" % slave_configured_path
  sys.exit(0)

if not slave and not broken and not admin:
  # Find existing databases
  cursor.execute("show databases")
  databases = [i[0] for i in cursor.fetchall()]
  if database_name in databases:
    print "database exists, assuming configuration has happened already"
    sys.exit(0)

So only the first joined unit creates the needed database, username,
and password. There is only one relation-set bucket per local unit,
so we can't really create a username/password per remote unit without
giving them all to all of the remote units. Also, because we don't
do fine-grained network access control, the first grant for that
user/password is all that is needed, hence the 'sys.exit(0)'. Eventually
I think this charm should actually do a grant per remote host too (and
maybe firewall too).

Anyway, I think that should clear a few things up for you. Thanks for
the excellent questions and suggestions on the bug tracker too!