API change: managing SSH keys in Juju

Jim Baker jim.baker at canonical.com
Wed Nov 23 22:53:21 UTC 2011

I would appreciate any comments on this proposed API change to Juju.


Currently managing SSH keys for Juju instances is a manual
process. The safe alternative is to use the process outlined by Scott
Moser in this blog post,
are non-safe alternatives too, and I suspect they are much more
commonly used.

Juju should manage this process in a secure fashion. Related bugs are
https://bugs.launchpad.net/bugs/802117 and

Managing SSH keys for ZooKeeper instances

The juju bootstrap step will generate SSH host keys (RSA, DSA, and
ECDSA) for the ZooKeeper instance(s), which are used as follows:

  1. Configure via cloud-init using `ssh_keys`, which itself is a
     dictionary that has this key to file correspondence::

            "rsa_private" : "/etc/ssh/ssh_host_rsa_key",
            "rsa_public"  : "/etc/ssh/ssh_host_rsa_key.pub",
            "dsa_private" : "/etc/ssh/ssh_host_dsa_key",
            "dsa_public"  : "/etc/ssh/ssh_host_dsa_key.pub",
            "ecdsa_private" : "/etc/ssh/ssh_host_ecdsa_key",
            "ecdsa_public"  : "/etc/ssh/ssh_host_ecdsa_key.pub"

     This process ensures that the SSH keys are set as desired on
     the new machine.

     .. note::

       Although the private key for each keypair is then part of the
       user-data per-instance metadata, and thus available to any
       processes running on the machine, this is only true of
       ZooKeeper servers. Ideally the ZK servers should not be used to
       run other service units (at least without LXC protection). In
       addition, the ZK secret (admin password of ZK) is also stored
       in instance metadata. So it doesn't get any worse with this

  2. The public keys for the ZK instances are then stored in the juju
     control bucket in the `provider-state` file (YAML formatted),
     under the key `instance_ssh_keys`. This a dict, with the key
     being the instance id; its value is identical to the public
     cloud-init contents of `ssh_keys` for each instance.

     The need to carefully update this file to enable future support
     of managing the set of ZK instances is not impacted by this

All providers in the Python implementation use a common implementation
to lookup ZK instances (`juju.providers.common.findzookeepers`). We
just need to modify `find_zookeepers` to return the
`instance_ssh_keys`, as well as the ZK machines.

The connection phase then checks for the existence of a
per-environment `known_hosts` file
(`~/.juju/${environment}/known_hosts`), creating as necessary, and
adds as necessary entries for the public keys of the chosen ZK machine
to connect to. This process ensures that these keys can be readily
shared across admin machines. (The security of this key propagation is
of course dependent on the `admin-secret` value in
`~/.juju/environments.yaml`; sharing this secret is what it makes it
possible to administer a Juju environment.)

With that place, juju commands that need to connect with ZK will now
work as desired.

Managing SSH keys for non-ZK instances

Non-ZK instances use the normal (directed by cloud-init) key
generation mechanism upon first boot of that instance.

In order to support `juju ssh` and `juju scp`, which connect directly
to a given machine, Juju needs a mechanism to register these SSH keys
as well. This is as done as follows:

  * Upon machine agent startup, the agent ensures that the contents
     (YAML formatted) of ZK node `/machines/${machine_internal_id}`, a
     dict, contains the key `ssh_keys`, with values the public key
     entries for the host as before. (Currently this ZK node only
     stores the provider-specific machine ID.)

  * Upon running the `juju ssh` or `juju scp` subcommands, the public
    keys are looked up for the given instance and stored if necessary
    in the per-environment known_hosts file.

Note that the storage policy means that for ZK instances, the public
SSH keys are stored in both the control bucket and ZK itself. This is
a minor redundancy.

Recycling machines

It is possible that a public IP address (in the form of a public DNS
name) will be recycled in a given environment for some providers (like
EC2), if suitably long-lived. The solution for this is to modify the
per-environment `known_hosts` file every time there's noted a
discrepancy with the control bucket (upon connect to ZK) or what's
stored in ZK (upon juju ssh/scp).

This update does not introduce a security issue because it's based on
what is known to be true for that instance and its corresponding DNS

Otherwise, there's no attempt (or need) to clean up the
per-environment `known_hosts` file.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 554 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/juju/attachments/20111123/eaf5f86a/attachment.pgp>

More information about the Juju mailing list