API change: adding a new config 'url' type

Jim Baker jim.baker at canonical.com
Wed Jan 25 20:28:32 UTC 2012


I would appreciate any comments on this proposed API change to Juju.


Overview
--------

A common configuration requirement for a service is that the service
units have access to the identical version of a given file. For
example, this functionality can be used to configure the logo for a
blog.

This proposal supports this requirement by adding a new config type,
`url`, to represent the source of such files; changes in the behavior
of `juju set` / `juju deploy` to manage the upload of files to
environment storage; and `config-get` to retrieve a Juju
environment-specific URL for the given file.

Similar functionality is seen in the `ch_get_file` function, which is
part of charm-helpers; however, in this case, this function is
primarily used for downloading non-Debian packages, then verifying a
corresponding cryptographic hash. This functionality overlaps with the
proposal here, and could take advantage of it.

This proposal is in response to the feature request
https://bugs.launchpad.net/juju/+bug/814974


Configuration file change
-------------------------

Config files (`config.yaml`) for a charm will now support the
specification of a `url` type. Example::

  logo:
    type: url
    default: https://example.com/default-image
    description: |
      Logo used for the blog

Defaults for url config options may only use `http` or `https` as a
protocol, and they must be a valid URL. This restriction is to prevent
inadvertent leaking of local files.

However, the user can also specify (via `juju set` or `juju deploy`)
arbitrary URLs as supported by the `curl` command. For example this
usage will upload the icon image from a local file on the admin's
computer::

  juju set myblog logo=file:///home/bob/images/blog.jpeg

In addition, the Juju user can also specify arbitrary `curl` options,
by prepending them to the URL just as it would be done with `curl`
normally::

  juju set myblog logo='--user user:password ftp://icons.example.com/images/blog.jpeg'

Regardless of how the configuration option is set (default or not), it
can be retrieved as set by using `juju get`::

  $ juju get myblog
  service: myblog
  charm: local:series/wordpress-3
  settings:
    logo:
      description: URL of the logo used for the blog
      type: url
      value: --user user:password ftp://icons.example.com/images/blog.jpeg
      version: a-version-or-file-hash-depends-on-file-storage

The version key depends on the underlying file storage. For example,
for S3, this could be the version ID associated with the version of
the file.


File uploading
--------------

The Juju client is responsible for uploading the file, either during
initial deployment of the service or afterwards by setting the
corresponding config option.

Uploading occurs by calling `curl` to retrieve the file, which is then
uploaded by using the provider API. For the Python implementation,
this would use `FileStorage.put`.

This behavior means that the CLI must wait until the appropriate
file(s) are uploaded into the environment file storage when using
`juju deploy` or `juju set`; this behavior occurs regardless whether
file option(s) have a default value or not.

Config setting changes are not written until the file has completed
uploading. This sequencing ensures that `config-changed` hooks will see
the new file (or new version of the file) when they fire and use
`config-get`.


File retrieval
--------------

Charm hooks may access a URL pointing to the file in the environment
file storage by using `config-get`. For example, for the EC2 provider,
`config-get` will return an authenticated URL to the S3 object for the
file.

Please note, retrieved files are not checked against a cryptographic
hash. However, it is certainly possible to specify the desired hash as
another config option, and to perform the check on the service unit
itself when the file is retrieved. This operation can be readily
implemented in charm-helpers.

Note that `config-get` does not retrieve the same value as `juju
get`. This ensures that the file to be retrieved by the hook is
identical across service units, unless a subsequent version is
uploaded with `juju set`. For the Python implementation, this uses
`FileStorage.get_url` to determine the necessary URL.


File storage
------------

The name of the object in the environment storage is not specified,
but instead left to the implementation. However, for the scenario of
the EC2 provider, it should be feasible for a user using the S3
administrative interface to find the file for debugging purposes.

It is important to use new versions of the file instead of overwriting
due to eventual consistency guarantees for overwrite in S3, Swift, and
similar storage providers.

One possible naming scheme for such files for S3 is as follows::

  config-<SERVICE_NAME>-<SETTING_NAME>

For S3, the version of the file is captured as part of the `PUT` via
the returned header `x-amz-version-id`; it can be retrieved by adding
the version id the query. The txAWS API supports the necessary
functionality (although it's possible that appending `/` to the URL
path will break the use of version id for S3). Alternatively, a hash
can be computed to use the following naming scheme, and this approach
may also provide more portability across storage providers::

  config-<SERVICE_NAME>-<SETTING_NAME>-<FILE HASH>

Orchestra and the local provider do not have these consistency issues,
of course, and thus they do not need to use versioning.


Garbage collection
------------------

A combination of timestamps and files not reachable from config
settings can determine what is garbage. Note that a newly uploaded
file will, for a brief period of time, not be linked from the config
in ZK, due to the need to upload the file (or a new version) first,
then write the new version of the config settings.

Rather than using ad hoc GC, this should be part of a general GC
mechanism for Juju, including the cleanup of garbage ZK nodes.


Destroying a service or environment
-----------------------------------

Destroying a service or environment should result in deleting any
corresponding stored config files. Ideally, destroying the environment
should simply result in the deletion of the bucket itself.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 554 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/juju/attachments/20120125/f707a1d8/attachment.pgp>


More information about the Juju mailing list