Best practices for "fat" charms

Michael Nelson michael.nelson at canonical.com
Wed Apr 2 07:41:25 UTC 2014


On Tue, Apr 1, 2014 at 10:05 PM, Matt Bruzek
<matthew.bruzek at canonical.com> wrote:
> Thanks Jorge,
>
> Not sure we want to call them "fat" charms, maybe "enterprise" charms.  Here
> is my approach when making a charm work on the enterprise or limited
> networks.
>
> 1) Find out what hook downloads the packages that we are unable to access
> (wget, curl, or special ppa repositories).  The enterprise network will
> block these requests often resulting in a charm hook failing.
> 2) Download the necessary packages from system that has access.
> 3) Upload the packages to the locked down system, copying the packages to a
> directory on the local charm.
> 4) Edit the local charm hooks to check for the package in the local
> directory first and if that does not exist, the charm would continue to
> download the files (using wget, or curl, or custom ppa).
>
> I believe we could provide a charm-tools method that does something like
> this and we could use this in charms to create "enterprise" charms that are
> able to be used on limited network environments.
>
> However this creates an interesting problem that I have not figured out a
> good way to resolve yet (your feedback requested).

Hi Matt,

We've found and used a few solutions when deploying charms internally
- as we don't deploy services that rely on PPAs or external resources.

One solution which we use in the elasticsearch charm is to rely on
repositories and ensure the config has sensible defaults. This works
well for us because our IS team maintains private repos which we can
use for deployments, and in this case, elasticsearch.org have their
own repository - but it won't work when either of those aren't true
for your situation.

You'll see in that config [1] that the default apt-repository is the
one from elasticsearch.org - perfect for the charm-store, but when we
deploy that we override it with our custom internal repository.
Similarly, there's an option in the config whether to install ansible
(which is used within the charm) from a ppa which defaults to yes for
the charmstore, but we override that. For us, this has been the best
way to bring the functionality to the public charm while allowing us
to deploy it internally.

If you don't have the option of using repositories as above, you can
do something similar to the suggestions of others above, but to be
explicit, the following in the install hook:

{{{
if {{ charm_dir }}/files/my-dependency.jar doesn't exist:
    download it from public url to /srv/foo/deps/my-dependency.jar
else:
    copy it from {{ charm_dir }}/files/my-dependency.jar to
/srv/foo/deps/my-dependency.jar

install or use /srv/foo/deps/my-dependency.jar
}}}

So it'll work from the charm store, but a user of your charm can also
ensure that it exists in the charm before they deploy it as part of
their build/deploy process. We used to do this with our elasticsearch
charm before elasticsearch.org publicised their repos.

Another option that you have, which we also use but for different
reasons, is to add an exec.d directory to the charm only when
deploying it in an enterprise environment, and ensure that the first
thing the charm does in the install hook is to call
charmhelpers.payload.execd.execd_preinstall(), like [2]. This enables
people to customise the unit, or put specific resources in place using
whatever method they need in their environment. IMO, all charms should
be calling charmhelpers.payload.execd.execd_preinstall() within the
install hook, so that the unit can be customised to some degree in
enterprise environments - but we need to be careful that it isn't used
to do something that should be done by the charm where everyone can
benefit.

Finally, another situation we have is for a generic wsgi-app charm
which we've started using, where we (will) deploy different
code-bundles depending on which service it's being used for (I've not
yet made this one public yet as it's not very juju-like to have a
wsgi-app charm that can be used for different services, although it
may make sense with bundles). In this case, we didn't want to use any
of the above because we don't want to have to 'upgrade-charm' to
deploy a new version of our application. So instead, the charm config
has two options - code_assets_uri and a code_archive (a relative path
from the code_assets_uri), like [3]. This could be a public url to a
git tarball, or in our case, the url to an internal swift container to
which we've published the build asset during an automated deploy test.
With that we can update our application code with a `juju set` (as
well as support rolling upgrades quite easily using a single juju run
command to update individual units to the latest code).

Sorry for the brain-dump nature of the above, hope it's got some
helpful bits :-)
-Michael

[1] http://bazaar.launchpad.net/~michael.nelson/charms/precise/elasticsearch/trunk/view/head:/config.yaml
[2] http://bazaar.launchpad.net/~michael.nelson/charms/precise/elasticsearch/trunk/view/head:/hooks/hooks.py
[3] http://paste.ubuntu.com/7193187/

>
> If the packages exist within the charm the URL will never be used.  Some
> charms allow the user to configure the download URL and sha1sum of the
> package.  Other charms do not allow this level of customization.
>
> For charms that have a config option for URL we could change it to also
> accept a file:// transport and some kind of $CHARM_DIR variable and use
> http:// or https:// for normal URLs.  But what to do with the charms that do
> not allow the URLs to be configurable, or the charms that use a custom ppa
> repository?
>
>    - Matthew Bruzek <matthew.bruzek at canonical.com>
>
>
> On Tue, Apr 1, 2014 at 2:07 PM, Jorge O. Castro <jorge at ubuntu.com> wrote:
>>
>> Hi everyone,
>>
>> Matt Bruzek and I have been doing some charm testing on a machine that
>> does not have general access to the internet. So charms that pull from
>> PPAs, github, etc. do not work.
>>
>> We've been able to "fatten" the charms by doing things like creating a
>> /files directory in the charm itself and putting the
>> package/tarball/jar file in there, and given the networking issues
>> that we might face in production environments that we should start
>> thinking about best practices for having charms with payloads instead
>> of pulling from a network source.
>>
>> Marco has some ideas on how we can generalize this and he will respond
>> to this thread.
>>
>> --
>> Jorge Castro
>> Canonical Ltd.
>> http://juju.ubuntu.com/ - Automate your Cloud Infrastructure
>>
>> --
>> Juju mailing list
>> Juju at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>



More information about the Juju mailing list