Django charms [Was: Re: [ubuntu-cloud] Ubuntu cloud use case]

Tue Sep 25 02:42:16 UTC 2012

Bruno,

Thanks for raising these questions. Having great Django support is very
important for Juju. I have outlined some of my thoughts below.

On 09/24/2012 10:30 AM, Bruno Girin wrote:
> On 21/09/12 20:45, Bruno Girin wrote:
>> On 20/09/12 19:32, Mark Mims wrote:
>>> <snip>
>>> Please by all means help with Django!!  I'm not sure what extra features
>>> / connections we should add to it at this stage and need input from
>>> peeps who're running it in production.
>> I will do. I'll probably start working on that in the next few weeks and
>> will tell you how I progress.
> Right, I've had a look and I need advice on how to proceed. If I look at
> what exists today, I find a django-rainbow charm [1], an initial
> Launchpad bug for a Django framework charm [2] initially worked on by
> Mike Nelson, an additional bug for a python-django with WSGI container
> charm [3] created by Patrick Hetu as a result of his idea on using a
> WSGI subordinate and a blog post from Michael Hall [4] who is looking at
> charming the app rather than the framework.
>
> Looking through this, django-rainbow [1] looks simple but doesn't suit
> my needs at it assumes Apache, MySQL and applications stored in
> Mercurial (I use nginx or gunicorn, PostgreSQL and have my application
> code hosted in git).

django-rainbow suits a very specific use case, as you mentioned.

>
> Looking at [2], [3] and [4], there are a lot of good ideas and reading
> on subordinate services [5], one way to get this to work would be:
>
> +- WSGI container as a principal service
>     +- django framework subordinate to wsgi container
>         +- django apps subordinate to django (could be any number of apps)

Right, a specific WSGI container should be the primary service for the
Django app, with a number of choices, including gunicorn, mod_wsgi, and
uWSGI. However, we don't want to model the Django framework as a
subordinate - we always want those bits to be installed: they're not
optional. Subordinates, as initially conceived, are about aspects where
there is a choice (so think of examples like logging or monitoring).

So this implies something like django-gunicorn, django-modwsgi, etc.,
similar to the tomcat6 and tomcat7 charms. uWSGI might be more
complicated, given that it can work with Cherokee and Nginx. However, I
don't think we have to support every possible combination; it's OK to be
opinionated.

The idea of using a subordinate to manage a specific Django app is an
intriguing option. However, stepping back, the question I would ask is,
would a Django stack benefit from having the lifecycle support of
subordinates? Also, is this really a good mapping between Django and
Juju? The Django-Mingus blog stack is a popular way to demonstrate
reuse, and despite being pretty simple, it uses 28 apps. It's not clear
to me that 28 subordinates is really going to work that well - although
it's certainly worth trying. And of course, non-demo Django stacks can
readily use many more apps.

But I do see a role for a set of Django apps being bundled as a
subordinate, especially when it may relate to other services. Consider
Haystack. You might want to relate this Solr, Xapian, and other search
engines. Deploying this subordinate also needs to rewrite settings.py.

>
> This would mean having a generic django charm that could be deployed
> into any WSGI container (Apache, nginx, gunicorn) plus simple charms for
> the apps. It's then a case of identifying what should be the
> responsibility of Django and what should be the responsibility of the
> apps. Django allows you to make apps agnostic from the way they are
> deployed so it should be feasible.

So if we don't have apps correspond to subordinates, then the other key
piece will be to ensure that all the bits for the apps are deployed to
the Django service units, and that this can be updated over time. PyPI
is an important source of truth, as well as launchpad, github,
bitbucket, and the vcs of the user's choice.

>
> When it comes to databases, Django uses a concept similar to the JavaEE
> data source, where you define the different data sources in the
> container and the apps just have to reference them by name. If you give
> no reference, it uses the database called "default". So what this means
> is the django charm should specify that it can have any number of "db"
> relationships, the default one being called "default". It should then be
> able to add new named db relationships to support applications that
> require more than one data source.

This is a key question, how to map database definitions in Django to Juju.

At a very simple level, Django stacks that use one database ("default")
can simply add a relation to the database service in question. The
config-changed hook can then ensure that settings.py has this
definition, including such info as host.

For multiple databases, it becomes tricky. First, there's a quite
relevant bug in Juju, see
https://bugs.launchpad.net/juju-core/+bug/1026422 re relation edge settings.

With that in mind, there are essentially two cases:

The Django stack is relating to multiple database services. This seems
the most plausible case, otherwise why not simply combine the relevant
schema under the same database and keep the management easy? Juju can
readily handle this case, so it becomes just a question, which Juju
database service corresponds to what database in Django's settings.py
(maybe DRY and use the same name, otherwise it requires config settings).

The Django stack wants to use the same given database service for
multiple databases. There's some degree of support for this in say the
mysql-admin interface, but from my perspective, this seems clunky, so I
don't see a reason to support this specific use case. Django makes it
easy to have apps with their schema in the same database. So let's just
use that functionality.

>
> The next question is how do you model the fact that a Django db
> connection could accept different interfaces? Ideally, if you do
> something like "juju add-relation django mysql", it should work out on
> its own that it needs to ensure the correct client libraries are
> installed for mysql on the django node and setup the correct
> configuration parameters.

Mark Mims recently wrote an email on this list re a charm's role
(https://lists.ubuntu.com/archives/juju/2012-August/001785.html). In a
nutshell, a Django charm should be able to relate via mysql or pgsql (or
other databases). If using pgsql, then it could relate to something like
pgpool, which if there was a charm for it, could act as middleware
between Django and an actual set of postgres servers.

The act of adding the relation could install the appropriate client bits
in the relation-joined hook.

>
> Last question (but not least!), how do you add nodes to a service
> defined like this with subordinate services? Are all the subordinate
> services extended to the new node as well?

Yes. That keeps things simple. And conversely, you cannot add units to a
subordinate.

Incidentally, there are other key pieces to deploying a scalable Django
stack, eg memcached. But in the case of caching infrastructure, this is
such a standard part of Django that it really should be part of the
Django charm.

Here's another good source for thinking about building scalable Django
stacks, https://github.com/jacobian/django-deployment-workshop - it has
the tutorial slides plus code using Fabric and Chef Solo to manage the
deployment of a Mingus blog.

- Jim

-- 
Jim Baker
Ubuntu Server team
jim.baker at canonical.com
jimbaker on freenode

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 551 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/juju/attachments/20120924/22e55641/attachment-0001.pgp>