Moving udd to django

[Vincent Ladeuil]

>>>>> James Westby <jw+debian at jameswestby.net> writes:

    > Hi,
    > I think there are a few reasons that we should consider moving udd to
    > django (more on what this actually means later.)

    >   1. We want to have remote commands so that we don't have to ssh
    >   in and run scripts for common tasks

Long term plan for udd, also a lot of effort[1].

    >   2. There are scripts in cron that generate html pages that are
    >   served. It would be nice to have these generated on request so
    >   that there is no latency.

+1. They also need to be part of the deployment instructions (they
aren't so far since they are re-spawn from the crontab, making failures
rare and without (known) fallouts).

    >   3. It would also allow for starting to move udd to an SOA, or at least
    >   make it easier.

Not a concern for udd so far.

    >   4. It would be nice to have a query builder, rather than all the
    >   hand-written sql.

Not a big problem for udd so far.

    > So, what exactly do I mean by moving to django?

    >   * Moving all database access to go through django's ORM.

    >   * Moving all HTML generation to be done in django views.

    > This has fairly wide-ranging consequences though, and obviously
    > needs to be broken down in to manageable steps.

    > Here's my idea for what those steps should be.

    > 1) Install python-django in production
    >    - So that it is there when it is needed.

This should be cheap. On the other hand, if newer versions are needed
(which would be known during the dev/test phase) this will also be
useless.

    > 2) Create a django project in the udd codebase.
    >    - Providing the skeleton to use, but not actually changing any
    >    behaviour

    > 3) Write django management commands for each current script that call
    > the same code.
    >    - This gets them running in the django env, but without changing
    >    anything else.
    >    - At this point we'll want to deploy, and change the init script and
    >    cronjobs to run the new scripts.

    > 4) Configure the django project to use multiple databases corresponding
    > to the existing database.
    >    - At this point we'll want to deploy and run some sanity checks to
    >    make sure that django can read the dbs.

    > 5) Pick a db table and write a django model to match. Convert users of
    > that table to call through the query builder. Repeat.
    >    - I think this will be ok table-at-a-time, but we should obviously
    >    test well.

    > 6) At this point we'll be using django just as an ORM, but can turn on
    > the admin site if we want.

    > 7) We can also start to write django views to replace the
    > html-generating scripts. Once they are written we can deploy and request
    > that apache actually run django and serve the views, and then turn of
    > the scripts.

    > 8) Then we'll be able to start adding forms to the views to accomplish
    > common tasks.

    > That's a fairly long road, but it is incremental, and the changed code
    > deployed at each point can be fairly small.

    > What do you think? Is this worth doing?

Frankly ? This certainly sounds like a very good plan.

I also think udd doesn't need any of that in the short term.

Now, if you do it for pkgme, we will certainly be interested in
cannibalizing part or all of it.

In the mean time, the less disruptive changes we do to udd, the more we
can focus on the import failures which is where the paint points are.

What I mean is that forcing the use of the same code base for two
projects with different goals sounds like a sure way to trigger failures
for the other project without benefits for the project needing the new
features. We've already encountered such issues at a small scale and
until the test suite become rock-solid, I fail to see while we won't
encounter new issues.

I.e.: It sounds easier to separate common parts from specific parts in
the actual udd code base while starting a *new* project than doing so in
the udd code base that is already in production. Once those parts are
clearly separated, the package importer can look at integrating the new
and well-separated udd.

Both projects will benefit from this separation: 

- pkgme can go ahead without caring for udd needs, as long as the actual
  code base evolve by separating old features from the new ones with a
  reasonable effort to make the new ones easier to integrate.

- udd won't have to integrate incremental steps that bring no added
  value but still cost deployment/debugging time or do so *only* when a
  tangible benefit emerge *and* addresses issues.

    Vincent

[1]: At least with the actual code base, it's a lot of effort to
     understand what is strictly needed and what should be fixed in the
     import-package stack.