[Maas-devel] Database Migrations and backports

[Raphaël Badin]

>> * We must be very careful when adding new migrations that nobody
>> else is working on one too, because one of you must base your
>> migration off the other's

The reason why this situation is bad (but not catastrophic) is that when 
we do this, South will use the state of the whole db, as recorded at the 
end of the last migration file to compute the changes to models.  If two 
migration number 10 where added (10a and 10b), one of them will be 'the 
last one' (say, 10a).  10a won't have the record of what happened in 10b 
and thus, next time one creates a migration, the changes in 10b will be 
seen by South as recent changes.

The solution is to manually create an empty migration (./bin/maas 
schemamigration --empty maasserver merge_models) after having merged the 
two branches.  This new migration will contain an accurate description 
of the state of the models.

>> * When amending old release branches (e.g. lp:maas/1.2) we *cannot*
>> add new migrations to it on their own, we *must* add it to trunk
>> and backport all the missing trunk migrations.  If this is not
>> done, it will break upgrades between releases.

That situation would be much more problematic as it would block us from 
moving forward in the future.  I think we can reasonably say that any DB 
migration done on the 1.2 branch must be done on trunk as well.

> 1) Create 0043 in branch 1.2. Merge that into trunk which also had an
>     0043 patch and an 0044 patch.
>     Upgrade worked correctly from 1.2 => trunk.

This is the first situation I described.  The only problem might be when 
someone else tries to create another migration.  (Because the changes 
from your migration 0043 are not recorded in the state of the db 0044.)

> 2) Trying to simply rename 0043 => 0045 in trunk fails because the DB
>     doesn't think it has the patch and tries to apply it twice

Internally, South keeps a record of the name of the applied migrations 
(the table is named 'south_migrationhistory') so renaming a migration is 
not an option.

> 3) Creating the patch as 0045 in trunk, and then putting just that into
>     1.2. It successfully migrates 1.2, adding the 0045 patch. However it
>     fails to apply 0043 and 0044 when upgrading to trunk because they
>     appear out of order. "migrate --merge" works, but isn't our default
>     setting.
>
> 4) Obviously creating it on trunk and merging all db migrations works.
>
>
> So while what you are recommending (4) works, I'd like to point out
> that it means we can never land anything on trunk that you don't want
> to backport to 1.2 if we ever want to land anything else from trunk on
> 1.2.
>
> I'd like to point out that (1) seemed to work quite well in my
> testing. So we might consider that instead of creating patches on
> trunk, and having to backport all pending DB migrations back to 1.2,
> we might want to try creating DB migrations on 1.2 and merging up into
> trunk where we need it.

Actually, (1) is exactly the same situation as (3), it just breaks for 
different use-cases.  Imagine someone using a package with the 
migrations 0043 and 0044.  When the fix containing your new migration 
0043 will be available and packaged, it will simply break the upgrade.

I think we want to be able to land DB migrations in trunk without having 
to land them in 1.2.  For that to work, the only way is to:
- make sure that *all* the DB migrations are landed on trunk.  (Some of 
them will be missing in 1.2)
- change the packaging to use '--merge' when applying the migrations. 
The only problem with that would a semantic problem between migrations 
but that should be picked up during the testing phase.

R.