Stormified importer has introduced unintended change in data being written to sqlite

[Max Bowsher]

On 21/04/12 23:58, Max Bowsher wrote:
> On 21/04/12 18:44, Max Bowsher wrote:
>> The stormification has introduced an unintended change in how the data
>> for testaments and revision ids are stored in the sqlite DBs.
>>
>> They are now being stored as binary blobs rather than strings.
>>
>> The importer is currently stopped whilst I investigate further.
> 
> This turns out to be fairly bad, since it has allowed what were supposed
> to be primary keys to be duplicated (once as a blob, once as text). I'm
> working on manually understanding the consequences and repairing the DB
> consistency.

I've repaired the database, mainly by making lots of use of:

UPDATE foo SET bar = CAST(bar AS text) WHERE typeof(bar) = 'blob';

However, in two tables (revids and should_retry), it wasn't that simple,
since the above technique would give a key uniqueness violation.

For those, I manually consulted the data and merged it appropriately. In
the revids table, this only affected 4 packages (6tunnel aalib acct ace).


Because of the above woes and because of the deadlocking issue, and some
discussion with Jelmer on #bzr, I've reverted Storm out of lp:udd.

I guess we can put it back in, in time, but this time let's be really
really sure we don't create operational issues.

With the above accomplished, the importer is now happily alive, and
chugging through the extreme backlog from the previous malfunction at a
concurrency level of 12 threads.

Max.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-distributed-devel/attachments/20120422/76729f90/attachment.pgp>