Pruning the txns collection

[Gustavo Niemeyer]

Hey Menno,

I'm copying the list to ensure we have this documented somewhere for future
reference.

You are right that it's not that simple, but it's not that complex either
once you understand the background.

Transactions are applied by the txn package by tagging each one of the
documents that will participate in the transaction with the transaction id
they are participating in. When mgo goes to apply a transaction in that
same document, it will tag the document with the new transaction id, and
then evaluate all the transactions it is part of. If you drop one of the
transactions that a document claims to be participating in, then the txn
package will rightfully complain since it cannot tell the state of a
transaction that explicitly asked to be considered for the given document.

That means the solution is to make sure removed transactions are 1) in a
final state; and 2) not being referenced by any tagged documents.

The txn package itself collects garbage from old transactions as new
transactions are applied, but it doesn't guarantee that right after a
transaction reaches a final state it will be collected. This can lead to
pretty old transactions being referenced, if these documents are never
touched again.

So, you have two choices to collect these old documents:

1. Clean up the transaction references from all documents

or

2. Just make sure the transaction being removed is not referenced anywhere

I would personally go for 2, as it is a read-only operation everywhere but
in the transactions collection itself, to drop the transaction document.

Note that the same rules here apply to the stash collection as well.

Please let me know if you run into any issues there.



On Tue, May 12, 2015 at 9:21 PM, Menno Smits <menno.smits at canonical.com>
wrote:

> Hi again,
>
> In response to the current production Juju issue, I've been tasked with
> adding something to Juju to keep the size of Juju's txns and txns.log
> collections under control so that they don't grow unbounded.
>
> The ticket is here: https://bugs.launchpad.net/juju-core/+bug/1453785
>
> Naively, one might think that transactions could be removed if they were
> (say) over a week old and marked as either applied or aborted but of course
> it's not that simple[1]. I must admit that I don't completely understand
> why this is case, even when I look at the code for mgo/txn. How does a
> pending transaction end up depending on the details of an applied
> transaction?
>
> Given that a typical Juju system has no maintenance window and there's
> (currently) no way to put a Juju system into a "read-only" mode can you
> think of any practical way for Juju to prune the txn and txn.stash
> collections?
>
> Any ideas would be most helpful.
>
> - Menno
>
> [1]
> http://grokbase.com/t/gg/mgo-users/13cj7c6kxt/when-to-delete-db-transaction
>
>


-- 
gustavo @ http://niemeyer.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20150513/ef5a9b82/attachment.html>