Quick analysis of 'stale' transactions

[John Meinel]

We're trying to look at reasons why we end up with some stale transaction
ids in the txn-queue. I did find a machine with 18 entries in its txn
queue, all of which are '6: applied'.

I ended up discovering that SetLinkLayerDevices and SetDeviceAddresses are
both causes of a fair number of assert-only transactions (especially
against the model object).
https://bugs.launchpad.net/juju/+bug/1677619

I also found out that it isn't just assert-only transactions. Transactions
that have an "update" field, but are ultimately updating the content to the
content that is already there *also* end up sticking around.
SetAgentVersion is guilty of this.

In the production database, I found a machine with 18 transactions in the
queue, but worse I found 4000 transactions in the model queue.

A couple of interesting aggregation queries:
   db.models.aggregate([
    {$project: {_id: 1, "size": {$size: "$txn-queue"}}},
    {$sort: {"size": -1}}])

this gives you a descending list of object ids sorted by the number of
entries in the ten-queue.

  db.models.aggregate([{$match: {"_id": SPECIFICID}},
     {"$project": {"_id": 0, "q": "$txn-queue"}},
     {$unwind: "$q"}, {$sample: {size: 20}}])

This lets you get a random subset of 20 ids out of one of those 4000 long
queue so you can go around poking at what those transactions are doing
without getting just the beginning or just the end.
Ideally it would actually do something to transform the transaction + nonce
ids into just transaction ObjectIds so that you could easily do deeper
queries, but its a start. (I can print those and transform them in an
editor.)

John
=:->
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20170330/8a447cc4/attachment.html>