[storm] using twisted-storm

[James Henstridge]

Please use the "reply to all" feature of your mail client, so that the
mailing list gets copied on the replies.  I'm happy to help you in a
public forum where others can benefit, but less inclined to do so in
private.

On Mon, Apr 20, 2009 at 9:43 AM, Lee Connell <lee.a.connell at gmail.com> wrote:
> ok, so what if i set the threadpoolsize to only 1, this way all my db calls
> will reuse this thread and i won't have to worry about thread
> synchronization against the orm from multiple threads trying to access it.
> Would this make sense?

I'd leave the maximum pool size above one, but setting the minimum
pool size to your average amount of concurrency is probably a good
idea.

If there are other processes accessing the database, it is always
possible to have a serialised transaction fail.

If the database code you defer to a thread has no side effects other
than changes to the database, it is pretty easy to implement a retry
policy.  A decorator like the following could help:

    from psycopg2.extensions import TransactionRollbackError
    from storm.exceptions import DisconnectionError, IntegrityError

    RETRY_ATTEMPTS = 3

    def retry_transaction(function):
        def wrapper(*args, **kwargs):
            attempt = 0
            while True:
                attempt += 1
                try:
                    return function(*args, **kwargs)
                except (DisconnectionError, IntegrityError,
                        TransactionRollbackError), exc:
                    if attempt >= RETRY_ATTEMPTS:
                        raise # tried too many times
        return mergeFunctionMetadata(function, wrapper)

Wrap that outside the write_transaction decorator, and it will retry
the transaction in the common cases where you'd want that.  If you're
using something other than PostgreSQL, you'd need to work out what
exception is raised on transaction serialisation errors instead of
TransactionRollbackError.

James.