[storm] RuntimeError with Storm(twisted integration)

[Drew Smathers]

On Fri, Jul 17, 2009 at 3:14 AM, Daniel
Yang<daniel.yang.zhenyu at gmail.com> wrote:
> Hi team,
> I am doing some experiments on  Storm(twisted integration) and Nevow to see
> how they can work together.
> I went across a RuntimeError when I kept refreshing a page. It said:
> ------------------------------------------------log--------------------------------------------------------------
> 2009-07-17 14:58:42+0800 [-] [Failure instance: Traceback: <type
> 'exceptions.RuntimeError'>: Resolving lazy values with the Twisted wrapper
> is not possible right now! Please refetch your object using
> store.get/store.find
> --- <exception caught here> ---
> /Library/Python/2.5/site-packages/Twisted-8.2.0_r27097-py2.5-macosx-10.5-i386.egg/twisted/internet/defer.py:323:_runCallbacks
> /Users/Daniel/Documents/workspace/NovaWeb/src/web/FromLocation.py:47:cb_all
> /Library/Python/2.5/site-packages/storm-0.14-py2.5-macosx-10.5-i386.egg/storm/properties.py:60:__get__
> /Library/Python/2.5/site-packages/storm-0.14-py2.5-macosx-10.5-i386.egg/storm/variables.py:178:get
> /Library/Python/2.5/site-packages/storm-0.14-py2.5-macosx-10.5-i386.egg/storm/event.py:53:emit
> /Library/Python/2.5/site-packages/storm-0.14-py2.5-macosx-10.5-i386.egg/storm/twisted/store.py:238:_resolve_lazy_value
> ]
> ------------------------------------------------log--------------------------------------------------------------
> (latest storm and Nevow, Twisted.Python 2.5.1, Osx 10.5.7)
> I am using StorePool to manage connections:
> ------------------------------------------------code--------------------------------------------------------------
>
> from storm.databases.sqlite import SQLite
>
> from storm.uri import URI
>
> from storm.twisted.store import StorePool
>
> database = SQLite(URI('sqlite:///test.db'))
>
> pool = StorePool(database, 5, 10)
>
> pool.start()
>
> ------------------------------------------------code--------------------------------------------------------------
>
>
> On this page, I have 2 renders to access database(sqlite) with storm(People
> list and Shop list):
>
> ------------------------------------------------code--------------------------------------------------------------
>
> def render_PeopleList(self, ctx, data):
>
> pat = inevow.IQ(ctx).patternGenerator('PeopleItem')
>
> def cb_find(results, s):
>
> results.config(offset=0, limit=10)
>
> return results.all().addCallback(cb_all, s)
>
> def cb_all(items, s):
>
> #recycle the store.
>
> pool.put(s)
>
> ps = []
>
> for item in items:
>
> p = pat()
>
> p.fillSlots('nickname', item.nickname)
>
> ps.append(p)
>
> return ctx.tag[ ps ]
>
> def cb_get(s):
>
> return s.find(People).addBoth(cb_find, s)
>
> return pool.get().addCallback(cb_get).addErrback(lambda error:
> log.msg(error))
>
> ------------------------------------------------code--------------------------------------------------------------
> and shop render is pretty much like this.
>
> In file "storm/twisted/store.py:238", I notice the _resolve_lazy_value
> function does nothing but raising an error.
> Do anyone have any idea of what is the problem?
> Thanks.


This is the problem:

  pool.put(s)
  ... doing something with object from store `s'

Once you put a store back in the pool this will effectively rollback
the store.  Now being in a new transaction, referring to an attribute
on the object will try to load the most recent data from the store to
enforce "serializable" semantics.  twisted-integration can't manage
this because attribute access isn't can't be deferred in the store's
thread -- without necessitating a very awkward API.

I think some good practices to avoid lazy value errors are:

1. Do everything you need to with objects before putting store back in
pool and then discard those objects
2. Re-fetch objects if you have done any inserts/updates (in same tx)
before referencing attributes on them
3. If you need to keep references lying around (caching as an
example), create read-only version of those objects which aren't
attached to store.

Lastly, unrelated to you core problem, but this code looks like a unit
test for twisted-integration. The tests are written to be compatible
with older versions of Python, so you may consider using
inlineCallbacks() to make your life easier if your deployment target
is 2.5>.  There is also a transact() method which takes care of
putting the store back in pool after your function exits.  An example
using inlineCallbacks() and transact():

from twisted.internet.defer import inlineCallbacks, returnValue

@inlineCallbacks
def transaction(store, offset, limit):
    dr = yield store.find(People)
    dr.config(offset=offset, limit=limit)
    items = yield dr.all()
    for item in items:
        ... do stuff with item
    # commit tx if you've changed objects
    yield store.commit()
    returnValue(ctx_tag[ps])

pool.transact(transaction, 0, 10)

-- 
\\\\\/\"/\\\\\\\\\\\
\\\\/ // //\/\\\\\\\
\\\/  \\// /\ \/\\\\
\\/ /\/ / /\/ /\ \\\
\/ / /\/ /\  /\\\ \\
/ /\\\  /\\\ \\\\\/\
\/\\\\\/\\\\\/\\\\\\
               d.p.s