[storm] Storm+twisted. store.invalidate() and sqllite

[Thomas Hervé]

Le lundi 13 octobre 2008 à 13:43 +1100, Justin Warren a écrit :
> I did some playing around with the twisted-integration branch of Storm
> on the weekend, and it's really ace, so thanks heaps for all the work on
> it.
> 
> One piece has me confused, though.
> 
> There's a monkey patch of storm.store.invalidate() to replace it with an
> empty function. Unfortunately, this breaks lazy loading, and the use of
> the __storm_loaded__ hook, which I find really handy and use quite a
> bit.
> 
> The only reason I can find for this seems to be a test that fails
> against a SQLite database:
> tests.twistorm.DeferredStoreTestCase.test_find_and_rollback
> 
> This test fetches an object from the SQLite database, and then does a
> rollback(). It then checks to see that the objects are still the same,
> but if the original storm.store.invalidate() is left in, the test fails
> with a sqlite error:
> 
> sqlite3.ProgrammingError: SQLite objects created in a thread can only be
> used in that same thread.
> 
> I'm not sure I understand, but it seems that PySQLite does some wild and
> crazy things with transactions, and Storm works around them by disabling
> the dodgy hacks. I'm guessing that the way the twisted wrapper uses a
> dedicated thread for the store access and its defer_to_thread() means
> that objects grabbed from a store end up in the wrong thread somehow,
> which causes this error.
> 
> Is this a SQLite only problem? If so, commenting out store.invalidate()
> seems a bit heavy handed. The test comment seems to suggest that
> commit() and rollback() have to be done manually with SQLite anyway, so
> perhaps this means you need to know that you have to re-fetch the object
> from the database manually, rather than rely on lazy-loading?
> 
> Sorry if I'm missing something obvious here. This is all deep-magic to
> me.

It is indeed deep-magic :). The problem with invalidate is not a SQLite
only problem. In fact SQLite is really handy here, because it tells us
that a connection is used in a different thread (here, the main one). So
the problem is actually that we have a query in the main thread instead
of the store thread, which we must not do at all (it's the purpose of
the wrapper, after all).

In your case, I guess removing the invalidate hack should be a good
idea. It doesn't fix the background problem with lazy values, which is
not only present at this point. Overall, the resolution of lazy values
is the main problem not solved currently by the twisted wrapper, and one
of the reason it may not be possible at all to do it, because it happens
behind the scene.

I think I mainly introduced this workaround for the Elisa guys at the
beginning. Every other solution is welcome... Relying on fetching the
object again may be a good idea, but it needs to crash the object
explicitely: you have an error in SQLite, but the same process in
Postgres will just make a query in the main thread silently. That may be
the best solution I can think of now: raise a big explicit error when
you're trying to access an invalidated object, to force users to use
store.get() again.

-- 
Thomas