[storm] Threading question

[Richard Boulton]

Gerdus van Zyl wrote:
> Ok silly me, it's wasn't a storm problem only the way storm uses
> sqlite. There is a sqlite connection parameter check_same_thread that
> by default is set to true that was giving me the exception when used
> by different threads.
> 
> I had to change the storm source (sqlite.py) class SQLite, connect method from:
>         raw_connection = sqlite.connect(self._filename, timeout=self._timeout,
>                                         isolation_level=None)
> to
>         raw_connection = sqlite.connect(self._filename, timeout=self._timeout,
> 
> isolation_level=None,check_same_thread=False)
> 
> If this small change can be included in some manner that would be great.
> So now I can use one store object in my program. It works correctly
> since it never uses the store and sqlite concurrently, just from
> different threads.

Are you sure this is safe to do?  As I understand it, it is often not 
safe to use SQLite objects in different threads than the thread they 
were created in, even if you ensure that they are not accessed 
concurrently.  The option not to check is simply a way of reducing the 
overhead in the case where you are sure that access is being done safely.

This is certainly what the pysqlite documentation implies:

   **check_same_thread** - SQLite connections/cursors can only safely be
     used in the same thread they were created in. pysqlite checks for
     this each time it would do a call to the SQLite engine. If you are
     confident that you are ensuring safety otherwise, you can disable
     that checks by setting check_same_thread to False.

Also see http://sqlite.org/faq.html#q6 - the problems involve 
limitations of fcntl, and it is safe to move connections between threads 
in only certain circumstances (SQLite version at least 3.3.1,
compiled with SQLITE_THREADSAFE = 1, and no fcntl locks are currently 
held by the connection).  If you're going to allow this in storm, you 
should probably be checking that these constraints are met at the storm 
level before you allow the threads to be moved, rather than just turning 
off the check and hoping.

It's probably safer just to use a separate pool of connections for each 
thread (using thread local storage, probably).

-- 
Richard