[storm] Storm and DBUtils

[Sean Allen]

On Mar 1, 2008, at 12:35 PM, Christopher Armstrong wrote:

> On Sat, Mar 1, 2008 at 12:33 PM, Christopher Armstrong
> <radix at twistedmatrix.com> wrote:
>> On Sat, Mar 1, 2008 at 12:30 PM, Sean Allen <sean at ardishealth.com>  
>> wrote:
>>> What I'm interested in is not closing out the connections period.
>>> If I read the DBUtils code correctly ( and maybe i dont because i  
>>> put
>>> more than 30, less than 60 minutes
>>> in on this ), database connection have to be explicitly closed via
>>> Persistent db,
>>> standard close calls etc wouldnt close the connection whereas
>>> storm does close the connections which then have to be reopened.
>>
>> No, Storm does not close the connections if you use it in the normal
>> way: have one Store per thread, and keep that Store around for the
>> lifetime of the thread.
>
> ... Assuming you have a thread pooling system. Most threaded web
> application servers these days do, so it should be fine.

Which requires a rewrite of our code.

Apache::DBI from perl does the following:

monkeypatches the connect and close methods in the DBI classes
so that a close doesnt really close and an open returns the
already open connection so you can do stuff like this:

( in pseudo )

some_method_def
{
     open_database;

     use_connection;

     close_database;
}

and not have to worry about the performance hit of opening and closing  
connections
if you have multiple small routines that do this AND you also dont  
have to worry
about setting up and keeping a database object at a module/global  
scoping level
to keep database connections from closing when the instance goes out  
of scope.

want to achieve this while still using storm.

the hard part here for us, is that all the code would need to be  
reworked to use
a single Store object global level where we really want to do is be able
to wrap the Store in our own classes that does thread saftey so we can  
use
same coding in both mod_python hosted code and multithreaded cron  
driven tasks.

The o the connection goes away and gets reopened is less of a concern  
than this
which is common in our current code:

for x in a
    some_method

some_method
{
   open_database;

    do_stuff;

    close_database;
}

because of Apache::DBI, that doesnt result in many copies of the  
database being opened
and closed over and over and we dont have to manage a global database  
object.
with storm as i read it, that  would be opening and closing the store  
and connection
over and over ( not good ) and the solution would be to move the store  
to a global
context ( not good for other code structuring reasons ).

i'm assuming nothing like this has been done with storm up to this  
point...