[storm] Pooling ?

[Andrew Holman]

1) The guys from storm I think are going to answer this question as  
they were the ones who advised me on it. Basically it just makes  
things easier for me. If you look at the code I look for any errors  
and do an auto rollback if needed.  Otherwise it attempts to do a  
commit to commit any changes. With this tool you can also turn this  
on / off with the tools.storm.on = False in your config.

2) Disconnects from the database are now fixed in the trunk version of  
storm. Storm will now try and auto reconnect to the db on the threads  
next request.

2.1)  Fumanchu_ when he means request, he thinks it is best to grab a  
connection from a pool during the request and then return it to the  
pool after it's finished as opposed to 1 connection per thread. I am  
sure a more experienced Python programmer could write a pooling  
mechanism for CherryPy / Storm, at this point I was not able to  
although Neimeiyer gave me some code to look into on how it can be done.

On a side note I am talking to Fumanchu_ to have him review my tool,  
just to see if there are any more features I can add  to make it rock  
solid with CherryPy.

-- Andy

On Nov 20, 2007, at 11:35 AM, Håvard Gulldahl wrote:

>
> On Sun, 2007-11-18 at 11:19 -0500, Andrew Holman wrote:
>> That is actually my code that I was working with the guys in #storm
>
> Well, that was quite a coincidence. :)
>
>> and #cherrypy on.  I wound up getting this working. I also added a
>> wiki entry to the tools section on the cherrypy.org website.  The  
>> code
>
> Nice. (The url is http://tools.cherrypy.org/wiki/Storm for anyone
> curious).
>
> I have some questions that I hope you don't mind me asking.
>
> 1)
> In the example you're commiting after a store.find() -- is that just  
> for
> the sake of argument?
>
> More generally, why are you store.commit()-ing on every request? Since
> the store is local to the thread, wouldn't getting stuff from the  
> storm
> cache (that commit() invalidates) be more convenient?
>
> 2)
> How do you handle disconnects?
>
> 2.1)
> Previously, one of the cherrypy devs (fumanchu) stated that[1] in
> certain circumstances, there is no clean way to detect and disconnect
> when a thread ends, and made the case for per-request database
> connections. Has this changed?
>
> [1]: http://osdir.com/ml/python.cherrypy/2006-10/msg00015.html
>
>
> Other than that - to these inexperienced eyes, it looks like a good,
> clean way to combine cherrypy and storm.
>
>
>> has been expanded and so far is working great. A big thanks go out to
>> Niemeyer, Radix, fumanchu_ (cherrypy), dowski and the gang in  
>> #cherrypy.
>> This code is for the beta version of Cherrypy 3.1 so things would be
>> different in 3.0.
>>
>
> I'd like to repeat those thanks to everyone involved. This is great
> stuff!
>
> Håvard
>
>
>> - Andy
>>
>>
>> On Nov 17, 2007, at 9:44 PM, Håvard Gulldahl wrote:
>>
>>> Hello, Andy, list,
>>>
>>> On 11/7/07, Andrew Holman <andy at belator.com> wrote:
>>>> I am working with CherryPy and am trying to figure out the best way
>>>> to incorporate Storm into it. Does Storm have any connection  
>>>> pooling?
>>>
>>> I'm currently in the same situation, and I freely admit that I find
>>> thread programming conceptually nauseating. However, I just came  
>>> over
>>> this little pastie:
>>>
>>> http://pastie.caboo.se/pastes/117963 (author unknown)
>>>
>>> and it seems to do the right thing: attach a new store to each newly
>>> created thread.
>>>
>>> def connect_db(thread_index):
>>> 	database =  
>>> create_database(cherrypy.config.get('storm.default.url'))
>>> 	cherrypy.thread_data.store = Store(database)
>>> 	
>>> cherrypy.engine.subscribe('start_thread', connect_db)
>>>
>>>
>>>> If not then what is the best way to handle ensuring the connections
>>>> are still alive with each thread?  I would assume we would have to
>>>> write something that would do a check/auto-connect if the threads
>>>> connection was no longer alive. How should that be handled?
>>>
>>>
>>> Now this doesn't handle disconnects, but it may serve as a useful
>>> starting point. What do you think?
>>>
>>> Håvard
>>>
>>>
>>>
>>> -- 
>>> Håvard Gulldahl <havard at gulldahl.no>
>>
>>
>>
>> =================================
>> Andrew Holman
>> E-Mail: andy at belator.com
>> Skype: belator-andy
>>
>>
>



=================================
Andrew Holman
E-Mail: andy at belator.com
Skype: belator-andy