<div><div><div><br></div><div class="gmail_quote">On Thu, Aug 27, 2009 at 8:28 AM, Kapil Thangavelu <span dir="ltr"><<a href="mailto:kapil.foss@gmail.com">kapil.foss@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br><br><div class="gmail_quote"><div><div></div><div class="h5">On Thu, Aug 27, 2009 at 7:48 AM, Stuart Bishop <span dir="ltr"><<a href="mailto:stuart.bishop@canonical.com" target="_blank">stuart.bishop@canonical.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>On Thu, Aug 27, 2009 at 4:56 PM, Gustavo Niemeyer<<a href="mailto:gustavo@niemeyer.net" target="_blank">gustavo@niemeyer.net</a>> wrote:<br>
> Hey Stuart,<br>
><br>
>> Has anyone looked into making Storm objects pickleable? I want to<br>
>> stuff expensive query results into memcached.<br>
>><br>
>> I'm using ZStorm so can just use the name to refer to the Store. I can<br>
>> put together a MaterializedResultSet class supporting a lot of the API<br>
>> from a materialized list of Storm objects. I think getting the Storm<br>
>> objects themselves pickled is going to be the tricky bit.<br>
><br>
> Pickling itself shouldn't be hard. How do you envison an unpickled<br>
> object should behave?<br>
<br>
</div>Its nice to know you don't forsee major roadblocks. I think the major<br>
difficulty is becoming familiar enough with the Storm internals - I've<br>
never dealt with ObjectInfo and friends before.<br>
<br>
I'd like it to behave like the original object as much as possible.<br>
The goal is drop in replacement of code like:<br>
<br>
results = store.find(... complex and costly conditions ...)<br>
<br>
with something like:<br>
<br>
results = cached(store, max_age, ... complex and costly conditions ...)<br>
<br>
So unpickled objects can be updated and code able to traverse from the<br>
unpickled objects to objects loaded from the Store. For the Storm<br>
objects, I expect they would be indistinguishable from one loaded from<br>
the Store (assemble object, swap in the Store, inject into the cache).<br>
<br>
I don't think I need the result set to support operations like union,<br>
find, or aggregates beyond count() so the result set can just be a<br>
list with a count() method.<br>
<div></div></blockquote><div><br></div><div><br></div></div></div><div>i have slightly different use cases for caching, i'd like to be able to cache these independent of a store.</div><div>effectively separate from store, pickle, unpickle, and attach to store. analogous to sqlalchemy session</div>
<div>detach/merge functionality.</div><div><br></div><div>afaics this involves clearing out the instance object info, and possibly unhooking the event system from tracking variable changes. </div><div><br></div><div>cheers,</div>
<div><br></div><div>kapil</div><div></div></div></blockquote><div><br></div><div><br></div>i ended up making a base class to mimic the store merge/detach functionality along with the pickling support (inline below) that i needed. modifying the object when its detached is effectively an unknown op as there isn't a store to track it, although upon reattaching any subsequent modification and flushes will include the modifications while detached. hasn't been tested/used with references. <div>
<br></div><div>cheers,</div><div><br></div><div>kapil</div><div><br></div><div><div><br></div><div><div>from storm import properties, info</div><div><br></div><div>marker = object()</div><div><br></div><div>class StormPersistent( object ):</div>
<div><br></div><div> def __getstate__( self ):</div><div> d = {}</div><div> for p,v in info.get_obj_info(self).variables.items():</div><div> d[<a href="http://p.name">p.name</a>] = v.get()</div>
<div> return d</div><div> </div><div> def __setstate__( self, o ):</div><div> for p,v in info.get_obj_info( self ).variables.items():</div><div> value = o.get( <a href="http://p.name">p.name</a>, marker )</div>
<div> if value is marker: continue</div><div> v.set( value, from_db=True )</div><div><br></div><div> def detach( self ):</div><div> obj_info = info.get_obj_info( self )</div><div> store = obj_info.get('store') </div>
<div> assert not obj_info in store._dirty, "Can't Detach Dirty Object"</div><div> store._remove_from_alive( obj_info )</div><div> store._disable_change_notification( obj_info )</div><div>
store._disable_lazy_resolving( obj_info )</div><div><br></div><div> def attach( self, store ):</div><div> obj_info = info.get_obj_info( self )</div><div> obj_info['store'] = store</div><div>
store._enable_change_notification( obj_info )</div><div> store._add_to_alive( obj_info ) </div></div></div><div> store._enable_lazy_resolving( obj_info ) </div></div><br></div></div>