[storm] Pickling Storm objects?

[Kapil Thangavelu]

On Thu, Aug 27, 2009 at 8:28 AM, Kapil Thangavelu <kapil.foss at gmail.com>wrote:

>
>
> On Thu, Aug 27, 2009 at 7:48 AM, Stuart Bishop <
> stuart.bishop at canonical.com> wrote:
>
>> On Thu, Aug 27, 2009 at 4:56 PM, Gustavo Niemeyer<gustavo at niemeyer.net>
>> wrote:
>> > Hey Stuart,
>> >
>> >> Has anyone looked into making Storm objects pickleable? I want to
>> >> stuff expensive query results into memcached.
>> >>
>> >> I'm using ZStorm so can just use the name to refer to the Store. I can
>> >> put together a MaterializedResultSet class supporting a lot of the API
>> >> from a materialized list of Storm objects. I think getting the Storm
>> >> objects themselves pickled is going to be the tricky bit.
>> >
>> > Pickling itself shouldn't be hard.  How do you envison an unpickled
>> > object should behave?
>>
>> Its nice to know you don't forsee major roadblocks. I think the major
>> difficulty is becoming familiar enough with the Storm internals - I've
>> never dealt with ObjectInfo and friends before.
>>
>> I'd like it to behave like the original object as much as possible.
>> The goal is drop in replacement of code like:
>>
>>   results = store.find(... complex and costly conditions ...)
>>
>> with something like:
>>
>>   results = cached(store, max_age,  ... complex and costly conditions ...)
>>
>> So unpickled objects can be updated and code able to traverse from the
>> unpickled objects to objects loaded from the Store. For the Storm
>> objects, I expect they would be indistinguishable from one loaded from
>> the Store (assemble object, swap in the Store, inject into the cache).
>>
>> I don't think I need the result set to support operations like union,
>> find,  or aggregates beyond count() so the result set can just be a
>> list with a count() method.
>>
>
>
> i have slightly different use cases for caching, i'd like to be able to
> cache these independent of a store.
> effectively separate from store, pickle, unpickle, and attach to store.
> analogous to sqlalchemy session
> detach/merge functionality.
>
> afaics this involves clearing out the instance object info, and possibly
> unhooking the event system from tracking variable changes.
>
> cheers,
>
> kapil
>


i ended up making a base class to mimic the store merge/detach functionality
along with the pickling support (inline below) that i needed. modifying the
object when its detached is effectively an unknown op as there isn't a store
to track it, although upon reattaching any subsequent modification and
flushes will include the modifications while detached. hasn't been
tested/used with references.
cheers,

kapil


from storm import properties, info

marker = object()

class StormPersistent( object ):

    def __getstate__( self ):
        d = {}
        for p,v in info.get_obj_info(self).variables.items():
            d[p.name] = v.get()
        return d

    def __setstate__( self, o ):
        for p,v in info.get_obj_info( self ).variables.items():
            value = o.get( p.name, marker )
            if value is marker: continue
            v.set( value, from_db=True )

    def detach( self ):
        obj_info = info.get_obj_info( self )
        store = obj_info.get('store')
        assert not obj_info in store._dirty, "Can't Detach Dirty Object"
        store._remove_from_alive( obj_info )
        store._disable_change_notification( obj_info )
        store._disable_lazy_resolving( obj_info )

    def attach( self, store ):
        obj_info = info.get_obj_info( self )
        obj_info['store'] = store
        store._enable_change_notification( obj_info )
        store._add_to_alive( obj_info )
        store._enable_lazy_resolving( obj_info )
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/archives/storm/attachments/20090828/f7a1ee4f/attachment.htm