[storm] Pickling Storm objects?

[Kapil Thangavelu]

On Thu, Aug 27, 2009 at 7:48 AM, Stuart Bishop
<stuart.bishop at canonical.com>wrote:

> On Thu, Aug 27, 2009 at 4:56 PM, Gustavo Niemeyer<gustavo at niemeyer.net>
> wrote:
> > Hey Stuart,
> >
> >> Has anyone looked into making Storm objects pickleable? I want to
> >> stuff expensive query results into memcached.
> >>
> >> I'm using ZStorm so can just use the name to refer to the Store. I can
> >> put together a MaterializedResultSet class supporting a lot of the API
> >> from a materialized list of Storm objects. I think getting the Storm
> >> objects themselves pickled is going to be the tricky bit.
> >
> > Pickling itself shouldn't be hard.  How do you envison an unpickled
> > object should behave?
>
> Its nice to know you don't forsee major roadblocks. I think the major
> difficulty is becoming familiar enough with the Storm internals - I've
> never dealt with ObjectInfo and friends before.
>
> I'd like it to behave like the original object as much as possible.
> The goal is drop in replacement of code like:
>
>   results = store.find(... complex and costly conditions ...)
>
> with something like:
>
>   results = cached(store, max_age,  ... complex and costly conditions ...)
>
> So unpickled objects can be updated and code able to traverse from the
> unpickled objects to objects loaded from the Store. For the Storm
> objects, I expect they would be indistinguishable from one loaded from
> the Store (assemble object, swap in the Store, inject into the cache).
>
> I don't think I need the result set to support operations like union,
> find,  or aggregates beyond count() so the result set can just be a
> list with a count() method.
>


i have slightly different use cases for caching, i'd like to be able to
cache these independent of a store.
effectively separate from store, pickle, unpickle, and attach to store.
analogous to sqlalchemy session
detach/merge functionality.

afaics this involves clearing out the instance object info, and possibly
unhooking the event system from tracking variable changes.

cheers,

kapil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/archives/storm/attachments/20090827/7fcc7591/attachment-0002.htm