[storm] why weakrefdict for cache?

[Christopher Armstrong]

On 8/31/07, Bernd Dorn <bernd.dorn at lovelysystems.com> wrote:
>
> we use storm with zope (version .10) and discovered that the
> store._cache attribute never caches, because the weakrefs are gone
> immediatly


Your conclusion is incorrect. The cache does work as intended; perhaps it is
not what you want, though. The cache is there to prevent loading objects
from the database when they are already in memory, and to avoid creating
duplicate objects when the same row is loaded twice (if a row is retrieved a
second time within the same transaction, the original object is updated
instead of creating a new one).


we changed the implementation to use a normal dict, and do a
> self._cache.clear() if an invalidate(None) is issued. which is done
> on commit and rollback. this makes sure there are no objects
> referenced after a transaction
>
> what about integrating this into storm?



Probably not. It's not clear why this is necessary. The cache works fine.
What problems did you have with its behavior?

To be extra-clear: this patch will cause many Storm applications severe
performance problems. There are many cases when it's necessary to process a
large number of objects in a single transaction, and keeping *all* of those
objects in memory for the life of the transaction is an unnecessary burden.
For example:

for object in store.find(Foo):
    print object.name

while it may take a long time on a table where Foo has 100,000 items, at
least won't keep all of those foo instances in memory. With your patch, it
will.

If your application calls for hard caching in a certain area, then you
should probably implement the caching just for that portion of your
application.

-- 
Christopher Armstrong
International Man of Twistery
http://radix.twistedmatrix.com/
http://twistedmatrix.com/
http://canonical.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/archives/storm/attachments/20070831/183f27d0/attachment.htm