[storm] Performance Problems using List and Pickle properties

[Jürgen Kartnaller]

So here is the code to reproduce the problem :

from storm.base import Storm
from storm import properties
from storm.database import create_database
from storm.store import Store
from storm.variables import MutableValueVariable

class WithList(Storm):
    __storm_table__ = 'withlist'

    def __init__(self, key):
        self.key = key

    key = properties.Int(primary = True)

    v = properties.Int()
    l = properties.List(
                    type=properties.Unicode(),
                    default_factory=list
                   )

detectCount = 0

def performanceTest():
    global detectCount

    # patch to see how often _detect_changes is called
    original_detect_changes = MutableValueVariable._detect_changes
    def patch_detect_changes(self, obj_info):
        global detectCount
        detectCount += 1
        original_detect_changes(self, obj_info)
    MutableValueVariable._detect_changes = patch_detect_changes

    database = create_database('use a database supporting lists')
    store = Store(database)
    try:
        store.execute("DROP TABLE withlist;")
    except :
        store.rollback()
    store.execute("""
          CREATE TABLE withlist(
           key int primary key,
           v int,
           l text[]
          );
          """)

    n = store.add(WithList(1))
    del n
    store.commit()

    for i in range(10):
        n = store.get(WithList, 1)
        # access the list
        n.l
        # remove strong reference so the object is removed from the _alive
        # WeakValueDictionary in the store
        del n
        store.commit()
        print 'detect calls =', detectCount
        detectCount = 0

The output will be :

detect calls = 3
detect calls = 4
detect calls = 6
detect calls = 8
detect calls = 10
detect calls = 12
detect calls = 14
detect calls = 16
detect calls = 18
detect calls = 20

Jürgen


On Sat, Oct 3, 2009 at 1:59 PM, Jürgen Kartnaller <
kartnaller at lovelysystems.com> wrote:

> The problem exists in 0.14 and 0.15
>
> I tried to reproduce the problem with a simple example without success :(
>
> Jürgen
>
>
> On Fri, Oct 2, 2009 at 10:16 PM, Jamu Kakar <jkakar at kakar.ca> wrote:
>
>> Hi Jürgen,
>>
>> On Fri, Oct 2, 2009 at 11:56 AM, Jürgen Kartnaller
>> <kartnaller at lovelysystems.com> wrote:
>> > After a long time having problems with performance going down in our
>> > Zope/Storm applications I could now figure out where it is coming from.
>> In
>> > my latest app using tornado and Storm I have the same problem.
>> >
>> > When using List or Pickle properties (both are derived from
>> > MutableValueVariable) I found that any time a property is read a flush
>> event
>> > hook is added but never removed. The longer the app runs the more hooks
>> are
>> > registered which result in low perfomance in store.flush.
>> > The flush event hook is unhooked on the 'stop-tracking-changes' event
>> which
>> > is only emited if an object is removed from the database.
>>
>> Are you using the latest version of Storm?  I remember a problem
>> like this being fixed some time ago.  I think this is the fix for
>> the problem I'm thinking of:
>>
>> revno: 275 [merge]
>> committer: Thomas Hervé <thomas at canonical.com>
>> branch nick: trunk
>> timestamp: Mon 2008-11-03 20:38:06 +0100
>> message:
>>  Merge variable-referenceset-leak [r=niemeyer,jkakar]
>>
>>  Change the way MutableValueVariable hooks up to the store event system,
>> by
>>  overriding get and set. This works around a leak discovered when using
>>  PickleVariable in a ReferenceSet.
>>
>> Thanks,
>> J.
>>
>
>
>
> --
> Jürgen Kartnaller, senior developer
>
> Lovely Systems AG
> Telefon +43 5572 908060, Fax +43 5572 908060-77
> Schmelzhütterstraße 26a, 6850 Dornbirn, Austria
>
> Sitz: Dornbirn, FB: Landesgericht Feldkirch, FN: 208859x, UID: ATU51736705
> Aufsichtsratsvorsitzender: Christian Lutz
> Vorstand: Manfred Schwendinger
>



-- 
Jürgen Kartnaller, senior developer

Lovely Systems AG
Telefon +43 5572 908060, Fax +43 5572 908060-77
Schmelzhütterstraße 26a, 6850 Dornbirn, Austria

Sitz: Dornbirn, FB: Landesgericht Feldkirch, FN: 208859x, UID: ATU51736705
Aufsichtsratsvorsitzender: Christian Lutz
Vorstand: Manfred Schwendinger
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/archives/storm/attachments/20091003/8ae63c52/attachment-0001.htm