[storm] why weakrefdict for cache?

[Michael Bayer]

On Sep 3, 2007, at 1:07 PM, Gustavo Niemeyer wrote:

> Ah, right.. it took me a while to get that stuff working as well.  The
> way we implemented mutable types in Storm is by offering variables the
> chance to register themselves in a "object-deleted" event, which gets
> called when the object is being deallocated (we use the callback in  
> the
> weakref).  E.g. PickleVariable does the following:
>
>     self.event.hook("object-deleted", self._detect_changes)

so if I had:

class Blub(object):
    def __init__(self, x, y):
        self.x = x
        self.y = y

and mapped it as a pickle type to Foober.blub:

f = Foober()
f.blub = Blub(7, 8)

where's the event hook placed ?  when i say f.blub.y = 12 for example.

>> dont have any of these requirements.   Though as it turns out, DBAPIs
>> like psycopg2 already buffer all the rows of a result set by  
>> default so
>> theres a lot more "load it all into memory" going on than people  
>> might
>> think anyway.
>
> Right.. but won't the maximum number of items kept in memory be
> cursor.arraysize, if you are using cursor.fetchmany?

one would think.  but no.  Psycopg2 buffers the whole result into  
memory the moment you say cursor.execute().  fetchone()/fetchmany()/ 
fetchall() all read from the buffer.

There *is* a way to override this, and that is to use a named  
cursor.   In that case psycopg2 maintains the cursor on the server  
side and each fetchXXX() call retrieves results over the wire.  I  
don't know why there isnt a simple flag "server_side=True", but on  
the psycopg2 list they seemed to hold the opinion that you shouldnt  
be selecting a result set larger than that which you can hold in  
memory, so they dont seem to view "server side cursors" as something  
anyone should really need (to them its just a side effect of using  
named cursors...hence not really documented or anything).

Anyway, somewhere in SA's 0.3 series we did an internal  
reorganization such that our postgres dialect *can* optionally use  
named cursors so that you get "server side" capability, however we  
had to jump through many hoops to make it work since Psycopg2s  
implementation makes it difficult (they will fail if used for  
anything other than a SELECT, and cursor.description is not available  
until the first row is fetched).