[storm] Profiling project: instrumenting find()

[James Henstridge]

On Fri, Jul 1, 2011 at 1:20 AM, Jeroen Vermeulen <jtv at canonical.com> wrote:
> Hi folks,
>
> Here's my next stumbling block for the access-pattern profiling project.
>
> When you follow a reference and that causes Storm to load another object
> from the database, the profiler will need to record the details.  There are
> two ways in which Reference can load the other object:
>
> 1. If the reference refers to a primary key, it uses Store.get.
>
> 2. Otherwise it uses Store.find.
>
> When the Reference does a "get," I can make it pass an extra argument to
> tell the profiler "this is actually following a reference from another
> object, and here are the details."  That works pretty well.
>
> But what do I do when it uses "find"?  I have to tell the method that it's
> following a reference, or it will be counted double as a regular free-form
> query as well.
>
> I can't just add a keyword argument because "find" accepts arbitrary
> **kwargs.  This is a feature to let us do things like Store.find(Person,
> Person.name="Smith").
>
> I can think of two things to do about this:
>
> (a) Create a separate, less flexible version of find() for this.
>
> (b) Poke the profiling information in the Result that comes out.
>
> Are there any other ways?  My worry with (a) is that dynamism makes it hard
> to see exactly what I'm affecting.  And (b) involves a lot of bookkeeping
> that it would be nice to avoid.
>
> So far my preference is (a).

Have you considered using stack inspection here?  You can easily
inspect the local variables of the calling stack frames, so checking
for profiling data in specially named locals shouldn't be too
difficult.  You could even discover multiple calling contexts by
analysing the entire stack.

I remember that Zope did something similar in its trace back pretty
printer, displaying the value of __traceback_info__ locals as a way to
annotate the traceback.

James.