[storm] Limit cache size by memory usage?

[James Henstridge]

On Thu, Jul 30, 2009 at 12:05 PM, Stuart
Bishop<stuart.bishop at canonical.com> wrote:
> On Thu, Jul 30, 2009 at 1:29 AM, Gustavo Niemeyer<gustavo at niemeyer.net> wrote:
>>> Generally, trying to manage memory yourself inside an application can
>>> work against you. See the ArchitectNotes for Varnish for a general
>>> overview of why and how:
>>
>> Yeah, I'm kind of concerned about it too, since there are so many
>> details in the way (Python's internal memory buffers, C library
>> allocation behavior, the application's use of objects, etc).
>> Promising an auto-tweaking cache which behaves poorly would be worse
>> than advising developers to tweak their cache size to fit their needs.
>
> I agree that auto-tweaking caches that behave poorly are not a good
> idea :-) I'm fishing for ideas on how to get one that doesn't behave
> poorly, or at least better than my first experiment. It may be
> impossible.

Perhaps it would be possible to implement some kind of profiling cache
implementation would be worthwhile then: something that would record
cache hits and misses, and some utilities to analyse that data.


> I do feel that if I had a half decent implementation and could tell a
> program 'Use up to 2GB of RAM', that would be useful for almost all of
> the code we deploy. Our current mechanism is to make an educated guess
> as to the cache size and forget about it unless performance or RAM
> consumption is an issue (multiple bits of code sharing a server means
> swapping is bad). Even if went to the trouble of calculating the ideal
> cache size, soon it would no longer be ideal as our data is constantly
> changing - the optimal cache size for importing a pofile with small
> translations is different to the optimal cache size for importing a
> pofile with lengthy translations because the objects are radically
> different in size.

Note that a larger cache isn't always going to be better: some
operations scan over the list of alive objects (e.g ResultSet.set), so
keeping more objects alive could slow those operations down.  If you
end up caching objects that never get used again, then you'd probably
be better off with a smaller cache.

And as I said in the previous email, a moderately complex Storm
application will likely have many caches, so having each cache make
decisions independently will likely give poor results (i.e. one cache
ends up really large and the others really small).


James.