[storm] Limit cache size by memory usage?

[Sidnei da Silva]

On Mon, Jul 27, 2009 at 7:31 AM, Stuart
Bishop<stuart.bishop at canonical.com> wrote:
> Can anyone think of any alternative approaches? I'm thinking my next
> attempt when I'm bored will be a GenerationalCache that keeps bumping
> up a global cache size when the cache is bumped until the memory limit
> is reached, at which point it is fixed for the life of the program.

Generally, trying to manage memory yourself inside an application can
work against you. See the ArchitectNotes for Varnish for a general
overview of why and how:

   http://varnish.projects.linpro.no/wiki/ArchitectNotes

For something like Storm though, I suspect having an externally
managed cache which has a proven track record would only do you good.
For this reason, I would suggest creating/investigating a
'MemcachedCache' if such thing doesn't exist yet.

The downsides of having to manage an external daemon and keep it
running are fully outweighed by not having to worry too much about how
the caching is performed yourself. As a bonus you get proper
statistics of memory usage and hit hates out of memcached, which you
would have to implement yourself in order to measure whether your
implementation solves a specific problem or not.

-- Sidnei