BigString (reducing peak memory)

[Martin Pool]

On 17 November 2011 03:26, Gordon Tyler <gordon at doxxx.net> wrote:
> This is just a wild idea, but would using Python generator expressions
> help with keeping only a small part of a large data file in memory? If you
> only ever need to deal with a non-random-access stream of bytes, then a
> custom generator could read data from the file in chunks as bytes are
> requested.

Yes, that's part of what we're doing.

> It would have the advantage of not having to make code aware of
> the fact that the data is being chunked.

That's not quite possible though: Python code that expects to operate
on a string will not just transparently work with a generator of
strings.  It can be changed to do so.  Code that eagerly joins up
iterators into single strings needs to be taught not to.

-- 
Martin