BigString (reducing peak memory)

[Barry Warsaw]

On Nov 16, 2011, at 11:26 AM, Gordon Tyler wrote:

>This is just a wild idea, but would using Python generator expressions
>help with keeping only a small part of a large data file in memory? If you
>only ever need to deal with a non-random-access stream of bytes, then a
>custom generator could read data from the file in chunks as bytes are
>requested. It would have the advantage of not having to make code aware of
>the fact that the data is being chunked.

You might take a look at email.feedparser.FeedParser which does something like
this.  It's line-oriented because that's what email needs, but how it works
probably aligns fairly closely with what you'd want for the above.

Cheers,
-Barry