BigString (reducing peak memory)

[Marco Pantaleoni]

On Wed, Nov 16, 2011 at 5:49 PM, Aaron Bentley <aaron at aaronbentley.com>wrote:

> On 11-11-16 11:26 AM, Gordon Tyler wrote:
> > This is just a wild idea, but would using Python generator
> > expressions help with keeping only a small part of a large data
> > file in memory?
>
> Not really.  The point is that the data needs to be dealt with in
> smaller chunks, whether they're read() from a file directly or
> iterated through from a generator.  Dealing with the interface
> difference between files and iterables is the easy bit.  It's avoiding
> reading entire files at once that seems to be problematic.
>

Why is the whole file needed in memory at once?
If it is read into a string, it would be quite easy to create a "virtual"
string handling "paging" of the file.
This would provide the same functionality as mmap(), but without the 32 bit
or OS-specific limitations.
It would be different if it is needed to handle writes, where the written
file is expected to be handled by an unknown consumer (a pipe for example).

Marco

-- 
Marco Pantaleoni
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/bazaar/attachments/20111116/ee6fd184/attachment.html>