[MERGE] get_record_stream().get_bytes_as('chunked')

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> Robert Collins has voted tweak.
> Status is now: Conditionally approved
> Comment:
>> For example, imagine updating the working tree using "iter_files_chunks"
>> rather than "iter_files_bytes". We could stream the bytes for each file
>> out, rather than having to hold the entire content in memory.
> 
> So for this, the interface is already generic enough - the bytes
> iterator could output the chunks.

Hmm... The current implementation returns a simple string of bytes.
Which is technically an iterable of bytes, but ".writelines(str)" is a
lot slower that ".writelines((str,))" or ".write(str)".

Certainly str = ''.join(str) is actually quite slow, because it rebuilds
the string from scratch. (Which is funny, because ''.join([str]) returns
str without copying.)


> 
>> This is actually fast enough, that I would consider re-implementing
>> osutils.split_lines() in terms of chunks_to_lines. (Timing shows it to
>> be about 2x faster.)
> 
> please!

K. I'll do that.

> 
> +    def test_smoketest(self):
> +        self.assertEqual(['foo\n', 'bar\n', 'baz\n'],
> +                         osutils.chunks_to_lines(['foo\nbar', '\nbaz\n']))
> +        self.assertEqual(['foo\n', 'bar\n', 'baz\n'],
> +                         osutils.chunks_to_lines(['foo\n', 'bar\n',
> 'baz\n']))
> 


I have those as part of the test__chunk_lines tests.
This is only meant to test that chunks_to_lines is being imported into
osutils correctly.

> I think you need tests for:
>  - no trailing \n
Have it

>  - leading \n

I added this one

>  - double \n
not sure if I had this one exactly, but I'll make sure to add it

>  - no \n at all (whole thing)
already done

>  - no \n in a single chunk at all
also done

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAklBalgACgkQJdeBCYSNAAMx5QCdEzGhLv4n4DhMsf7Zo9Vd5xyh
8+IAniW0ktRj4KA3pwCF8YTbnHdxNp0m
=3Cug
-----END PGP SIGNATURE-----