[bug] bundle creation problems

[holger krekel]

On Mon, Aug 14, 2006 at 10:51 -0500, John Arbash Meinel wrote:
> Martin Pool wrote:
> > On 11 Aug 2006, holger krekel <holger at merlinux.de> wrote:
> >> Hi folks, 
> >>
> >> while playing with bundle parsing, I noticed that 
> >>
> >>     bzr bundle -r 1..-1 --output=test.bundle 
> >>
> >> on current bzr.dev (1912 revisions) fails with: 
> >>
> >>     ... 
> >>     File "bzrlib/testament.py", line 144, in as_text_lines
> >>         return [line.encode('utf-8') for line in r]
> >>     UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128)
> >>
> >> where the 'line' in question is created by _revprops_to_lines 
> >> which encodes non-ascii characters in utf8 - and then the
> >> attempt in as_text_lines to encode again fails (depends
> >> on the locale default encoding).  The attached patch fixes 
> >> this particular problem. 
> 
> Well, at least to some respect, I already have submitted this bug:
> https://launchpad.net/products/bzr/+bug/54723
> 
> I came across it a while ago.
> 
> His specific fix is probably the wrong one to make. Because he basically
> skips doing any utf-8 encoding. And I think it depends more on whether
> the internal stanza is encoding or not.

Maybe i am missing something, but _revprops_to_lines is only
called once and if bzr is to use unicode internally and only
convert at barriers then that internal function does not
appear to be such a barrier, rather the last line in the
calling as_text_lines ...

    return [line.encode('utf-8') for line in r]

appears to be such a barrier.  
 
> We will also have a bug during decoding. So I think we need a more
> explicit set of tests that nested stanzas handle non-ascii characters.

hum, i'd rather leave this issue to you, John, as i 
don't fully grasp the involved implications, apparently. 
And admittedly, i am confused by the various encode/decode
calls in bzr (overall around 100 encode() and 50 decode() calls 
if i am not mistaken) - they appear too delicate for me to 
analyse them from my limited local perspective.  

best, 

    holger