[MERGE][1.12][URGENT] Fix failing tests under rpython-2.5.2 on a 64bit host.

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vincent Ladeuil wrote:
> I wanted to build the ppas for bzr and bzrtools for 1.12 and
> realized that Martin didn't upload the right .tar.gz on
> launchpad (the rc1 seems to have been uploaded instead)
> 
> Trying to create it myself (RM can have week-ends too) I ran into
> 'make check' failing...
> 
> It turned out that under strange circumstances (yet to be fully
> understood), the following test failed:
> 

...

> ERROR: tree_implementations.test_path_content_summary.TestPathContentSummary.test_unicode_symlink_content_summary(PreviewTreePost)
>     'ascii' codec can't decode byte 0xce in position 172: ordinal not in range(128)
> 
>   File "_dirstate_helpers_c.pyx", line 900, in _dirstate_helpers_c._update_entry
>   File "/home/vila/src/bzr/trunk/bzrlib/dirstate.py", line 1591, in _read_link
>     return os.readlink(abspath.encode(osutils._fs_enc))
> 
> Removing that encode() fixes the problem.

The fact that 'ascii' codec is failing, seems to indicate that your
fs_enc isn't set correctly.

Does your system realize that its filesystem is UTF-8 encoded? Is this
failing during the "LANG=C python ./bzr selftest ..." portion?

I don't quite understand why we need to encode the abspath before
calling 'readlink'. I would have thought we would need:

return os.readlink(abspath).decode(osutils._fs_enc)

But I'm not looking at the code in its entirety.


> 
> ...
> 
> Only to fail a bit later on:
> 
> ./bzr selftest -v  -s 'bzrlib.tests.workingtree_implementations.test_parents.TestSetParents.test_unicode_symlink' 
> 
> <shudder>
> 
> At that point I decided to get feedback before trying to do a
> release with or without that fix.
> 
> I tracked down the modification related to the first problem to
> revno 3975 :
> '(Jelmer) Support symlinks with unicode characters in the symlink name'
> 
> I.e. the fix for bug 319323.
> 
> I'm not throwing stones here ! Obviously pqm accepted the merge
> so the tests have been passing there as on Jelmer host, no
> doubt. I put Jelmer in copy as he may shed some light of why he
> made the modification, if the modification is really needed (the
> patch history makes me wonder if that line is really needed, and
> the tests passing without it even more).
> 
> I also wonder if we get enough tests in that area, but may be
> that can wait until after the release ?
> 
>     Vincent

I was a bit concerned that Jelmer's fix wasn't going to get all the edge
cases, but I never actually dug into it. The summary of what Jelmer
needed to do:

1) When reading a symlink, call .decode() to convert the 8-bit symlink
target into a Unicode string.
2) When writing a symlink, call .encode() to convert the Unicode string
back into 8-bit str.

The difficulty is just making sure we track down all of the places that
do that sort of thing. As a bit more complexity, the dirstate file
should save that information in UTF-8, and not the local encoding.

I don't have a specific problem reverting the change, but I would rather
fix bugs and support non-ascii symlinks, rather than continually push
the feature off.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmVgv8ACgkQJdeBCYSNAAPzJQCgixcQUkF/WaQNpPqHUu1/65G1
FDsAn2rU9m29XCnVC3iNymI+FYKJ+Qmu
=QzA1
-----END PGP SIGNATURE-----