[MERGE][1.12][URGENT] Fix failing tests under rpython-2.5.2 on a 64bit host.

[Vincent Ladeuil]

>>>>> "jam" == John Arbash Meinel <john at arbash-meinel.com> writes:

    jam> Vincent Ladeuil wrote:
    >> I wanted to build the ppas for bzr and bzrtools for 1.12 and
    >> realized that Martin didn't upload the right .tar.gz on
    >> launchpad (the rc1 seems to have been uploaded instead)
    >> 
    >> Trying to create it myself (RM can have week-ends too) I ran into
    >> 'make check' failing...
    >> 
    >> It turned out that under strange circumstances (yet to be fully
    >> understood), the following test failed:
    >> 

    jam> ...

    >> ERROR: tree_implementations.test_path_content_summary.TestPathContentSummary.test_unicode_symlink_content_summary(PreviewTreePost)
    >> 'ascii' codec can't decode byte 0xce in position 172: ordinal not in range(128)
    >> 
    >> File "_dirstate_helpers_c.pyx", line 900, in _dirstate_helpers_c._update_entry
    >> File "/home/vila/src/bzr/trunk/bzrlib/dirstate.py", line 1591, in _read_link
    >> return os.readlink(abspath.encode(osutils._fs_enc))
    >> 
    >> Removing that encode() fixes the problem.

    jam> The fact that 'ascii' codec is failing, seems to indicate that your
    jam> fs_enc isn't set correctly.

fs_enc == 'UTF-8'

    jam> Does your system realize that its filesystem is UTF-8
    jam> encoded?

Yes.

    jam> Is this failing during the "LANG=C python ./bzr selftest
    jam> ..." portion?

No. The command above is enough to make it fail.

    jam> I don't quite understand why we need to encode the
    jam> abspath before calling 'readlink'. I would have thought
    jam> we would need:

    jam> return os.readlink(abspath).decode(osutils._fs_enc)

    jam> But I'm not looking at the code in its entirety.


    >> 
    >> ...
    >> 
    >> Only to fail a bit later on:
    >> 
    >> ./bzr selftest -v  -s 'bzrlib.tests.workingtree_implementations.test_parents.TestSetParents.test_unicode_symlink' 
    >> 
    >> <shudder>
    >> 
    >> At that point I decided to get feedback before trying to do a
    >> release with or without that fix.
    >> 
    >> I tracked down the modification related to the first problem to
    >> revno 3975 :
    >> '(Jelmer) Support symlinks with unicode characters in the symlink name'
    >> 
    >> I.e. the fix for bug 319323.
    >> 
    >> I'm not throwing stones here ! Obviously pqm accepted the merge
    >> so the tests have been passing there as on Jelmer host, no
    >> doubt. I put Jelmer in copy as he may shed some light of why he
    >> made the modification, if the modification is really needed (the
    >> patch history makes me wonder if that line is really needed, and
    >> the tests passing without it even more).
    >> 
    >> I also wonder if we get enough tests in that area, but may be
    >> that can wait until after the release ?
    >> 
    >> Vincent

    jam> I was a bit concerned that Jelmer's fix wasn't going to
    jam> get all the edge cases, but I never actually dug into
    jam> it. The summary of what Jelmer needed to do:

    jam> 1) When reading a symlink, call .decode() to convert the
    jam> 8-bit symlink target into a Unicode string.

    jam> 2) When writing a symlink, call .encode() to convert the
    jam> Unicode string back into 8-bit str.

    jam> The difficulty is just making sure we track down all of
    jam> the places that do that sort of thing. As a bit more
    jam> complexity, the dirstate file should save that
    jam> information in UTF-8, and not the local encoding.

    jam> I don't have a specific problem reverting the change,
    jam> but I would rather fix bugs and support non-ascii
    jam> symlinks, rather than continually push the feature off.

I'd prefer that too, the problem here is to decide what to do
about the release ?

Should we just forget about the problem considering that:

- my setup is weird (I wonder why, but let's admit that)

- I will soon understand why (I don't like having a weird setup
  to build releases...)

- I build the ppas for the rc and didn't encounter problems.

I'm fine with that and fixing the issue may wait.

But in that case, I'm back to square one about uploading a proper
bzr-1.12.tar.gz (about which I was already uncomfortable (acting
as a self-appointed RM-backup without prior experience is a sure
recipe for disasters) *before* encountering the bug :-)

Can you help there ?

    Vincent