[MERGE] OSX test suite passing (well as of 2008/09/08 :)

Vincent Ladeuil v.ladeuil+lp at free.fr
Mon Sep 8 17:12:13 BST 2008

@aaron: If you can at least review test_case_insensitive_clash
changed into test_rollback_on_directory_clash, that will be

>>>>> "john" == John Arbash Meinel <john at arbash-meinel.com> writes:

    john> Vincent Ladeuil wrote:
    >> Below are the tests that are erroring or failing on OSX as of
    >> today. I also include the expected failures specific to OSX.
    >> ,----
    >> | ERROR: blackbox.test_export.TestExport.test_tar_export
    >> |     'NoneType' object has no attribute 'startswith'
    >> `----
    >> Not totally diagnosed yet. But the traceback ends up in
    >> python2.5/tarfile.py and there are a *lot* of differences between
    >> the stock OSX 10.4 python-2.5.1 and python-2.5.2 on hardy.

    john> certainly sounds like a tarfile discrepency. But
    john> without the traceback...

As Robert pointed out, it's a bug in python 2.5.1, bad vila,
upgrade ! Ok, done, problem vanished :)

    >> ,----
    >> | ERROR: blackbox.test_init.TestInit.test_init_unicode
    >> |     'ascii' codec can't encode character u'\xb5' in position 18: ordinal not in range(128)
    >> `----
    >> Running with LC_CTYPE=en_US.UTF-8 and LC_ALL=en_US.UTF-8 makes it
    >> pass, what should we do here ?
    >> That seems to be linked to osutils.get_user_encoding() try to
    >> work around some bug on darwin:
    >> if sys.platform == 'darwin':
    >> # work around egregious python 2.4 bug
    >> sys.platform = 'posix'
    >> try:
    >> import locale
    >> finally:
    >> sys.platform = 'darwin'
    >> else:
    >> import locale
    >> I'd love some info about that egregious bug :-/

    john> Specifically, without doing the workaround,
    john> locale.getpreferredencoding() always returns
    john> 'MacRoman'. Which isn't true anymore. Without doing the
    john> workaround, setting LANG has no effect on
    john> getpreferredencoding().

    john> The issue is that the filesystem is UTF-8. I'm guessing
    john> we are having a problem printing something out to the
    john> user, etc. Which would be a terminal encoding issue. I
    john> know a bug there is that terminal encoding *is* utf-8
    john> (at least for the stock kernel) but without setting
    john> LANG=....UTF-8 it thinks it is actually MacRoman or
    john> somesuch...

Yeah, the infamous 'some such' :D

    john> So MacPython doesn't really detect its encodings
    john> correctly, so we have to sort of hack around
    john> them. (Such as asking users to set LANG.)

Yeah, I love programs who knows what to do and tell me to do it,
I never want to tell them JFDI ! Never :D

    john> I suppose we could just force the user encoding,
    john> terminal encoding, etc on Mac.

So I did that, force os.environ['LANG'] = 'en_US.UTF-8' if LANG
is not set.

It may not be perfect, but it's better than failing like we do

    >> ,----
    >> | ERROR: tree_implementations.test_test_trees.TestTreeShapes.test_tree_with_merged_utf8(WorkingTreeFormat4)
    >> |     Working tree "/private/tmp/testbzr-vJk5hc.tmp/bzrlib.tests.tree_implementations.test_test_trees.TestTreeShapes.test_tree_with_merged_utf8(WorkingTreeFormat4)/work/" has uncommitted changes.
    john> ...

    >> Yeah, right, go find the bug in a ~20.000 lines patch 8-(

    john> I'll point you in the right location. 

Pfewwww :)

    john> It is using a NFC utf-8 string.  Change the test to use
    john> a string that can't be combining.

    john> For example instead of using u'\xe5' == å == u'a\u030a' use something
    john> like "\N{omega}".

I went with the euro sign:
- Unicode: U+20AC
- utf-8: 0xE2 0x82 0xAC

\N{omega} wasn't recognized, I used \u2375 but it wasn't present
in the font I used under Linux and hard to read under OSX.

I tried the copyright sign but there are several possible

and I thought we had enough confusion.

The euro sign is quite recent, so there is a single encoding, the
money was introduced 6 years ago, and most fonts on OSX and on
Linux include it.

    >> And since the test does:
    >> self.build_tree([u'tree2/b\xe5r/z\xf7z'])
    >> tree2.add([u'b\xe5r/z\xf7z'], [u'z\xf7z-id'.encode('utf-8')])
    >> tree2.commit(u'to m\xe9rge', rev_id=u'r\xe9v-2'.encode('utf8'))
    >> It should be a normalization problem.
    >> I've yet to understand if the problem is in how the test is
    >> written of if dirstate lacks some normalization.
    >> Comments welcome while the investigations continue...

    john> I would change the test to not use a character than
    john> plays poorly with normalization. We have that problem
    john> on Mac, but we don't need to test it at every
    john> turn. When we *fix* that, we will write tests
    john> explicitly for it.

All right. I take that as meaning 'the test used an invalid file
name which higher layers will catch in production code'. And
given the other failing tests in this thread, that sounds
correct. Right ?

    >> ,----
    >> | FAIL: workingtree_implementations.test_rename_one.TestRenameOne.test_rename_to_denormalised_fails(WorkingTreeFormat4)
    >> |     (<class 'bzrlib.errors.InvalidNormalization'>, <type 'exceptions.UnicodeEncodeError'>) not raised
    >> | 
    >> | FAIL: workingtree_implementations.test_rename_one.TestRenameOne.test_rename_to_denormalised_fails(WorkingTreeFormat3)
    >> |     (<class 'bzrlib.errors.InvalidNormalization'>, <type 'exceptions.UnicodeEncodeError'>) not raised
    >> | 
    >> | FAIL: workingtree_implementations.test_rename_one.TestRenameOne.test_rename_to_denormalised_fails(WorkingTreeFormat2)
    >> |     (<class 'bzrlib.errors.InvalidNormalization'>, <type 'exceptions.UnicodeEncodeError'>) not raised
    >> | 
    >> `----
    >> These ones fail since they have been introduced (bzr at 2843),
    >> Robert, can you explain the intent of this test ? Or better its
    >> consequences.
    >> I can add a:
    >> if osutils.normalizes_filenames():
    >> from bzrlib import tests
    >> raise tests.TestNotApplicable('OSX normalizes filenames')
    >> But is this correct ? If it's to guard against invalid filenames
    >> on the file system, AFAIK, OSX provides that.

    john> Specifically, we want to refuse to add or rename a
    john> non-normalized filename. Eventually we want to relax
    john> that restriction.

Not a problem on OSX where you can't create such a file.

    john> But until we can get there, we need to filter at add
    john> and rename time.

I read that as 'bzr filter at add and rename' *now*. So even when
checking out on OSX (or on an HFS mounted fs) bzr never
*produces* such files.

It comes that these tests can be safely skipped on OSX.

Or do we want to add a specific one that indeed, you can't create
such a file ?

    aaron> Vincent Ladeuil wrote:
    >>>>>>> "aaron" == Aaron Bentley <aaron at aaronbentley.com> writes:

    >> Roughly speaking, in that test, we rely on _FileMover via
    >> TreeTransform to detect a name clash when the file system is case
    >> insensitive.

    aaron> Actually, that test is about making sure that if
    aaron> renaming a file fails during a TreeTransform, we roll
    aaron> back properly.  It doesn't really have anything to do
    aaron> with case sensitivity.  I should not have used
    aaron> CaseInsensitiveFilesystemFeature for this.


    aaron> Well, what I really mean is changing the test so that
    aaron> it tries to rename a directory on top of another
    aaron> directory.  I believe this causes failures on every
    aaron> platform.

So I changed test_case_insensitive_clash into

Does that fit ?

    >> Below are the tests that are erroring or failing on OSX as of
    >> today. I also include the expected failures specific to OSX.

    martin> Thanks for trying them.  I actually thought there would be more...

Guillermo began fixing some.


    >> ,----
    >> | XFAIL: test_bundle.V08BundleTester.test_unicode_bundle
    >> | Mac OSX doesn't preserve unicode combining characters.
    >> |
    >> | XFAIL: test_bundle.V09BundleKnit1Tester.test_unicode_bundle
    >> | Mac OSX doesn't preserve unicode combining characters.
    >> |
    >> | XFAIL: test_bundle.V09BundleKnit2Tester.test_unicode_bundle
    >> | Mac OSX doesn't preserve unicode combining characters.
    >> |
    >> | XFAIL: test_bundle.V4BundleTester.test_unicode_bundle
    >> | Mac OSX doesn't preserve unicode combining characters.
    >> `----
    >> This ones will be there for a long time.

    martin> It seems like these could be fixed by changing the
    martin> test to use unicode characters that either are
    martin> already normalized (as letter+combining character) or
    martin> just have no other representation (like the copyright
    martin> symbol say).  The mac unicode behaviour is not
    martin> essential to this test as far as I can see.


    >> ,----
    >> | XFAIL: test_bundle.V4WeaveBundleTester.test_unicode_bundle
    >> | Bug #141438: fails for WorkingTree3 on OSX
    >> `----
    >> Waiting for WorkingTree3 to be deprecated :)

    martin> It seems like we're using xfail for a couple of
    martin> different things here.  One is a bug for which we
    martin> have a test but not yet a fix, and we want to at
    martin> least commit that to mainline, and we intend to fix
    martin> it in the future.  I think it's useful they get
    martin> flagged as xfail does.

    martin> But if it's something that's a limitation of a
    martin> previous format and we're never going to actually fix
    martin> it in that situation, but it is fixed in a later
    martin> format, why report it like this?  NotApplicable or
    martin> something might be better.

Well in that case the test was already raising:

            self.knownFailure("Mac OSX doesn't preserve unicode"
                              " combining characters.")

but for workingTree3 he didn't even reach that, so I filed a bug
and mention it because there was another problem I didn't
diagnose at the time and I wanted to keep them separate.

But both are gone now.

So, sorry for that loong mail, here is the patch,


-------------- next part --------------
A non-text attachment was scrubbed...
Name: osx_tests_fix-3654.patch
Type: text/x-patch
Size: 25446 bytes
Desc: BZR merge
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20080908/952e4fe8/attachment-0001.bin 

More information about the bazaar mailing list