[Bug 846628] Re: gnu sort extremely slow in non C locale

[Bijan Soleymani]

Hi Dave,

Thanks for the response. I think you are definitely right about the
sorting order being correct (a feature and not a bug), but I think that
the much slower speed is a bug. It might be a sort of wishlist type bug,
but there's no theoretical reason why a faster method of sorting can't
be used (even though there might be a practical reason not to: no demand
from users or no time for developers). It may be that the algorithm used
for sorting 8-bit ASCII will not perform well for 16-bit random
collating order.

But of course the locale specific unicode can be converted to a
representation that sorts quickly. For example the first character in
collating order could become 0, the second becomes 1, and so on to
2^16-1 for example (for 16-bit characters). Then you can sort it using
plain ASCII sorting rules.

Bijan

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to coreutils in Ubuntu.
https://bugs.launchpad.net/bugs/846628

Title:
  gnu sort extremely slow in non C locale

Status in “coreutils” package in Ubuntu:
  Invalid

Bug description:
  I tried sorting an ascii file of about 300 Megs and 8 million lines
  with gnu sort and it was taking forever.

  After 10 minutes I stopped it. I tried another sort program and it
  finished in about 40 seconds.

  I then took the output of that second sort and I checked it in gnu
  sort, which reported that some lines were out of order.

  The following lines:
  ....bbbbbbbbbwbbwwbwwwwwww.ww...1
  ....bbbbbbbbbwbbwwbwwwwwwwww....0
  ....bbbbbbbbbwbwwbwbwwwww.ww..w.1

  But they are not as far as I can tell. Then I thought the problem was the locale. Indeed my locale was set to:
  LANG=en_CA.UTF-8

  setting it to:
  LANG=C

  both made gnu sort finish the sort in 40 seconds, and confirm the
  proper order.

  Since the file is %100 ASCII (it only has the 6 characters ".01bw\n" I
  think this is a bug, that the locale should make any difference.

  Best regards,
  Bijan

  ProblemType: Bug
  DistroRelease: Ubuntu 11.04
  Package: coreutils 8.5-1ubuntu6
  ProcVersionSignature: Ubuntu 2.6.38-11.48-generic 2.6.38.8
  Uname: Linux 2.6.38-11-generic i686
  Architecture: i386
  Date: Sat Sep 10 15:59:07 2011
  InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release i386 (20110427.1)
  ProcEnviron:
   LANGUAGE=en_CA:en
   LANG=en_CA.UTF-8
   SHELL=/bin/bash
  SourcePackage: coreutils
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/846628/+subscriptions