[Bug 1331521] [NEW] column has problems with different file encodings

[Jürgen Kahnert]

Public bug reported:

Ubuntu 10.04: bsdmainutils 8.0.1ubuntu1
Ubuntu 12.04: bsdmainutils 8.2.3ubuntu1
Ubuntu 14.04: bsdmainutils 9.0.5ubuntu1

column won't produce any output if the file encoding isn't recognized.
With Ubuntu 8.04 the output of unknown characters were crippled, but
there was still an output. This changed since fgetws is used to read the
characters (instead of fgets).

With Ubuntu 14.04 there is at least an error message instead of an empty
output:

    [ubu1404] # column -t -s ';' bar
    column: Invalid or incomplete multibyte or wide character

    [ubu1204] # column -t -s ';' bar
    [ubu1204] # echo $?
    0

    [ubu1204] # cat bar
    1;ä
    2;ö
    3;ü

    [ubu1404] # file foo bar
    foo: UTF-8 Unicode text, with CRLF line terminators
    bar: ISO-8859 text, with CRLF line terminators

Even with the correct locale setting it won't change anything:

    [ubu1404] # LC_CTYPE=de_DE.ISO-8859-1 column -t -s ';' bar
    column: Invalid or incomplete multibyte or wide character

With foo everything works as expected:

    [ubu1404] # cat foo
    1;ä
    2;ö
    3;ü

    [ubu1404] # column -t -s ';' foo
    1  ä
    2  ö
    3  ü

I guess this is related to bug #1065329 - but if I see that correctly,
it's not limited to Ubuntu, it's a general fgetws problem.

** Affects: bsdmainutils (Ubuntu)
     Importance: Undecided
         Status: New

** Attachment added: "both test files (foo & bar)"
   https://bugs.launchpad.net/bugs/1331521/+attachment/4134232/+files/foobar.tgz

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to bsdmainutils in Ubuntu.
https://bugs.launchpad.net/bugs/1331521

Title:
  column has problems with different file encodings

Status in “bsdmainutils” package in Ubuntu:
  New

Bug description:
  Ubuntu 10.04: bsdmainutils 8.0.1ubuntu1
  Ubuntu 12.04: bsdmainutils 8.2.3ubuntu1
  Ubuntu 14.04: bsdmainutils 9.0.5ubuntu1

  column won't produce any output if the file encoding isn't recognized.
  With Ubuntu 8.04 the output of unknown characters were crippled, but
  there was still an output. This changed since fgetws is used to read
  the characters (instead of fgets).

  With Ubuntu 14.04 there is at least an error message instead of an
  empty output:

      [ubu1404] # column -t -s ';' bar
      column: Invalid or incomplete multibyte or wide character

      [ubu1204] # column -t -s ';' bar
      [ubu1204] # echo $?
      0

      [ubu1204] # cat bar
      1;ä
      2;ö
      3;ü

      [ubu1404] # file foo bar
      foo: UTF-8 Unicode text, with CRLF line terminators
      bar: ISO-8859 text, with CRLF line terminators

  Even with the correct locale setting it won't change anything:

      [ubu1404] # LC_CTYPE=de_DE.ISO-8859-1 column -t -s ';' bar
      column: Invalid or incomplete multibyte or wide character

  With foo everything works as expected:

      [ubu1404] # cat foo
      1;ä
      2;ö
      3;ü

      [ubu1404] # column -t -s ';' foo
      1  ä
      2  ö
      3  ü

  I guess this is related to bug #1065329 - but if I see that correctly,
  it's not limited to Ubuntu, it's a general fgetws problem.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/bsdmainutils/+bug/1331521/+subscriptions