[Bug 839609] Re: [11.10 beta1] UnicodeDecodeError crash on localized input in multiple encodings/languages

[Dennis Chua]

Reviewed this issue with Oneiric Beta2, updating command-not-found to
0.2.44ubuntu1. With the bogus Simplified Chinese test, command-not-found
does not throw an exception.

However, the output text does not appear to coincide with the language
encoding of the shell environment. Compare the following, Natty vs.
Oneiric Beta2:

=== Natty. command-not-found 0.2.41ubuntu2 ===

u at u-VirtualBox:~$ 我
我：找不到命令

u at u-VirtualBox:~$ mgcc
未找到 'mgcc' 命令，您要输入的是否是：
 命令 'mlcc' 来自于包 'mlterm-tools' (universe)
 命令 'cgcc' 来自于包 'sparse' (multiverse)
 命令 'gcc' 来自于包 'gcc' (main)
 命令 'gcc' 来自于包 'pentium-builder' (universe)
mgcc：找不到命令
u at u-VirtualBox:~$

u at u-VirtualBox:~$ locale
LANG=zh_CN.UTF-8
LANGUAGE=zh_CN:en_US:en
LC_CTYPE="zh_CN.UTF-8"
LC_NUMERIC="zh_CN.UTF-8"
LC_TIME="zh_CN.UTF-8"
LC_COLLATE="zh_CN.UTF-8"
LC_MONETARY="zh_CN.UTF-8"
LC_MESSAGES="zh_CN.UTF-8"
LC_PAPER="zh_CN.UTF-8"
LC_NAME="zh_CN.UTF-8"
LC_ADDRESS="zh_CN.UTF-8"
LC_TELEPHONE="zh_CN.UTF-8"
LC_MEASUREMENT="zh_CN.UTF-8"
LC_IDENTIFICATION="zh_CN.UTF-8"
LC_ALL=

=== Oneiric Beta2. command-not-found 0.2.44ubuntu1 ===

u at u-VirtualBox:~$ 我
我: command not found

u at u-VirtualBox:~$ mgcc
No command 'mgcc' found, did you mean:
 Command 'mlcc' from package 'mlterm-tools' (universe)
 Command 'cgcc' from package 'sparse' (multiverse)
 Command 'gcc' from package 'gcc' (main)
 Command 'gcc' from package 'pentium-builder' (universe)
mgcc: command not found

u at u-VirtualBox:~$ locale
LANG=zh_CN.UTF-8
LANGUAGE=zh_CN:en_US:en
LC_CTYPE=zh_CN.UTF-8
LC_NUMERIC="zh_CN.UTF-8"
LC_TIME="zh_CN.UTF-8"
LC_COLLATE=zh_CN.UTF-8
LC_MONETARY="zh_CN.UTF-8"
LC_MESSAGES=zh_CN.UTF-8
LC_PAPER="zh_CN.UTF-8"
LC_NAME="zh_CN.UTF-8"
LC_ADDRESS="zh_CN.UTF-8"
LC_TELEPHONE="zh_CN.UTF-8"
LC_MEASUREMENT="zh_CN.UTF-8"
LC_IDENTIFICATION="zh_CN.UTF-8"
LC_ALL=

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to command-not-found in Ubuntu.
https://bugs.launchpad.net/bugs/839609

Title:
  [11.10 beta1] UnicodeDecodeError crash on localized input in multiple
  encodings/languages

Status in command-not-found handler:
  Fix Released
Status in “command-not-found” package in Ubuntu:
  Fix Released

Bug description:
  The command-not-found package crashes on input of a simplified chinese
  character representing a bogus command. The problem was found with in
  11.10 beta1, for both the x86/i386 and amd64 systems. Debugging the
  python script in /usr/lib/command-not-found shows that a
  UnicodeDecodeError is thrown. The crash_guard() callback framework
  catches this and reports the error.

  Here are further observations.

  (1) With the same simplified chinese input, 11.04 handles the test case gracefully, returning a message
      explaining that the command is not found.

  (2) Between these two series, python has change: 11.04 (Python 2.7.1+)
  versus 11.10 beta1 (Python 2.7.2+).

  To elaborate on this problem, the following files have been included:

  (1) Screen shots showing step-by-step how to reproduce the bug. As switching to Simplified Chinese is
      difficult to explain in words, a video was taken to show how this process.

  (2) A screen shot showing /usr/lib/command-not-found script traced by means of the Python pdb module.
      This shows the zh_CN.UTF-8 byte stream input and the point where UnicodeDecodeError is thrown.

  This issue was investigated in 11.10 beta1 host running in VirtualBox.

  ===

  Taken from To_Reproduce_Bug.txt attachment.

  01_After_ISO_Installation.png   The VirtualBox VM with default English
  locale.

  02_Open_Lanugage_Support.png	Prepare to switch to Simplified Chinese locale.
  				See the accompanying video for this process.

  03_Enable_IBUS_Pinyin.png	After switching to Simplified Chinese. Note the locale 
  				environment variables. Click the IBUS keyboard icon 
  				and select Pinyin input.

  04_Pinyin_Enabled.png           Ready for Pinyin input. Note the blue
  IBUS icon.

  05_Type_Phonetic_Pinyin.png	Type in two letters: 'w' followed by 'o'. Phonetically these
  				correspond to the Chinese character representing 'I' or 'Myself'.
  				IBUS displays options. You want the first one. Hit the space
  				bar to choose it.

  06_Chinese_Input_Complete.png	Chinese 'wo' in zh_CN.UTF-8 is ready to be passed to the Bash.
                                  Hit the return key to do so.

  07_Crash_command_not_found.png  Bash calls command-not-found, which
  can't handle the input.

  08_Disable_Pinyin_Input.png     Instruct IBUS to disable Simplified
  Chinese input.

  ProblemType: Bug
  DistroRelease: Ubuntu 11.10
  Package: command-not-found 0.2.43ubuntu1 [modified: usr/lib/command-not-found]
  ProcVersionSignature: Ubuntu 3.0.0-9.15-generic 3.0.3
  Uname: Linux 3.0.0-9-generic x86_64
  Architecture: amd64
  Date: Fri Sep  2 10:23:08 2011
  InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Beta amd64 (20110901)
  PackageArchitecture: all
  SourcePackage: command-not-found
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/command-not-found/+bug/839609/+subscriptions