[Bug 1309447] Re: Unicode decode error during upgrade to 14.04 if sources.list contains non-ascii characters and locale is non-US

[Mekk]

>From the comment which I see in this patch ("open in unicode mode..."),
it looks like somebody intended to go towards handling file content as
unicode. Your patch reverts it towards binary. Maybe it would be better
to attempt input decoding (guessing the encoding somehow, at least
considering utf-8 and locale-preferred encoding)? I don't remember
what/how this exactly  works in python3, so feel free to ignore this
remark.

(sidenote: I am fairly curious what would happen if I saved some config
files as utf-16 ;-))

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apt-clone in Ubuntu.
https://bugs.launchpad.net/bugs/1309447

Title:
  Unicode decode error during upgrade to 14.04 if sources.list contains
  non-ascii characters and locale is non-US

Status in “apt-clone” package in Ubuntu:
  Triaged

Bug description:
  TEST CASE:
  1. On Saucy or Precise add non-ascii characters to the file /etc/apt/sources.list like:

  #  Français accentué
  deb http://archive.ubuntu.com/ubuntu saucy main restricted universe multiverse
  deb http://archive.ubuntu.com/ubuntu saucy-updates main restricted universe multiverse
  deb http://security.ubuntu.com/ubuntu saucy-security main restricted universe multiverse

  2. Upgrade with a non-us locale
  LANG=fr_FR.utf8 do-release-upgrade

  ACTUAL RESULT
  This crash:
  Traceback (most recent call last):
    File "/tmp/ubuntu-release-upgrader-ui8dvi/trusty", line 10, in <module>
      sys.exit(main())
    File "/tmp/ubuntu-release-upgrader-ui8dvi/DistUpgrade/DistUpgradeMain.py", line 240, in main
      save_system_state(logdir)
    File "/tmp/ubuntu-release-upgrader-ui8dvi/DistUpgrade/DistUpgradeMain.py", line 133, in save_system_state
      scrub_sources=True)
    File "/tmp/ubuntu-release-upgrader-ui8dvi/DistUpgrade/apt_clone.py", line 150, in save_state
      self._write_state_sources_list(tar, scrub_sources)
    File "/tmp/ubuntu-release-upgrader-ui8dvi/DistUpgrade/apt_clone.py", line 231, in _write_state_sources_list
      "./etc/apt/sources.list")
    File "/tmp/ubuntu-release-upgrader-ui8dvi/DistUpgrade/apt_clone.py", line 251, in _add_file_to_tar_with_password_check
      for line in f.readlines():
    File "/usr/lib/python2.7/encodings/ascii.py", line 26, in decode
      return codecs.ascii_decode(input, self.errors)[0]
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128)

  == Original report ==
  do-release-upgrade fails as below. I am using Polish utf-8 locale, so I suspect something called reports error in Polish with non-ascii characters...

  File "/tmp/ubuntu-release-upgrader-j2xvu5/trusty", line 10, in
  <module>
  sys.exit(main())

  File
  "/tmp/ubuntu-release-upgrader-j2xvu5/DistUpgrade/DistUpgradeMain.py",
  line 240, in main
  save_system_state(logdir)

  File
  "/tmp/ubuntu-release-upgrader-j2xvu5/DistUpgrade/DistUpgradeMain.py",
  line 133, in save_system_state
  scrub_sources=True)

  File "/tmp/ubuntu-release-upgrader-j2xvu5/DistUpgrade/apt_clone.py",
  line 150, in save_state
  self._write_state_sources_list(tar, scrub_sources)

  File "/tmp/ubuntu-release-upgrader-j2xvu5/DistUpgrade/apt_clone.py",
  line 231, in _write_state_sources_list
  "./etc/apt/sources.list")

  File "/tmp/ubuntu-release-upgrader-j2xvu5/DistUpgrade/apt_clone.py",
  line 251, in _add_file_to_tar_with_password_check
  for line in f.readlines():

  File "/usr/lib/python2.7/encodings/ascii.py", line 26, in decode
  return codecs.ascii_decode(input, self.errors)[0]

  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position
  2604: ordinal not in range(128)

  ProblemType: Bug
  DistroRelease: Ubuntu 13.10
  Package: ubuntu-release-upgrader-core 1:0.205.5
  ProcVersionSignature: Ubuntu 3.11.0-19.33~ppa1-generic-tuxonice 3.11.10.5
  Uname: Linux 3.11.0-19-generic-tuxonice x86_64
  NonfreeKernelModules: fglrx
  ApportVersion: 2.12.5-0ubuntu2.2
  Architecture: amd64
  CrashDB: ubuntu
  Date: Fri Apr 18 11:54:23 2014
  DistributionChannelDescriptor:
   # This is a distribution channel descriptor
   # For more information see http://wiki.ubuntu.com/DistributionChannelDescriptor
   canonical-oem-somerville-oneiric-amd64-20111116-1
  EcryptfsInUse: Yes
  InstallationDate: Installed on 2012-05-07 (711 days ago)
  InstallationMedia: Ubuntu 11.10 "Oneiric" - Build amd64 LIVE Binary 20111116-18:24
  MarkForUpload: True
  PackageArchitecture: all
  SourcePackage: ubuntu-release-upgrader
  Symptom: release-upgrade
  UpgradeStatus: Upgraded to saucy on 2014-04-18 (0 days ago)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt-clone/+bug/1309447/+subscriptions