[Bug 1473533] Re: CountryNameDict function trying to parse UTF-8 iso3166.tab as US-ASCII

[Steve Langasek]

Hello Martin, or anyone else affected,

Accepted python-tz into trusty-proposed. The package will build now and
be available at https://launchpad.net/ubuntu/+source/python-tz/2012c-
1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
enable and use -proposed.  Your feedback will aid us getting this update
out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-needed to verification-done. If it does not fix the
bug for you, please add a comment stating that, and change the tag to
verification-failed.  In either case, details of your testing will help
us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: python-tz (Ubuntu Trusty)
       Status: In Progress => Fix Committed

** Tags added: verification-needed

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to python-tz in Ubuntu.
https://bugs.launchpad.net/bugs/1473533

Title:
  CountryNameDict function trying to parse UTF-8 iso3166.tab as US-ASCII

Status in pytz:
  Fix Released
Status in python-tz package in Ubuntu:
  Fix Released
Status in python-tz source package in Precise:
  Fix Committed
Status in python-tz source package in Trusty:
  Fix Committed
Status in python-tz source package in Vivid:
  In Progress
Status in python-tz package in Debian:
  New

Bug description:
  [Impact]

   * The latest tzdata update change the content type for iso3166.tab
     and zone.tab to UTF-8 which causes an exception in pytz

  [Test Case]

  $ apt-get install -y python-tz python3-tz

  # These should produce exceptions for country_names.  This test
  # case also includes a test for country_timezones because the
  # tzdata file for zones has changed to use utf-8 but has yet
  # to include utf-8 characters.  You can hand edit 
  # /usr/share/zoneinfo/zone.tab to include a UTF-8 character
  # to force the exception and then test the proposed package.

  $ python -c 'import pytz
  for item in pytz.country_names.items():
    pass'

  $ python -c 'import pytz
  for item in pytz.country_timezones.items():
    pass'

  $ python3 -c 'from pytz import country_timezones
  for item in country_timezones.items():
    pass'

  $ python3 -c 'from pytz import country_names
  for item in country_names.items():
    pass'

  # A recreate will raise an exception like this:
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "/usr/lib/python3.4/_collections_abc.py", line 497, in __iter__
      for key in self._mapping:
    File "/usr/lib/python3/dist-packages/pytz/lazy.py", line 41, in __iter__
      self._fill()
    File "/usr/lib/python3/dist-packages/pytz/__init__.py", line 350, in _fill
      line = line.decode('US-ASCII')
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)

  [Regression Potential]

   * Older tzdata releases should be okay (and this has been tested)

  [Other Info]
   
   * None

  ---- Original Description ----

  Since tzdata-2015e there are UTF-8 characters in iso3166.tab, see:
  http://mm.icann.org/pipermail/tz/2015-May/022258.html
  http://mm.icann.org/pipermail/tz/2015-June/022306.html

  pytz/__init__.py:_CountryNameDict(LazyDict) is using:
          zone_tab = open_resource('iso3166.tab')
          try:
              for line in zone_tab.readlines():
                  line = line.decode('US-ASCII')

  to read it and fails on AX, CI, RE lines, using UTF-8 fixes the issues
  and should work OK even with older tzdata releases.

To manage notifications about this bug go to:
https://bugs.launchpad.net/pytz/+bug/1473533/+subscriptions