[Bug 1655447] Re: IEEE indexes in netaddr package may not match data in ieee-data package

[Andres Rodriguez]

** Changed in: maas
    Milestone: 2.4.0beta4 => 2.4.x

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to python-netaddr in Ubuntu.
https://bugs.launchpad.net/bugs/1655447

Title:
  IEEE indexes in netaddr package may not match data in ieee-data
  package

Status in MAAS:
  Triaged
Status in MAAS 2.3 series:
  Triaged
Status in python-netaddr package in Ubuntu:
  Confirmed

Bug description:
  Before reading further, note that this bug report is for the bug in
  the Debian package for `netaddr`. Bug #1655452 is for the bug in
  upstream `netaddr` code (related to cached index coherency).

  ---

  The Debian package for `netaddr` ships with a pre-calculated index for
  entries in the IEEE OUI and IAB files, which the `ieee-data` package
  places here:

  $ dpkg -L python-netaddr | grep idx$
  /usr/lib/python2.7/dist-packages/netaddr/eui/oui.idx
  /usr/lib/python2.7/dist-packages/netaddr/eui/iab.idx

  The index files only work if they match the IEEE data at the time the
  `netaddr` package was built. If anything changes in the IEEE data, the
  seek locations in the index will be thrown off, causing undefined
  behavior.

  However, the way `netaddr` is packaged doesn't guarantee that the
  index will be kept in sync with the data files in the `ieee-data`
  pacakge. In the `netaddr` packages, symbolic links are created to the
  data files in the `ieee-data` package as follows:

  $ dpkg -L ieee-data | grep usr.share.*.txt
  /usr/share/ieee-data/iab.txt
  /usr/share/ieee-data/oui.txt

  $ ls -la /usr/lib/python2.7/dist-packages/netaddr/eui/*.txt
  lrwxrwxrwx 1 root root 38 Oct 23  2015 /usr/lib/python2.7/dist-packages/netaddr/eui/iab.txt -> ../../../../../share/ieee-data/iab.txt
  lrwxrwxrwx 1 root root 38 Oct 23  2015 /usr/lib/python2.7/dist-packages/netaddr/eui/oui.txt -> ../../../../../share/ieee-data/oui.txt

  Upstream, `netaddr` ships with its own copy of these files. During the
  build process, `netaddr` generates an index file for fast lookups into
  the OUI and IAB text files.

  However, there is a problem with this arrangement that only occurs in
  the Debian package: if the `ieee-data` on a given system is out-of-
  sync with the index in the `python-netaddr` or `python3-netaddr`, the
  undefined behavior begins.

  In MAAS, we make heavy use of netaddr, and the symptoms of the index
  file being out of date have ranged from an IndexError being raised to
  UnicodeDecodeError, since `netaddr` assumes that the index will always
  match the data file, and caches the index for the life of the
  process.[1]

  Symptoms
  ========

  This bug can cause `netaddr` to raise tracebacks during operation,
  such as `IndexError`, and possibly others, as a result of the
  incorrect indexes. For examples, see also:

      Bug #1655049
      Bug #1628761

  Suggested ways to fix
  =====================

  (1) The `netaddr` package MUST always be in a state where the index
  file is consistent with the IEEE data.

  (1a) The `netaddr` package COULD ship with a copy of the IEEE data
  files as-designed, so that there is never a mismatch regarding the on-
  disk data. (This is the simplest fix, but is likely a Debian policy
  violation due to the duplication of data.)

  (1b) The `netaddr` package COULD make a copy of the current state of
  the OUI and IAB indexes in `ieee-data` when the package is installed
  or updated. (Perhaps hard links would work, so that `netaddr` can
  continue working with the old data until the indexes are regenerated.
  But it would need to fall back to a data copy in case of cross-device
  links.)

  (1c) If the `ieee-data` package changes, and the `netaddr` is using
  the data from it, the `netaddr` package MUST correspondingly update
  its data files and index.

  (2) The `netaddr` upstream code MUST detect if the index file changes
  and reload the index (to be usable in a long-running process backed by
  a Debian package that could be updated at any time).

  Short term, the most important thing needed is a fix for (1), so that
  will be the focus for this bug.

  Workaround
  ==========

  Regenerate the indexes (for python 2.x and 3.x) as root by running:

      sudo python -m netaddr.eui.ieee

      sudo python3 -m netaddr.eui.ieee

  Either or both commands may be needed, depending on which `netaddr`
  packages are installed.

  Then, restart any long-running processes (such as MAAS) which may rely
  on `netaddr` for OUI lookups.

  Note that even if this bug is fixed, the "long-running process" issue
  still exists. To fix that, (2) above needs to be resolved.

  ---

  [1]: That means there is a related issue: long-running processes will
  be exposed to the same symptoms of this bug, if the `netaddr` package
  is upgraded while the long-running process is continues to run.
  Arguably, this is a bug that upstream should address by recalculating
  the index if the data file changes, and/or forcing an index
  recalculation if an integrity check fails, such as by storing a size
  and hash of the file. I assume it would be counter-productive to do
  too many integrity checks, though, because by the time you get done
  with them, you may as well have recalculated the entire index. (And on
  a slow embedded system, this might be unacceptable.)

To manage notifications about this bug go to:
https://bugs.launchpad.net/maas/+bug/1655447/+subscriptions