[Bug 1655447] Re: IEEE indexes in netaddr package may not match data in ieee-data package
andreserl at ubuntu-pe.org
Wed Feb 14 18:54:49 UTC 2018
** Changed in: maas
Milestone: 2.4.0alpha1 => 2.4.0alpha2
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to python-netaddr in Ubuntu.
IEEE indexes in netaddr package may not match data in ieee-data
Status in MAAS:
Status in MAAS 2.3 series:
Status in python-netaddr package in Ubuntu:
Before reading further, note that this bug report is for the bug in
the Debian package for `netaddr`. Bug #1655452 is for the bug in
upstream `netaddr` code (related to cached index coherency).
The Debian package for `netaddr` ships with a pre-calculated index for
entries in the IEEE OUI and IAB files, which the `ieee-data` package
$ dpkg -L python-netaddr | grep idx$
The index files only work if they match the IEEE data at the time the
`netaddr` package was built. If anything changes in the IEEE data, the
seek locations in the index will be thrown off, causing undefined
However, the way `netaddr` is packaged doesn't guarantee that the
index will be kept in sync with the data files in the `ieee-data`
pacakge. In the `netaddr` packages, symbolic links are created to the
data files in the `ieee-data` package as follows:
$ dpkg -L ieee-data | grep usr.share.*.txt
$ ls -la /usr/lib/python2.7/dist-packages/netaddr/eui/*.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/iab.txt -> ../../../../../share/ieee-data/iab.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/oui.txt -> ../../../../../share/ieee-data/oui.txt
Upstream, `netaddr` ships with its own copy of these files. During the
build process, `netaddr` generates an index file for fast lookups into
the OUI and IAB text files.
However, there is a problem with this arrangement that only occurs in
the Debian package: if the `ieee-data` on a given system is out-of-
sync with the index in the `python-netaddr` or `python3-netaddr`, the
undefined behavior begins.
In MAAS, we make heavy use of netaddr, and the symptoms of the index
file being out of date have ranged from an IndexError being raised to
UnicodeDecodeError, since `netaddr` assumes that the index will always
match the data file, and caches the index for the life of the
This bug can cause `netaddr` to raise tracebacks during operation,
such as `IndexError`, and possibly others, as a result of the
incorrect indexes. For examples, see also:
Suggested ways to fix
(1) The `netaddr` package MUST always be in a state where the index
file is consistent with the IEEE data.
(1a) The `netaddr` package COULD ship with a copy of the IEEE data
files as-designed, so that there is never a mismatch regarding the on-
disk data. (This is the simplest fix, but is likely a Debian policy
violation due to the duplication of data.)
(1b) The `netaddr` package COULD make a copy of the current state of
the OUI and IAB indexes in `ieee-data` when the package is installed
or updated. (Perhaps hard links would work, so that `netaddr` can
continue working with the old data until the indexes are regenerated.
But it would need to fall back to a data copy in case of cross-device
(1c) If the `ieee-data` package changes, and the `netaddr` is using
the data from it, the `netaddr` package MUST correspondingly update
its data files and index.
(2) The `netaddr` upstream code MUST detect if the index file changes
and reload the index (to be usable in a long-running process backed by
a Debian package that could be updated at any time).
Short term, the most important thing needed is a fix for (1), so that
will be the focus for this bug.
Regenerate the indexes (for python 2.x and 3.x) as root by running:
sudo python -m netaddr.eui.ieee
sudo python3 -m netaddr.eui.ieee
Either or both commands may be needed, depending on which `netaddr`
packages are installed.
Then, restart any long-running processes (such as MAAS) which may rely
on `netaddr` for OUI lookups.
Note that even if this bug is fixed, the "long-running process" issue
still exists. To fix that, (2) above needs to be resolved.
: That means there is a related issue: long-running processes will
be exposed to the same symptoms of this bug, if the `netaddr` package
is upgraded while the long-running process is continues to run.
Arguably, this is a bug that upstream should address by recalculating
the index if the data file changes, and/or forcing an index
recalculation if an integrity check fails, such as by storing a size
and hash of the file. I assume it would be counter-productive to do
too many integrity checks, though, because by the time you get done
with them, you may as well have recalculated the entire index. (And on
a slow embedded system, this might be unacceptable.)
To manage notifications about this bug go to:
More information about the Ubuntu-openstack-bugs