i386 autopkgtest [Re: i386 in focal: an update]

Tue Dec 10 19:03:30 UTC 2019

Hi all,

After a few days of bugfixing tests to be compatible with the new
autopkgtest cross-arch infrastructure, I thought it would be useful to give
folks some stats, and point you to a list of packages requiring further
investigation that you can help drive down.

Of 311 autopkgtests that we care about on i386, 24% are currently passing,
8% have been confirmed as not-cross-testable and blacklisted by the release
team, and 68% still need investigation.  Details, including a list of
packages to investigate, produced by the attached script:

success:			74
blacklisted:			26
failed, needs investigation:	211
acl
alglib
alsa-lib
ann
bind9
binutils
boost1.67
brotli
bubblewrap
bzip2
capnproto
cfitsio
cluster-glue
cmake
colord
confuse
coreutils
cunit
cups
cython
datefudge
dbus
devscripts
dietlibc
distro-info
dlm
docbook-to-man
double-conversion
doxygen
faketime
fenix
fenix-plugins
fftw3
fpc
freeglut
fribidi
fyba
game-data-packager
gcc-8
gcc-9
gcc-snapshot
gdbm
gem2deb
gl2ps
glibc
gnupg2
gpgme1.0
graphite2
grep
gtk+2.0
gtk+3.0
gvfs
gzip
harfbuzz
ibus
icu
inetutils
init-system-helpers
iproute2
jemalloc
json-glib
krb5
kronosnet
lapack
lbfgsb
libanyevent-perl
libarchive
libassuan
libasync-interrupt-perl
libb-hooks-op-check-perl
libbsd-resource-perl
libcaca
libcap2
libclone-perl
libcommon-sense-perl
libdaemon
libdatrie
libdevel-callchecker-perl
libevent-perl
libev-perl
libfile-libmagic-perl
libfilter-perl
libgd2
libgd-perl
libgpg-error
libgphoto2
libhtml-parser-perl
libinput
libio-pty-perl
libiptcdata
libjsoncpp
libkate
liblist-moreutils-perl
liblocale-gettext-perl
libmicrohttpd
libnetaddr-ip-perl
libnet-ssleay-perl
libnfs
libofa
libomxil-bellagio
libopenmpt
libparams-classify-perl
libparams-util-perl
libparams-validate-perl
libperlio-gzip-perl
libpipeline
libproxy
libpsl
libqb
libraw
libsass
libsdl2
libsdl-sge
libseccomp
libsecret
libsocket6-perl
libsort-key-perl
libsoup2.4
libsoxr
libssh
libsub-identify-perl
libsub-name-perl
libtaint-runtime-perl
libtest-leaktrace-perl
libtest-taint-perl
libtext-charwidth-perl
libtext-iconv-perl
libthai
libtommath
libunicode-utf8-perl
libusb-1.0
libutempter
libuuid-perl
libuv1
libvariable-magic-perl
libvisual
libvncserver
libvorbis
libwacom
libwant-perl
libxcb
libxml2
libxml-libxml-perl
libxml-parser-perl
libyaml-libyaml-perl
linux
livecd-rootfs
llvm-toolchain-9
matplotlib
mir
mpi4py
multipath-tools
ncurses
network-manager
nghttp2
nss-wrapper
numpy
openjdk-lts
openssh
openssl
pacemaker
perl
pillow
pinentry
poppler
postgresql-12
protobuf
pulseaudio
pupnp-1.8
python2.7
python3.7
python3.8
python-apt
python-bcrypt
python-cffi
python-coverage
python-cryptography
python-evdev
python-lz4
python-nacl
python-numpy
python-scipy
py-ubjson
pyyaml
raqm
re2
ruby2.5
ruby-json
ruby-nokogiri
samba
sane-backends
sdlgfx
sed
stunnel4
superlu
systemd
systemtap
talloc
tdb
twisted
ubuntu-drivers-common
uchardet
udisks2
upower
util-linux
valgrind
vtk6
vtk7
xterm
zope.interface
zsh

Cheers,
-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                   https://www.debian.org/
slangasek at ubuntu.com                                     vorlon at debian.org

On Fri, Dec 06, 2019 at 03:57:21PM -0800, Steve Langasek wrote:
> Hi folks,
> 
> The changes to i386 have proceeded as described in my earlier email.  The
> autopkgtest infrastructure has been converted to run tests of i386 binaries
> on an amd64 VM, and i386 binaries not in the whitelist are now in the
> process of being removed.  As of this moment, there are 1,679 source
> packages in the whitelist for i386, and 8,921 source packages with published
> i386 binary packages in focal, vs. 14,344 source packages with published
> amd64 binaries.  At the current rate, I expect the binary removals to
> complete some time tomorrow.
> 
> On the autopkgtest side, there are now a total of 311 source packages
> producing i386 binaries which also have autopkgtests[1].  Any packages
> reporting autopkgtest regressions on i386 which are not on this list should
> have their test failures ignored; if you see i386 autopkgtest failures for
> other packages blocking migrations out of focal-proposed, please ask a
> member of the release team to override.
> 
> It's good that the number of remaining autopkgtests for i386 is as small as
> it is, because while the /infrastructure/ is now capable of cross-testing
> i386 packages on amd64, the vast majority of these tests don't run correctly
> in such an environment without modification.  I have triggered runs of all
> of these tests on the amd64 testbed, so current status of the tests is
> visible on autopkgtest.ubuntu.com for each package.
> 
> Since these tests have also de facto regressed in the release pocket due to
> changes to the infrastructure, it is appropriate for those test failures to
> be ignored for proposed-migration purposes as well.  I intend to go through
> and systematically mark these as bad tests, but in the meantime if you have
> package migrations that are being blocked by one of these, please reach out.
> 
> 
> The other thing that you can do is help fix the failing autopkgtests, so
> that we have better test coverage on i386.  I've identified a few common
> patterns so far in the test failures:
> 
>  - Tests of Essential or Priority: required/import packages that require removing
>    the amd64 package in favor of the i386 package (e.g.: bash, apt,
>    systemd).  These tests will never work, and - outside of launchpad
>    builds, where we are still building i386 natively instead of on an amd64
>    host - these packages aren't relevant to the i386 experience on focal. 
>    These test failures should be permanently ignored.
> 
>  - Tests with test dependencies on python on perl modules.  It's a
>    longstanding known issue with multiarch that python and perl modules pose
>    problems for cross-installation, because they depend on the interpreter
>    of the matching architecture.  Neither perl nor python can be
>    cross-installed on the testbed (the amd64 versions are both part of the
>    test infrastructure), so these tests will always fail with 'badpkg'. 
>    While some of these tests may be more relevant to the behavior of our
>    i386 runtime libraries, fixing them is largely intractable so these are a
>    low priority.
> 
>  - Tests with other uninstallable test dependencies ("badpkg").  Many of
>    these are going to be fixable, but require either annotations of
>    additional packages in the archive as Multi-Arch: foreign (if
>    appropriate), or annotations of the test dependencies in
>    debian/tests/control with architecture information.  The autopkgtest
>    dependency resolver for cross-testing uses apt-get build-dep, so that the
>    full range of multiarch build-dependency specifiers are available as
>    described at <https://wiki.ubuntu.com/MultiarchCross>.
> 
>  - Build tests.  These are common tests for libraries, and some of the
>    easiest to fix, but most don't work out of the box because they don't
>    invoke the cross-toolchain.  Here are some example patches for various
>    types of build systems used in build tests:
>    - pkg-config: https://bugs.debian.org/946303
>    - meson: https://bugs.debian.org/946292
> 
> Happy fixing,
> -- 
> Steve Langasek                   Give me a lever long enough and a Free OS
> Debian Developer                   to set it on, and I can move the world.
> Ubuntu Developer                                   https://www.debian.org/
> slangasek at ubuntu.com                                     vorlon at debian.org
> 
> [1] See attachment.  List calculated by:
> $ join -j1 <(wget -O - -q -o /dev/null \
>                https://people.canonical.com/~ubuntu-archive/germinate-output/i386.focal/i386+build-depends.sources \
>              | awk '{ print $1 }' | sort) \
>            <(cat /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_focal_*Sources \
>              | grep-dctrl -r -FTestsuite . -a '!' -FArchitecture -X all \
>                           -sPackage -n | sort)
> 
> On Thu, Nov 28, 2019 at 12:46:34PM -0800, Steve Langasek wrote:
> > Thanks to thorough feedback from our community, we now have a reasonably
> > comprehensive answer to the question of what 32-bit compatibility library
> > packages are needed on x86 for Ubuntu 20.04. 
> > 
> > https://discourse.ubuntu.com/t/community-process-for-32-bit-compatibility/12598/46
> > 
> > Some developers will have noticed changes this week to the behavior of focal
> > builds in Launchpad.  Out of 30,000 source packages in focal, there is now a
> > whitelist of about 1,700 source packages which will trigger builds on i386
> > in Launchpad.  This means that other packages which previously built on i386
> > will need to have the binaries from the old version of the package removed
> > before they will be migratable from focal-proposed to focal.
> > 
> > 
> > As a side note, the implementation of this also affects PPA builds, because
> > the whitelist applies to the focal series as a whole.  In general, you
> > should not expect to need i386 builds of third-party packages in PPAs for
> > focal either, given that i386 in focal exists solely for compatibility with
> > legacy binary software.  However, if you have a third-party package that you
> > believe it's important to continue producing i386 binary builds of in
> > Launchpad for Ubuntu 20.04, please contact the Ubuntu archive admins
> > (ubuntu-release at lists.ubuntu.com, or #ubuntu-devel on freenode.net for best
> > results), and we can evaluate including your PPA package in the whitelist.
> > 
> > 
> > At the moment, I am doing some manual removals of the i386 binaries as I see
> > them show up as blockers on
> > https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html
> > and as I'm able to determine that the removals aren't going to cause
> > near-term knock-on problems.  But if some i386 binaries aren't being removed
> > fast enough and this is blocking your work, feel free to reach out to an
> > archive admin to ask for their removal.
> > 
> > In the slightly less near term, the plan is to do a mass binary removal of
> > all of the i386 binary packages in focal built from sources other than those
> > in the whitelist.  However, before pulling the trigger on this mass removal,
> > there are some changes that should be landed to our autopkgtest
> > infrastructure, so that we can continue to run autopkgtests for those
> > remaining 1700 packages.  In summary: the plan is not to retain the test
> > dependencies of those 1700 packages on i386, but instead to cross-test the
> > i386 libraries on an amd64 host, which ultimately means testing them in an
> > environment that better models the expected real-world usage.  The work is
> > in progress for this change and I'm currently anticipating landing it next
> > week.
> > 
> > In the meantime, if you need any help getting packages migrating to the
> > focal release from -proposed, please reach out on #ubuntu-release on IRC.
> > 
> > Thanks,
> > -- 
> > Steve Langasek                   Give me a lever long enough and a Free OS
> > Debian Developer                   to set it on, and I can move the world.
> > Ubuntu Developer                                   https://www.debian.org/
> > slangasek at ubuntu.com                                     vorlon at debian.org

> acl
> alglib
> alsa-lib
> ann
> aom
> apache2
> apparmor
> apt
> atk1.0
> at-spi2-atk
> at-spi2-core
> attr
> bash
> bind9
> binutils
> bluez
> boost1.67
> brotli
> bubblewrap
> bzip2
> cairo
> capnproto
> ceph
> cfitsio
> cluster-glue
> clutter-1.0
> cmake
> colord
> confuse
> coreutils
> corosync
> cunit
> cups
> cython
> dash
> datefudge
> dbus
> dbus-glib
> dbus-python
> dconf
> devscripts
> dietlibc
> distro-info
> dlm
> docbook-to-man
> double-conversion
> doxygen
> dpkg
> dxvk
> ed
> espeak
> espeak-ng
> evolution-data-server
> fakeroot
> faketime
> faudio
> fenix
> fenix-plugins
> ffmpeg
> fftw3
> fig2dev
> firebird3.0
> flite
> fluidsynth
> fpc
> freeglut
> fribidi
> fyba
> game-data-packager
> gcc-8
> gcc-9
> gcc-snapshot
> gconf
> gcr
> gdbm
> gdisk
> gdk-pixbuf
> gem2deb
> geocode-glib
> gl2ps
> glib2.0
> glibc
> glib-networking
> gnupg2
> gnuplot
> gnutls28
> gobject-introspection
> gpgme1.0
> graphene
> graphite2
> graphviz
> grep
> gspell
> gtk+2.0
> gtk+3.0
> gtkmm2.4
> gvfs
> gzip
> harfbuzz
> hwloc
> ibus
> icu
> imagemagick
> inetutils
> initramfs-tools
> init-system-helpers
> iproute2
> iptables
> jdupes
> jemalloc
> json-glib
> kconfig
> keyutils
> krb5
> kronosnet
> ladspa-sdk
> lapack
> lbfgsb
> leveldb
> libanyevent-perl
> libarchive
> libassuan
> libasync-interrupt-perl
> libb-hooks-op-check-perl
> libbsd-resource-perl
> libcaca
> libcap2
> libclone-perl
> libcommon-sense-perl
> libdaemon
> libdatrie
> libdevel-callchecker-perl
> libevent-perl
> libev-perl
> libfile-libmagic-perl
> libfilter-perl
> libgd2
> libgdata
> libgd-perl
> libglib-perl
> libgpg-error
> libgphoto2
> libhtml-parser-perl
> libical3
> libinput
> libio-pty-perl
> libiptcdata
> libjsoncpp
> libkate
> liblinux-epoll-perl
> liblist-moreutils-perl
> liblocale-gettext-perl
> libmicrohttpd
> libnetaddr-ip-perl
> libnet-ssleay-perl
> libnfs
> libnotify
> libofa
> liboggz
> libomxil-bellagio
> libopenmpt
> libparams-classify-perl
> libparams-util-perl
> libparams-validate-perl
> libperlio-gzip-perl
> libpipeline
> libproxy
> libpsl
> libqb
> libraw
> librsvg
> libsass
> libsdl2
> libsdl-sge
> libseccomp
> libsecret
> libsocket6-perl
> libsort-key-perl
> libsoup2.4
> libsoxr
> libssh
> libstb
> libsub-identify-perl
> libsub-name-perl
> libtaint-runtime-perl
> libtest-leaktrace-perl
> libtest-taint-perl
> libtext-charwidth-perl
> libtext-iconv-perl
> libthai
> libtheora
> libtommath
> libunicode-utf8-perl
> libusb-1.0
> libutempter
> libuuid-perl
> libuv1
> libvariable-magic-perl
> libvisual
> libvncserver
> libvorbis
> libvpx
> libwacom
> libwant-perl
> libwmf
> libxcb
> libxml2
> libxml-libxml-perl
> libxml-parser-perl
> libyaml-libyaml-perl
> libzstd
> linux
> livecd-rootfs
> llvm-toolchain-9
> matplotlib
> mawk
> mir
> mpi4py
> mpich
> multipath-tools
> munge
> mysql-8.0
> ncurses
> ndctl
> network-manager
> nghttp2
> ninja-build
> nss-mdns
> nss-wrapper
> numpy
> openjdk-lts
> openldap
> openssh
> openssl
> pacemaker
> pango1.0
> perl
> php7.3
> pillow
> pinentry
> policykit-1
> poppler
> postgresql-12
> protobuf
> pulseaudio
> pupnp-1.8
> pygobject
> pypy
> pyqt5
> python2.7
> python3.7
> python3.8
> python-apt
> python-bcrypt
> python-cffi
> python-coverage
> python-cryptography
> python-evdev
> python-lz4
> python-nacl
> python-numpy
> python-scandir
> python-scipy
> py-ubjson
> pyyaml
> raqm
> re2
> re2c
> rhash
> rsync
> ruby2.5
> ruby-defaults
> ruby-json
> ruby-nokogiri
> samba
> sane-backends
> sdlgfx
> sed
> snapd-glib
> sonic
> spdlog
> stunnel4
> superlu
> svox
> swig
> systemd
> systemtap
> talloc
> tdb
> twisted
> ubuntu-drivers-common
> uchardet
> udisks2
> umockdev
> update-notifier
> upower
> util-linux
> valgrind
> vim
> vlc
> vorbis-tools
> vtk6
> vtk7
> x264
> x265
> xapian-core
> xdg-dbus-proxy
> xterm
> zope.interface
> zsh
> zvbi

-------------- next part --------------
#!/bin/sh

set -e

pass=
blacklist=
fail=

wget -c http://autopkgtest.ubuntu.com/static/autopkgtest.db
[ -d hints-ubuntu ] || bzr branch lp:~ubuntu-release/britney/hints-ubuntu/

while read source; do
	exitval=$(sqlite3 /tmp/autopkgtest.db "SELECT r.exitcode FROM current_version cv LEFT JOIN test t ON cv.release=t.release AND cv.package=t.package LEFT JOIN result r ON t.id=r.test_id WHERE cv.release='focal' AND r.version=cv.version AND t.arch='i386' AND cv.package='$source' ORDER BY r.run_id DESC LIMIT 1;")
	if [ $exitval -eq 0 ] || [ $exitval -eq 2 ] || [ $exitval -eq 8 ]; then
		pass="$pass $source"
	elif grep -q "force-badtest $source/all/i386" hints-ubuntu/ubuntu-release
	then
		blacklist="$blacklist $source"
	else
		fail="$fail $source"
	fi
done < /tmp/i386-whitelisted-autopkgtests.txt

goodcount=$(echo $pass | wc -w)
badcount=$(echo $fail | wc -w)
blacklistcount=$(echo $blacklist | wc -w)
echo "success:			$goodcount"
echo "blacklisted:			$blacklistcount"
echo "failed, needs investigation:	$badcount"
for i in $fail; do
	echo $i
done
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-devel/attachments/20191210/0d7a3de3/attachment.sig>