Analyzing migrations: britney, update_output.txt, chdist, dose-distcheck, apt solver 3

[Adrien Nader]

Hi,

This is a documentation message rather than a report. Let's face it: I'm
at the stage where I barely understand some parts of britney's reports
but as everyone knows, this is also the stage where we feel the most
confident teaching about things.

This is written as a story rather than documentation with a clear plan
but I'll forgive myself as I'll be talking about things which are
basically not documented and very little known: anything is an
improvement!

Finally, let's be frank: you won't read this e-mail in full, it's too
long and it will feel too remote from your day-to-day tasks.

But! You should however get a high-level idea of what it's talking about
and come back when you have a use what is discussed here.


# Context


As part of my +1 shift last week (this was written 6 days ago), I was
using my rewritten update_excuses page:
https://ubuntu.dcln.fr/update_excuses.html

(I announced it here in an another e-mail a few minutes ago)

In this page, excuses are categorized. I've designed the categories as
best as I could to make them meaningful (I know there are some things to
improve). At some point on Tuesday, the categories and corresponding
excuses count were as follows:

- needing attention                     254
- blocked by another                     65
- merely waiting                        106
- no issues so far                        6
- waiting for another item's results      9
- requiring approval                      1
- Britney missing information           164

The high count for "merely waiting" surprised me as this means britney
says the following:

    Will attempt migration (Any information below is purely
    informational)

Several of these are very recent (expected) but many are a month old!
Britney says it's going to migrate them, why is that not happening?

I noticed kicad there, I love kicad, let's look at it.
The report says it should migrate but it's five days old already.
There's no other indication but it depends on opencascade. Opencascade
should also migrate too and it's seventeen days old. At this point, we
don't have an easy way to figure out the issue from the page (or rather,
we didn't have one).


# Britney's other output: update_output.txt


We can turn to update_output.txt which is documented in exactly two
places it seems:

- https://wiki.ubuntu.com/ProposedMigration#The_update_output.txt_file_is_completely_unreadable.21
- https://release.debian.org/doc/britney/short-intro-to-migrations.html

We look for kicad in update_output.txt.

    trying: kicad
    skipped: kicad (115, 1, 42)
        got: 5+0: a-3:a-0:a-0:i-0:p-0:r-1:s-1
        * riscv64: kicad

>From the "trying: kicad" chunk, we learn that kicad would make kicad
uninstallable on riscv64. That's not very helpful (at least to my
untrained eyes) and riscv64 is a red herring as britney only complains
about one architecture and riscv64 seems to be today's victim.

Kicad is also listed in the ipywidgets and opencascade blocks.

    skipped: ipywidgets (0, 3, 73)
        got: 5+0: a-4:a-0:a-0:i-0:p-0:r-0:s-1
        * amd64: python-ipywidgets-doc
    [...]
        * riscv64: f3d, getdp, getdp-sparskit, gmsh, kicad, libadacgi-dev,
          [...], ada-bar-codes opencascade slic3r-prusa
        - splitting the component into single items and retrying them
    trying: slic3r-prusa
    [...]
    trying: opencascade
    skipped: opencascade (0, 5, 142)
        got: 25+0: a-3:a-0:a-0:i-0:p-0:r-21:s-1
        * riscv64: f3d, getdp, getdp-sparskit, gmsh, kicad, [...]
    trying: ada-bar-codes
    [...]

It seems that installing either will make kicad uninstallable. There's a
large overlap between the set of packages that would become
uninstallable with these two packages. At that point I had no idea what
to look at next so I looked at the first package that would become
uninstallable in the lists: f3d.

Unlike kicad and opencascade, f3d is reported as blocked due to "another
item". Britney mentions "vtk9 (not considered)" and "opencascade". The
"not considered" part means that this dependency is stuck; don't ask me
why it's "not considered"; don't ask me about which words are used in
any of britney's messages.

Back to update_output.txt, we look for "f3d" and there's no "trying:
f3d": it seems it's only a victim and in any case, it seems we're not
going to get more out of update_output.txt for f3d.


# Back to update_excuses.html


We look at vtk9 and there's an issue with camitk tests on amd64:
dependencies cannot be installed. Logs show:

    76s Broken libvtk9.3:amd64 Breaks on libvtk9.1:amd64 < none @un H > (< 9.3.0+dfsg1-1)
    76s   Considering libvtk9.1t64:amd64 9 as a solution to libvtk9.3:amd64 8
    76s   Re-Instated libvtk9.1t64:amd64

The (first) answer is written but I wasn't satisfied with the short
answer and how tedious it was to get there. Moreover I was now wondering
why vtk9 was blocking kicad which doesn't depend on it.

Indeed, libvtk9.3 has a Breaks on libvtk9.1 and that's probably my issue
but I didn't want to stop at "probably". Moreover, kicad doesn't use
vtk9 so why am I looking at it?


# Getting conflict information for any release/arch with apt and chdist


Shengjing Zhu mentioned that I could use apt directly to get more infos.
At that point I didn't have a container running oracular but there's a
useful tool to create APT trees for several distributions without
actually running them: chdist.

    chdist create oracular http://archive.ubuntu.com/ubuntu oracular main universe
    # Enable -proposed too
    sed -i -n 'p; s/\<oracular\>/\0-proposed/ p' ~/.chdist/oracular/etc/apt/sources.list
    chdist apt oracular update

Then we can try to install probably-conflicting packages, this involves
finding the right names since we've only seen source package names so
far. We need to find a conflict and much to my surprise, "kicad
camitk-actionstatemachine" didn't trigger one. Luckily, adding "vtk9"
helped:

    chdist apt oracular install --assume-no -t oracular-proposed kicad camitk-actionstatemachine vtk9

    The following packages have unmet dependencies:
     libvtk9.3 : Breaks: libvtk9.1 (< 9.3.0+dfsg1-1)
    E: Unable to correct problems, you have held broken packages.

Wonderful, we've replicated the error it seems. We also have a clean
error message while I had a more complex one when I tried with a
different set of packages (can't remember which one). It tells us we
have incompatibilities between versions of libraries but it doesn't tell
us which packages try to pull each version.


# apt --solver 3.0


At that point my oracular lxd image had finished downloading, I enabled
proposed, installed camitk-actionstatemachine and then f3d from
proposed:

    apt install --solver 3.0 -o APT::Get::Show-Versions=1 -t oracular-proposed f3d

I had been showing package versions too, which was helpful. The apt in
oracular has the "--solver 3.0" option to enable the new (experimental)
solver which I wanted to look at. Reformatted:

    Error: Conflict: 
     -> camitk-actionstatemachine:amd64
    but
        f3d:amd64=2.4.0+dfsg-2build2
     -> libvtk9.3:amd64=9.3.0+dfsg1-1
     -> not libvtk9.1t64:amd64=9.1.0+really9.1.0+dfsg2-8
     -> not libvtkgdcm-9.1t64:amd64=3.0.24-1
     -> not libcamitk5:amd64=5.2.0-1build7
     -> not camitk-actionstatemachine:amd64=5.2.0-1build7
     -> not camitk-actionstatemachine:amd64

We're not comparing apples-to-apples though: the set of packages is
different but I can attest the error message is clearer than what I had
on my machine without solver 3.
(but, there are still some things to improve probably)

Except that kicad is still nowhere to be seen. The issue is that kicad
alone has no issues. It's only the combination of everything that is
problematic. We've been looking at only one package at a time or maybe a
couple at once but britney looks at everything.


# Experiments with dose-distcheck


Enter dose-distcheck:

    dose-distcheck --failures --explain --coinst 'vtk9 (=9.3.0+dfsg1-1),camitk-actionstatemachine' $(printf -- " --fg deb://%s" /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_oracular*_Packages)

I know, my commands have only gotten more and more obscures.

There's a bit of little-known printf magic: try "printf '%s\n' a b c"
and see how the format specifier is reused. This crafts part of the
command-line but that's only convenience.

The important part is what to use with --coinst: we pick vtk9 at its
version from proposed and camitk-actionstatemachine at any version. It
feels like this selection could be automated if it were not for what is
probably a small bug in dose unfortunately.
NB: I didn't try apt with a forced versions for vtk9

The output from dose-distcheck is fairly long but it shows the right
things:

    pkg1:
     package: libvtk9.3
     version: 9.3.0+dfsg1-1
     architecture: amd64
     unsat-conflict: libvtk9.1 (< 9.3.0+dfsg1-1)
    pkg2:
     package: libvtk9.1t64
     version: 9.1.0+really9.1.0+dfsg2-8
     architecture: amd64

    [...]
      depchain:
       -
        package: camitk-actionstatemachine
        version: 5.2.0-1build7
        architecture: amd64
        depends: libvtk9.1t64 (>= 9.1.0+really9.1.0+dfsg2)

We see that camitk-actionstatemachine depends on "libvtk9.1t64 (>=
9.1.0+really9.1.0+dfsg2)" but we're also trying to pull libvtk9.3
9.3.0+dfsg1-1 which has a conflict with "libvtk9.1 (< 9.3.0+dfsg1-1)".

Julian told me that apt's solver 3 should allow for an apt-distcheck.
At the moment it doesn't (publicly?) exist AFAICT but knowledge should
be somewhat transferable between the two.

At this point, we see we need to rebuild camitk against vtk 9.3 which
should hopefully be a no-change rebuild and that's what Gianfranco
Costamagna did in
https://launchpad.net/ubuntu/+source/camitk/5.2.0-1build8 .

Unfortunately it didn't build! The build log shows us an issue with vtk9
again and "libinsighttoolkit5-dev" which we can feed to dose-distcheck
with

    --coinst 'vtk9 (=9.3.0+dfsg1-1),libinsighttoolkit5-dev'

And we see that libinsighttoolkit5.3:

    depends: libvtk9.1 (>= 9.1.0+really9.1.0+dfsg2) | libvtk9.1 (>= 9.1.0+really9.1.0+dfsg2)

Same issue. We can try with apt --solve 3.0 too:

    Error: Conflict:
       libinsighttoolkit5-dev:amd64=5.3.0-7build4
    -> libinsighttoolkit5.3:amd64=5.3.0-7build4
    -> libvtk9.1t64:amd64=9.1.0+really9.1.0+dfsg2-8
    but
       vtk9:amd64=9.3.0+dfsg1-1
    -> libvtk9.3:amd64=9.3.0+dfsg1-1
    -> not libvtk9.1t64:amd64=9.1.0+really9.1.0+dfsg2-8

Same thing. I find dose-distcheck's output clearer (which is quite
expected considering it's the purpose of the tool and it can afford to
be more verbose).

At the moment, no upload for a no-change rebuild of insighttoolkit5 has
been done.


# Finding reverse-dependencies with grep-dctrl


Finally, we can find which packages need rebuilds with the following:

    comm -23 \
      <(grep-dctrl --no-field-names --field=Depends --show-field=Source:Package libvtk9.1 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_oracular_*Packages | sort -u) \
      <(grep-dctrl --no-field-names --field=Depends --show-field=Source:Package libvtk9.3 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_oracular-proposed_*Packages | sort -u)

With grep-dctrl we find all source packages of packages that:

  a) depend on libvtk9.1 in oracular
  b) depend on libvtk9.3 in oracular-proposed

Using comm, we compute the set differences a) minus b) which gives us
the list of packages in oracular with no appropriate rebuild in
oracular-proposed.

Results (as of Tuesday evening; this includes builds which failed or are
waiting for dependencies):

    camitk
    gdcm
    insighttoolkit5
    odin
    opencv
    sight
    therion

There could be other vtk9 libraries than libvtk9.1/3, I didn't look.


# And kicad?


Well, I think it's really only waiting. It needs the opencascade from
oracular-proposed which would make oracular's f3d uninstallable as we've
seen. Therefore, for opencascade to migrate (and thus kicad), f3d and
vtk9 need to migrate too, which needs a bunch of rebuilds as we've seen.


# A look at debian


In Debian, camitk needs to be rebuilt against the new vtk9 library.
However, opencascade and kicad have already migrated. Somehow they
haven't been entangled in the vtk9 transition. There's probably less
incentive to do a no-change rebuild of camitk and others there compared
to Ubuntu since fewer packages are probably held back (although camitk
seems to depends on a lot of libraries and there are other transitions
ongoing).

On a related note, I have no idea if merge-o-matic can efficiently
handle no-change rebuilds in Ubuntu.


# Closing thoughts


I think two main things are missing in update_excuses.html (and my
rewrite).

First, for each package involved in dependencies, I've added a list of
packages which are being blocked by any given migration: you are used to
see "kicad is blocked by opencascade" and I've added "opencascade blocks
kicad, f3d, ..." and which lets you navigate from opencascade to f3d and
then vtk9.

Second, it would probably be beneficial to have the list of old and new
package names in migrations if they are different. (and then show
package status when hovering over a package's name!)

Also, interestingly, https://github.com/mclemenceau/visual-excuses/
doesn't show the issue. I think this is a testament to how difficult it
is to make sense of britney's output more than anything else.

That was a lot. I'll probably forget some of this myself. I hope it is
useful to others. Comments and corrections welcome.

-- 
Adrien