[Bug 1934992] Re: rsync 3.2.x in Groovy depends on broken libxxhash 0.7.x

Mon Jul 12 11:03:40 UTC 2021

Thank you for the further investigation.

It sounds like upstream broke ABI between libxxhash0 0.7 and libxxhash0
0.8 but did not bump the soname when they did so. Perhaps that's because
upstream still considers this experimental as it has a version < 1.0, or
perhaps that was a mistake. For a proper fix in the development release,
I'd expect upstream to bump the soname to make the symbol's behaviour
unambiguous. Without that, proper ABI compatibility is impossible since
different distributions and releases will have different behaviour for
the same soname and symbol depending on the version used to build the
library.

> To remedy that, I would suggest a rebuild of xxhash + libxxhash0 on
Focal where all xxh128 symbols/functionality is removed/blacklisted.

I'm not convinced. The use case you've presented is still of
low/unsupported priority and not properly resolved upstream. On the
other hand, we try not to break ABI in a stable release. And as pointed
out:

> Results produced by v0.7.x are not comparable with results
> from v0.7.y. However, the API is completely stable, and it
> can safely be used for ephemeral data (local sessions).

...so if a user has built against libxxhash for such a use case that is
supported by upstream, your proposal for changing Focal would break
them.

Leaving Focal as-is still seems like the least-worst answer to me.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to xxhash in Ubuntu.
https://bugs.launchpad.net/bugs/1934992

Title:
  rsync 3.2.x in Groovy depends on broken libxxhash 0.7.x

Status in xxhash package in Ubuntu:
  Triaged
Status in xxhash source package in Groovy:
  Won't Fix

Bug description:
  **NOTE: This ticket has been moved/altered. Original problem referred
  to rsync behaviour with libxxhash0. New subject is the changed ABI of
  libxxhash0.**

  Original-Subject: rsync 3.2.x in Groovy depends on broken libxxhash
  0.7.x

  Actual problem: libxxhash0 in Focal exposes 128-bit xxhash (xxh128)
  symbols which behave differently on versions of libxxhash0 in Groovy
  and above.

  In xxhash 0.7.x there is this notice:

  https://github.com/Cyan4973/xxHash/blob/173e50be0509c6fb6c1bb74d95049ef61d7fdced/xxhash.h#L465-L476

  > The XXH3 algorithm is still in development.
  > The results it produces may still change in future versions.
  >
  > Results produced by v0.7.x are not comparable with results
  > from v0.7.y. However, the API is completely stable, and it
  > can safely be used for ephemeral data (local sessions).
  >
  > Avoid storing values in long-term storage until the
  > algorithm is finalized. XXH3's return values will be
  > officially finalized upon reaching v0.8.0.

  Unfortunately these ended up as usable symbols in xxhash 0.7.x in
  Debian:

  > XXH128 at Base 0.7.0
  (etc..)

  This means that those linking against libxxhash0 would get a binary
  dependency on 0.7.0. But behaviour of xxh128 differs between 0.7 and
  0.8. So if you're linking against xxh128 from 0.8.x, it should binary-
  depend on >=0.8 and not >=0.7. (And if possible, you should not use
  xxh128 at all in 0.7.x.)

  The following Debian changeset fixes this for newer libxxhash0 builds:

  https://github.com/norbusan/debian-
  xxhash/commit/b51053c86952dd13573b08344101172c88de847f

  > debian/libxxhash0.symbols
  > - XXH128 at Base 0.7.0
  > + XXH128 at Base 0.8.0

  But, this is not fully resolved until:

  - that Debian change lands in libxxhash0 in Ubuntu;
  - packages linking against libxxhash0 (e.g. rsync) are rebuilt against this newer package, so the Depends is bumped to >=0.8;
  - xxh128 in libxxhash0 in Focal is clearly marked as ephemeral (or removed altogether; this applies to xxh128 symbols libxxhash0 and an `xxh128sum` binary in the xxhash package)

  Cheers,
  Walter Doekes
  OSSO B.V.

  ---vvv--- Original ticket below ---vvv---

  **Problem**

    $ rsync root at focal-system:/etc/.pwd.lock .
    ERROR: .pwd.lock failed verification -- update discarded.
    rsync error: some files/attrs were not transferred
      (see previous errors)
      (code 23) at main.c(1816) [generator=3.2.3]

    $ rsync root at focal-system:/etc/.pwd.lock . --debug=all
    opening connection using: ssh -l root focal-system rsync \
      --server --sender \
      -e.LsfxCIvu . /etc/.pwd.lock  (10 args)
    (Client) Protocol versions: remote=31, negotiated=31
    Client negotiated checksum: xxh128
    ...

  **Cause**

    focal-system# dpkg -l | grep -E 'libxxhash|rsync'
    ii  libxxhash0:amd64  0.7.3-1         amd64
    ii  rsync             3.2.3-2ubuntu1  amd64

  **Why this affects only us and not more people?**

  On Ubuntu/Focal, there is no rsync 3.2.3, only 3.1.3-8. But because we
  need the lz4 compression support we've fetched a newer rsync (from
  Groovy).

  However: the rsync 3.2.3 depends on libxxhash0 0.7.1+, while in fact
  it needs 0.8+.

  **Details**

  On a Ubuntu/Focal system we have installed a rsync 3.2.3 package from
  Ubuntu/Groovy because we need the lz4 compression support.

  focal-system# apt-cache show rsync
  Package: rsync
  ...
  Version: 3.2.3-2ubuntu1
  Depends: lsb-base, libacl1 (>= 2.2.23), libc6 (>= 2.15),
    liblz4-1 (>= 0.0~r130), libpopt0 (>= 1.14), libssl1.1 (>= 1.1.0),
    libxxhash0 (>= 0.7.1), libzstd1 (>= 1.3.8), zlib1g (>= 1:1.1.4)
  ...

  Alongside this we had libxxhash0 0.7.3-1 from Focal:

  focal-system# apt-cache policy libxxhash0
  libxxhash0:
    Installed: 0.7.3-1
    Candidate: 0.7.3-1
    Version table:
   *** 0.7.3-1 500
          500 http://ARCHIVE/ubuntu focal/universe amd64 Packages
          100 /var/lib/dpkg/status

  According to the dependencies, this should work. But the combination does not, as this quote from the rsync maintainer would tell you:
  https://github.com/WayneD/rsync/issues/122#issuecomment-737690913
  > Yeah, Cyan4973 could have told you that the 128-bit xxhash only
  > just stabilized in its 0.8.0 release, so anything older than
  > that isn't compatible.

  **The fix**

  As the maintainer points out, version 0.7 is not stable (= broken for
  our intents and purposes) and thus not fit for use with rsync 3.2.

  I would argue that it's a good idea to bump the dependency of rsync
  3.2.3 on Groovy to libxxhash0>=0.8

  After all, in Groovy there is a libxxhash0 0.8.0-1ubuntu1.20.10.1, so
  that would not be a problem. And it would fix issues for those mixing
  and matching packages.

  Thanks!

  Walter Doekes
  OSSO B.V.

  (*) possible patch:

  $ diff -pu debian/control{.orig,}
  --- debian/control.orig	2021-07-08 09:56:57.646861644 +0200
  +++ debian/control	2021-07-08 09:57:38.499029903 +0200
  @@ -8,7 +8,7 @@ Build-Depends: debhelper-compat (= 13),
                  libacl1-dev,
                  libpopt-dev,
                  liblz4-dev,
  -               libxxhash-dev,
  +               libxxhash-dev (>= 0.8),
                  libzstd-dev,
                  zlib1g-dev,
                  libssl-dev

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/xxhash/+bug/1934992/+subscriptions