[Bug 2148798] Re: gnocchi-metricd enters permanent crash loop on LZ4BlockError from truncated Carbonara object

[Timo Aaltonen]

Hello Seyeong, or anyone else affected,

Accepted gnocchi into resolute-proposed. The package will build now and
be available at
https://launchpad.net/ubuntu/+source/gnocchi/4.7.0+git2026012910.795f8f24-0ubuntu1.1
in a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
resolute to verification-done-resolute. If it does not fix the bug for
you, please add a comment stating that, and change the tag to
verification-failed-resolute. In either case, without details of your
testing we will not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: gnocchi (Ubuntu Resolute)
       Status: In Progress => Fix Committed

** Tags added: verification-needed-resolute

** Changed in: gnocchi (Ubuntu Noble)
       Status: In Progress => Fix Committed

** Tags added: verification-needed-noble

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2148798

Title:
  gnocchi-metricd enters permanent crash loop on LZ4BlockError from
  truncated Carbonara object

Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive caracal series:
  In Progress
Status in Ubuntu Cloud Archive epoxy series:
  In Progress
Status in Ubuntu Cloud Archive flamingo series:
  In Progress
Status in Ubuntu Cloud Archive gazpacho series:
  In Progress
Status in Ubuntu Cloud Archive yoga series:
  In Progress
Status in gnocchi package in Ubuntu:
  Fix Released
Status in gnocchi source package in Jammy:
  Fix Committed
Status in gnocchi source package in Noble:
  Fix Committed
Status in gnocchi source package in Questing:
  Won't Fix
Status in gnocchi source package in Resolute:
  Fix Committed

Bug description:
  [Impact]

  gnocchi-metricd workers enter a permanent crash loop when a Carbonara object in the Ceph RADOS storage pool is truncated (e.g. when metricd is OOM-killed or restarted mid-write). On the next cycle unserialize() calls lz4.block.decompress() on the short buffer.
  This raises lz4.block.LZ4BlockError, which is a direct subclass of Exception (not ValueError), so the existing "except ValueError" guard in carbonara.py does not catch it and the worker crashes.
  metricd retries the same object every cycle, so a single truncated object blocks the entire telemetry pipeline indefinitely.

  Also uploaded PR for github.
  https://github.com/gnocchixyz/gnocchi/issues/1348

  [Test case]

  On a Jammy,Noble,Questing container with python3-gnocchi and
  python3-lz4 installed:

      import lz4.block, os
      from gnocchi import carbonara
      data = b"c" + lz4.block.compress(os.urandom(16384))
      truncated = data[:20]
      try:
          carbonara.AggregatedTimeSerie.unserialize(
              truncated, key=None, aggregation=None)
      except lz4.block.LZ4BlockError as e:
          print("BUG:", e)
      except carbonara.InvalidData as e:
          print("FIXED:", e)

  Before patch

  prints "BUG: Decompression failed: ..." with a traceback into
  carbonara.py AggregatedTimeSerie.unserialize().

  After patch

  prints "FIXED: Unable to unpack, invalid data".

  Repeat the same test against carbonara.BoundTimeSerie.unserialize() (data passed directly, no marker byte).
  It must also transition from BUG to FIXED.

  [Where problems could occur]

  The fix wraps lz4.block.decompress() in both unserialize() methods with "except lz4.block.LZ4BlockError: raise InvalidData".
  InvalidData is already the sentinel used by the surrounding numpy.frombuffer ValueError branch, so no caller changes are needed.
  metricd already skips metrics that raise InvalidData.

  The only new behavior is that a non-truncation corruption (bitflip,
  disk error) is now silently skipped instead of crashing the worker,
  which is consistent with how the existing ValueError guard already
  treats downstream decode corruption.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2148798/+subscriptions