[Bug 1644498] Re: apt-get update returns "AppStream cache update completed, but some metadata was ignored due to errors." periodically

[3DRaven]

I confirm this bug. Ubuntu 16.04.

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1644498

Title:
  apt-get update returns "AppStream cache update completed, but some
  metadata was ignored due to errors." periodically

Status in appstream package in Ubuntu:
  Fix Released
Status in appstream source package in Xenial:
  In Progress
Status in Debian:
  Invalid

Bug description:
  [Impact]

   * The recent update of the Ubuntu asgen instance yielded it to
  generate modern metadata. While we could tell asgen to generate old-
  style metadata, fixing appstream is the better idea, since we already
  applied similar patches to appstream-glib (and having this package
  hold back updates is a bad idea). Currently, the unknown metadata is
  considered to be broken by appstream and it displays a warning about
  that issue during an `apt update`

   * The attached patch includes a second change to make AppStream more
  robust against broken YAML data, ensuring LP #1579712 can not be
  triggered on purpose or due to broken disks creating damaged YAML
  structure.

  [Test Case 1]

   * apt update
   * Check for errors about metadata being ignored

  [Test Case 2]

   * appstreamcli refresh --force
   * Check for errors about metadata being ignored
   * Check if we ignored data that should be present
     (search for it via `appstreamcli search <term>`)
   * Run `appstreamcli status` and see if we still have information on the same amount of components as prior to the update.

  [Regression Potential]

   * Medium; The works case would be KDE Discover and GNOME Software
  loosing information on available software, but this risk is very low
  and the patch to fix this bug is very simple and has almost no
  regression potential. The YAML parser fix patch is a bit more complex,
  but should have no sideeffects.

  [Original report]

  Periodically I get emails from a machine with the subject:

  Cron <root at Jenkins> if [ -x /etc/munin/plugins/apt_all ]; then
  /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x
  /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12
  >/dev/null; fi

  And a body of:

  AppStream cache update completed, but some metadata was ignored due to
  errors.

  I have a machine monitored by Munin.  One of the plugins used on that
  machine is /etc/munin/plugins/apt which periodically counts the number
  of packets that can be upgraded.  This is achieved by a crontab entry
  installed from the package which looks like this:

  #
  # cron-jobs for munin-node
  #

  MAILTO=root

  # If the APT plugin is enabled, update packages databases approx. once
  # an hour (12 invokations an hour, 1 in 12 chance that the update will
  # happen), but ensure that there will never be more than two hour (7200
  # seconds) interval between updates..
  */5 * * * *     root if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi

  For many months everything has been running smoothly.  On the 23rd
  November 2016 I started getting the "AppStream cache update completed,
  but some metadata was ignored due to errors." messages at a rate of
  about ten a day.

  I did a quick Google search which turned up a handful of bugs from earlier this year, such as
  Bug1575248.  That is marked as Fixed Released though, so I am raising this new bug.

  If there are any further diagnostic details I can provide, please let
  me know.

  ProblemType: Bug
  DistroRelease: Ubuntu 16.04
  Package: appstream 0.9.4-1ubuntu1
  ProcVersionSignature: Ubuntu 4.4.0-47.68-generic 4.4.24
  Uname: Linux 4.4.0-47-generic x86_64
  ApportVersion: 2.20.1-0ubuntu2.1
  Architecture: amd64
  Date: Thu Nov 24 09:57:18 2016
  InstallationDate: Installed on 2013-05-08 (1295 days ago)
  InstallationMedia: Ubuntu 13.10 "Saucy Salamander" - Alpha amd64 (20130508)
  JournalErrors:
   -- Logs begin at Wed 2016-11-23 19:25:01 GMT, end at Thu 2016-11-24 09:57:24 GMT. --
   Nov 24 02:22:17 hostname systemd-tmpfiles[27330]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
  ProcEnviron:
   TERM=xterm-256color
   SHELL=/bin/bash
   PATH=(custom, no user)
   LANG=en_GB.UTF-8
   LANGUAGE=en_GB:en
  SourcePackage: appstream
  UpgradeStatus: Upgraded to xenial on 2016-08-24 (91 days ago)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/appstream/+bug/1644498/+subscriptions