[Bug 1657256] Re: Percona crashes when doing a a 'larger' update

Thu Sep 21 16:05:17 UTC 2017

Deploying Openstack Ocata, w/ Juju on arm64 hosts, the mysql crashed
with the following:

2017-09-20 21:38:39 17783 [Warning] 'user' entry 'root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:38:39 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:38:39 17783 [Warning] 'user' entry 'sstuser at ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:38:39 17783 [Warning] 'proxies_priv' entry '@ root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:02 17783 [Warning] 'user' entry 'root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:02 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:02 17783 [Warning] 'user' entry 'sstuser at ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:39:02 17783 [Warning] 'proxies_priv' entry '@ root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:20 17783 [Warning] 'user' entry 'root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:20 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:20 17783 [Warning] 'user' entry 'sstuser at ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:39:20 17783 [Warning] 'proxies_priv' entry '@ root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:41 17783 [Warning] 'user' entry 'root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:41 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:41 17783 [Warning] 'user' entry 'sstuser at ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:39:41 17783 [Warning] 'proxies_priv' entry '@ root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:53 17783 [Warning] 'user' entry 'root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:53 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:53 17783 [Warning] 'user' entry 'sstuser at ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:40:53 17783 [Warning] 'proxies_priv' entry '@ root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:56 17783 [Warning] 'user' entry 'root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:56 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:56 17783 [Warning] 'user' entry 'sstuser at ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:40:56 17783 [Warning] 'proxies_priv' entry '@ root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:58 17783 [Warning] 'user' entry 'root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:58 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:58 17783 [Warning] 'user' entry 'sstuser at ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:40:58 17783 [Warning] 'proxies_priv' entry '@ root at juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 22:05:59 ffff583fa1e0 InnoDB: Assertion failure in thread 281472162308576 in file ut0lst.h line 271
InnoDB: Failing assertion: list.count > 0
InnoDB: We intentionally generate a memory trap.

At the time I was doing a'lot of things at once, (import images to
glance, configuring neutron and trying to launch an instance via nova)
However, I've redeployed at least 3 times today and have seen a crash
similar to this with each deployment. I've attached the error.log

ubuntu at juju-24a047-0-lxd-1:/var/log/mysql$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.3 LTS
Release:	16.04
Codename:	xenial

ubuntu at juju-24a047-0-lxd-1:/var/log/mysql$ uname -a
Linux juju-24a047-0-lxd-1 4.10.0-35-generic #39~16.04.1-Ubuntu SMP Wed Sep 13 09:01:34 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux

ii percona-galera-3 3.19-0ubuntu0.16.04.1 arm64 Galera replication framework for Percona XtraDB Cluster
ii percona-xtrabackup 2.3.7-0ubuntu0.16.04.2 arm64 Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-server-5.6 5.6.34-26.19-0ubuntu0.16.04.1 arm64 Percona XtraDB Cluster database server binaries

I was able to reproduce this on a standalone arm64 host , by installing percona-cluster and running the sysbench test.  I have attached those logs as well.

** Attachment added: "errorlog.txt"
   https://bugs.launchpad.net/charm-test-infra/+bug/1657256/+attachment/4954496/+files/errorlog.txt

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1657256

Title:
  Percona crashes when doing a a 'larger' update

Status in OpenStack Charm Test Infra:
  Confirmed
Status in percona-xtradb-cluster-5.5 package in Ubuntu:
  Confirmed
Status in percona-xtradb-cluster-5.6 package in Ubuntu:
  In Progress
Status in percona-xtradb-cluster-5.5 source package in Trusty:
  Confirmed
Status in percona-xtradb-cluster-5.6 source package in Xenial:
  Confirmed
Status in percona-xtradb-cluster-5.6 source package in Yakkety:
  Confirmed
Status in percona-xtradb-cluster-5.6 source package in Zesty:
  Confirmed
Status in percona-xtradb-cluster-5.5 source package in Artful:
  Confirmed
Status in percona-xtradb-cluster-5.6 source package in Artful:
  In Progress

Bug description:
  [Impact]

   * Percona will segfault when exposed to medium load, almost imediately
   * This is because of a bug in upstream, which manifests only on PPC

  [Test Case]

   * Install and configure percona-xtradb-cluster-server
    * Run sysbench against configured node (only one node is needed, no need for proper cluster):
     # sysbench --test=oltp --oltp-test-mode=complex --max-time=60 --num-threads=110 run
   * mysqld will segfault seconds withing starting the test

  [Regression Potential]

   * This is a cherry-pick from an upstream fix (https://jira.mariadb.org/browse/MDEV-6450)
   * This is not fixed in upstream Percona becasue Percona does not officially support non-intel archs.
   * Because code adds additional memory barriers there was a chance of performance degradation on i386/amd64. However, intensive sysbench syntetic loads proved this is not the case - there are no performance penalties.

  [Other Info]

   *  percona-xtradb-cluster-5.5 is only available for Trusty.
  $ rmadison percona-xtradb-cluster-5.5
   percona-xtradb-cluster-5.5 | 5.5.34-25.9+dfsg-0ubuntu4          | trusty/universe          | source
   percona-xtradb-cluster-5.5 | 5.5.37-25.10+dfsg-0ubuntu0.14.04.1 | trusty-security/universe | source
   percona-xtradb-cluster-5.5 | 5.5.37-25.10+dfsg-0ubuntu0.14.04.2 | trusty-updates/universe  | source

  * See comment #22 for more context about other releases that offers percona-xtradb-cluster-5.6 : 
  https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.5/+bug/1657256/comments/22

   * Upstream commit:
  https://github.com/MariaDB/server/commit/40497577ffd9f85557b15e08ad913f627b2e9530

  [Original Description]

  I'm trying to set up percona-xtradb-cluster-5.5 on PPC machine. While
  the package installs fine, as soon as I run sysbench oltp becnhmark
  against it, Percona dies (even when I start the benchmark with just
  one connection).

  I can also crash mysql manually, by updating the sbtest table (which
  is created by the sysbench utility):

  mysql> update sbtest set pad = 'mario1' limit 1000000;
  ERROR 2013 (HY000): Lost connection to MySQL server during query

  Sometimes I need to repeat this update (with different values for
  'pad' field) few times. This happens regardless of whether I run the
  UPDATE inside the transaction or not.

  This is the assertion found in the log file:

  170117 21:10:55 InnoDB: Assertion failure in thread 70366668321152 in file buf0buf.ic line 1277
  InnoDB: Failing assertion: block->page.buf_fix_count > 0

  This is a single-node percona-xtradb-cluster server, without
  wsrep_provied configured, run inside 14.04 lxc container on 16.04
  host.

  I'm attaching the full log file, mysql configuration file and the core
  dumped.

  The version of the package installed is this 5.5.37-25.10+dfsg-
  0ubuntu0.14.04.2.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-test-infra/+bug/1657256/+subscriptions