[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Adam Conrad]

Hello Brad, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be
available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu6
in a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
enable and use -proposed.  Your feedback will aid us getting this update
out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-needed to verification-done. If it does not fix the
bug for you, please add a comment stating that, and change the tag to
verification-failed.  In either case, details of your testing will help
us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: systemd (Ubuntu Xenial)
       Status: In Progress => Fix Committed

** Tags added: verification-needed

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

Status in percona-xtradb-cluster-5.6 package in Ubuntu:
  Won't Fix
Status in systemd package in Ubuntu:
  Fix Committed
Status in systemd source package in Xenial:
  Fix Committed
Status in percona-cluster package in Juju Charms Collection:
  Won't Fix
Status in systemd package in Debian:
  Confirmed

Bug description:
  I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial
  using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack
  being HA as possible.

  When deployed, after running for a while mysql (which is a 3 node
  cluster) starts refusing connections, and erroring:

    2016-05-03 01:25:28 13795 [ERROR] Error log throttle:         50
  'Can't create thread to handle new connection' error(s) suppressed

  When I look at systemd-cgtop, I can see it maxing out at 512
  connections.

  To get it going again I do a:

    $ sudo systemctl edit mysql

  and set:

    TasksMax=infinity

  Sometimes I even need to edit /etc/systemd/system.conf and bump
  DefaultTasksMax to 1024 or higher, depending on long its been left
  running.

  I've noticed that dropping worker-multiplier setting on nova-cloud-
  controller, neutron-api etc all help to reduce the load, but I still
  need to bump it up.

  Please let me know if you need any more information.

  SRU INFORMATION
  ---------------
  Impact: Introducing a default #thread limit of 512 broke an unknown set of services which regularly run many threads.
  Fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 (essentially, revert the upstream commit that enabled it)
  Regression potential: Very low -- this just restores the pre-228 behaviour and does not impose any new restriction.
  Test case:
   - Pick some unit like cron.service or mysql.service that does not specify an explicit TaskMax= limit.
   - Check its TaskMax: systemctl show -p TasksMax cron.service
   - In current xenial this is "512", after the update it should be a very big number (maxint minus 1, which means "infinity")

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions