[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial
Launchpad Bug Tracker
1578080 at bugs.launchpad.net
Mon May 23 06:58:42 UTC 2016
This bug was fixed in the package systemd - 229-4ubuntu6
systemd (229-4ubuntu6) xenial-proposed; urgency=medium
* Add a DHCP exit hook for pushing received NTP servers into timesyncd.
* Revert "enable TasksMax= for all services by default, and set it to 512".
Introducing a default limit on number of threads broke a lot of software
which regularly needs more, such as MySQL and RabbitMQ, or services that
spawn off an indefinite number of subtasks that are not in a scope, like
LXC or cron. 512 is way too much for most "simple" services, and it's way
too little for the ones mentioned above. Effective (and much stricter)
limits should instead be put into units individually.
(Closes: #823530, LP: #1578080)
* debian/gbp.conf: Switch to ubuntu-xenial branch.
-- Martin Pitt <martin.pitt at ubuntu.com> Thu, 12 May 2016 10:39:30
** Changed in: systemd (Ubuntu Xenial)
Status: Fix Committed => Fix Released
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
percona cluster hits resource limits in HA Openstack cloud with xenial
Status in percona-xtradb-cluster-5.6 package in Ubuntu:
Status in systemd package in Ubuntu:
Status in systemd source package in Xenial:
Status in percona-cluster package in Juju Charms Collection:
Status in systemd package in Debian:
I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial
using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack
being HA as possible.
When deployed, after running for a while mysql (which is a 3 node
cluster) starts refusing connections, and erroring:
2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50
'Can't create thread to handle new connection' error(s) suppressed
When I look at systemd-cgtop, I can see it maxing out at 512
To get it going again I do a:
$ sudo systemctl edit mysql
Sometimes I even need to edit /etc/systemd/system.conf and bump
DefaultTasksMax to 1024 or higher, depending on long its been left
I've noticed that dropping worker-multiplier setting on nova-cloud-
controller, neutron-api etc all help to reduce the load, but I still
need to bump it up.
Please let me know if you need any more information.
Impact: Introducing a default #thread limit of 512 broke an unknown set of services which regularly run many threads.
Fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 (essentially, revert the upstream commit that enabled it)
Regression potential: Very low -- this just restores the pre-228 behaviour and does not impose any new restriction.
- Pick some unit like cron.service or mysql.service that does not specify an explicit TaskMax= limit.
- Check its TaskMax: systemctl show -p TasksMax cron.service
- In current xenial this is "512", after the update it should be a very big number (maxint minus 1, which means "infinity")
To manage notifications about this bug go to:
More information about the foundations-bugs