[Bug 1578080] Re: percona cluster hits resource limits in HA Openstack cloud with xenial

[Martin Pitt]

This was discussed upstream last November:
https://lists.freedesktop.org/archives/systemd-
devel/2015-November/035006.html

And then enabled by default in 228 in
https://github.com/systemd/systemd/commit/9ded9cd14.

So in retrospect, having a default limit there was not such a good idea
after all: 512 is way too much for most "simple" services, and it's way
too little for others such as the ones mentioned above. There is also no
particular rationale about "512", so even if we'd bump it to 1024 we'd
just make the limit even less useful while still breaking software.

So I think we should disable the default limit at least for Xenial in an
SRU, but probably also in devel. It is both much safer and also much
more effective in terms of guarding against berserk
programs/bugs/unintended fork bombs etc. to set limits in units
individually. Once someone looks at one, this is then a great time to
also flip on the other resource and privilege limitations that systemd
offers, such as CapabilityBoundingSet=, SecureBits=, PrivateDevices=,
PrivateNetwork=, ProtectSystem=, ProtectHome=, etc.


** Changed in: percona-xtradb-cluster-5.6 (Ubuntu)
       Status: New => Won't Fix

** Changed in: systemd (Ubuntu)
   Importance: Undecided => High

** Changed in: systemd (Ubuntu)
       Status: New => In Progress

** Also affects: systemd (Ubuntu Xenial)
   Importance: Undecided
       Status: New

** Also affects: percona-xtradb-cluster-5.6 (Ubuntu Xenial)
   Importance: Undecided
       Status: New

** Bug watch added: Debian Bug tracker #823530
   http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=823530

** Also affects: systemd (Debian) via
   http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=823530
   Importance: Unknown
       Status: Unknown

** No longer affects: percona-xtradb-cluster-5.6 (Ubuntu Xenial)

** Changed in: systemd (Ubuntu Xenial)
       Status: New => In Progress

** Changed in: systemd (Ubuntu Xenial)
   Importance: Undecided => High

** Changed in: systemd (Ubuntu Xenial)
     Assignee: (unassigned) => Martin Pitt (pitti)

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1578080

Title:
  percona cluster hits resource limits in HA Openstack cloud with xenial

Status in percona-xtradb-cluster-5.6 package in Ubuntu:
  Won't Fix
Status in systemd package in Ubuntu:
  In Progress
Status in systemd source package in Xenial:
  In Progress
Status in percona-cluster package in Juju Charms Collection:
  New
Status in systemd package in Debian:
  Unknown

Bug description:
  I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial
  using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack
  being HA as possible.

  When deployed, after running for a while mysql (which is a 3 node
  cluster) starts refusing connections, and erroring:

    2016-05-03 01:25:28 13795 [ERROR] Error log throttle:         50
  'Can't create thread to handle new connection' error(s) suppressed

  When I look at systemd-cgtop, I can see it maxing out at 512
  connections.

  To get it going again I do a:

    $ sudo systemctl edit mysql

  and set:

    TasksMax=infinity

  Sometimes I even need to edit /etc/systemd/system.conf and bump
  DefaultTasksMax to 1024 or higher, depending on long its been left
  running.

  I've noticed that dropping worker-multiplier setting on nova-cloud-
  controller, neutron-api etc all help to reduce the load, but I still
  need to bump it up.

  Please let me know if you need any more information.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1578080/+subscriptions