[Bug 1385134] [NEW] CRITICAL glance [-] error: [Errno 24] Too many open files

[James Page]

Public bug reported:

Its quite easy to hit the nofiles limit with high levels of concurrency
and a ceph backend:

2014-10-24 08:29:05.356 22327 CRITICAL glance [-] error: [Errno 24] Too many open files
2014-10-24 08:29:05.356 22327 TRACE glance Traceback (most recent call last):
2014-10-24 08:29:05.356 22327 TRACE glance   File "/usr/bin/glance-api", line 10, in <module>
2014-10-24 08:29:05.356 22327 TRACE glance     sys.exit(main())
2014-10-24 08:29:05.356 22327 TRACE glance   File "/usr/lib/python2.7/dist-packages/glance/cmd/api.py", line 83, in main
2014-10-24 08:29:05.356 22327 TRACE glance     server.start(config.load_paste_app('glance-api'), default_port=9292)
2014-10-24 08:29:05.356 22327 TRACE glance   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 271, in start
2014-10-24 08:29:05.356 22327 TRACE glance     self.run_child()
2014-10-24 08:29:05.356 22327 TRACE glance   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 321, in run_child
2014-10-24 08:29:05.356 22327 TRACE glance     self.run_server()
2014-10-24 08:29:05.356 22327 TRACE glance   File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 343, in run_server
2014-10-24 08:29:05.356 22327 TRACE glance     debug=False)
2014-10-24 08:29:05.356 22327 TRACE glance   File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 671, in server
2014-10-24 08:29:05.356 22327 TRACE glance     client_socket = sock.accept()
2014-10-24 08:29:05.356 22327 TRACE glance   File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 179, in accept
2014-10-24 08:29:05.356 22327 TRACE glance     res = socket_accept(fd)
2014-10-24 08:29:05.356 22327 TRACE glance   File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 56, in socket_accept
2014-10-24 08:29:05.356 22327 TRACE glance     return descriptor.accept()
2014-10-24 08:29:05.356 22327 TRACE glance   File "/usr/lib/python2.7/socket.py", line 202, in accept
2014-10-24 08:29:05.356 22327 TRACE glance     sock, addr = self._sock.accept()
2014-10-24 08:29:05.356 22327 TRACE glance error: [Errno 24] Too many open files
2014-10-24 08:29:05.356 22327 TRACE glance
2014-10-24 08:29:05.840 22316 ERROR glance.wsgi.server [-] Not respawning child 22327, cannot recover from termination

Especially as the api workers run as thread, not separate processes.

Increasing the default nofiles via upstart would be a good solution.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: glance-api 1:2014.2-0ubuntu1~cloud0 [origin: Canonical]
ProcVersionSignature: User Name 3.13.0-35.62-generic 3.13.11.6
Uname: Linux 3.13.0-35-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.5
Architecture: amd64
CrashDB:
 {
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
             }
Date: Fri Oct 24 09:08:50 2014
PackageArchitecture: all
SourcePackage: glance
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.glance.glance.api.conf: [deleted]
modified.conffile..etc.glance.glance.api.paste.ini: [deleted]
modified.conffile..etc.glance.glance.cache.conf: [deleted]
modified.conffile..etc.glance.glance.scrubber.conf: [deleted]
modified.conffile..etc.glance.policy.json: [deleted]
modified.conffile..etc.glance.schema.image.json: [deleted]

** Affects: cloud-archive
     Importance: Undecided
         Status: New

** Affects: glance (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: amd64 apport-bug scale-testing third-party-packages trusty uec-images

** Tags added: scale-testing

** Also affects: glance (Ubuntu)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to glance in Ubuntu.
https://bugs.launchpad.net/bugs/1385134

Title:
  CRITICAL glance [-] error: [Errno 24] Too many open files

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1385134/+subscriptions