[Bug 1078242] [NEW] juju cannot connect to zookeeper - too many connections (max 10)

[Zygmunt Krynicki]

Public bug reported:

My juju cluster running on precise had severe connection issues to
zookeeper. Looking at zookeeper logs I can see that my juju nodes cannot
connect as they already have pending connections.

Zookeeper log is full of lines like (quick count gives me around 30,000)

2012-11-08 06:46:48,777 - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory at 247] -
Too many connections from /10.55.60.51 - max is 10

There are also session expiration messages (around 1000)

2012-11-08 06:46:58,000 - INFO  [SessionTracker:ZooKeeperServer at 316] -
Expiring session 0x13ad043e9460400, timeout of 10000ms exceeded

And exceptions (about 4000)

2012-11-08 06:47:05,663 - ERROR [SyncThread:0:NIOServerCnxn at 445] - Unexpected Exception: 
java.nio.channels.CancelledKeyException
        at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
        at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
        at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
        at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
        at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:162)
        at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:135)

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: juju 0.5+bzr531-0ubuntu1.3
ProcVersionSignature: User Name 3.2.0-32.51-virtual 3.2.30
Uname: Linux 3.2.0-32-virtual x86_64
ApportVersion: 2.0.1-0ubuntu14
Architecture: amd64
Date: Tue Nov 13 11:11:30 2012
Ec2AMI: ami-000000bf
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: juju
UpgradeStatus: No upgrade log present (probably fresh install)

** Affects: juju (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: amd64 apport-bug ec2-images precise

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju in Ubuntu.
https://bugs.launchpad.net/bugs/1078242

Title:
  juju cannot connect to zookeeper - too many connections (max 10)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/juju/+bug/1078242/+subscriptions