[Bug 1557669] [NEW] port binding issues with docker-1.10.2 on Ubuntu 16.04

Thu Mar 24 17:08:56 UTC 2016

You have been subscribed to a public bug:

port binding issues with docker-1.10.2 on Ubuntu 16.04

---Steps to Reproduce---
1.  Install Ubuntu 16.04 guest on PowerKVM 3.1.1 #5.1
2.  Install docker
        apt-get update
        apt-get install docker.io
3.  create base image
         debootstrap xenial xenial

root at u1604base:~# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
xenial-base         latest              d1d4fe4bb11e        About an hour ago   329.9 MB

4.  create a container with with various port binding ranges

case 1 port ranges  600-700  -----------> Worked

root at u1604base:~# docker run --rm -it -p 600-700:600-700 xenial-base sh
# 

case 2 port ranges 600-710 -------------> Not working

root at u1604base:~# docker run --rm -it -p 600-710:600-710 xenial-base sh
docker: Error response from daemon: failed to create endpoint nauseous_ride on network bridge: iptables failed: iptables --wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 602 -j ACCEPT:  (fork/exec /sbin/iptables: resource temporarily unavailable).

The reported port binding issue has been initially found from Watson
research team while creating more number of containers.  We later
investigated and seen its an issue with port binding.

I'm still trying to investigate more on this by starting and stopping
daemon and checking, does that make any difference.

== Comment: #6 - Gowrishankar Muthukrishnan <gowrishankar.m at in.ibm.com>
- 2016-03-15 08:21:56 ==

Ok. I spent sometime playing with docker daemon today and at last found real
cause for this. You could solve this by running:

   echo `pidof docker` > /sys/fs/cgroup/pids/cgroup.procs

Details as below. First, the error as seen in docker run would mislead unless 
the details are not read in journalctl. I turned on debug mode for daemon and 
checked verbose logs through journalctl -p docker.service. In otherway, you can 
stop systemctl service and run directly docker daemon -D, which would also print 
detail info on console.

In the verbose logs, interesting observation is that, I could see iptables
insert option (-I) being called until to some port mapping successfully done 
(beginning from higher port number and decrement by one after updating nat and
filter tables). For ex, port map ranging between 45700 and 45600:

time="2016-03-15T07:22:39.929406000-04:00" level=debug msg="/sbin/iptables, [--wait -t nat -A DOCKER -p tcp -d 0/0 --dport 45659 -j DNAT --to-destination 172.17.42.2:45659 ! -i docker0]" 
time="2016-03-15T07:22:39.932421000-04:00" level=debug msg="/sbin/iptables, [--wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.42.2 --dport 45659 -j ACCEPT]" 
time="2016-03-15T07:22:39.934363000-04:00" level=debug msg="/sbin/iptables, [--wait -t nat -A POSTROUTING -p tcp -s 172.17.42.2 -d 172.17.42.2 --dport 45659 -j MASQUERADE]" 

# Confirms that, port mapping done until 45659 (from 45700).

time="2016-03-15T07:22:39.943160000-04:00" level=debug msg="/sbin/iptables, [--wait -t nat -D DOCKER -p tcp -d 0/0 --dport 45659 -j DNAT --to-destination 172.17.42.2:45659 ! -i docker0]" 
time="2016-03-15T07:22:39.943760000-04:00" level=warning msg="Failed to allocate and map port 45659-45659: Error starting userland proxy: " 

# Interestingly, userland proxies could not come up meanwhile. It could be 
due to capping on forked processes (as iptables are run as commands along with
other docker threads). Also, below sample of error is actually thrown on 
docker run command finally.

time="2016-03-15T07:22:39.944949000-04:00" level=debug msg="/sbin/iptables, [--wait -t nat -D DOCKER -p tcp -d 0/0 --dport 45700 -j DNAT --to-destination 172.17.42.2:45700 ! -i docker0]" 
time="2016-03-15T07:22:39.945419000-04:00" level=error msg="Error on iptables delete: iptables failed: iptables --wait -t nat -D DOCKER -p tcp -d 0/0 --dport 45700 -j DNAT --to-destination 172.17.42.2:45700 ! -i docker0:  (fork/exec /sbin/iptables: resource temporarily unavailable)" 
...
..

Then I compared 16.04 and 15.10 ubuntu kernels and found that latter did not 
have this problem and also found cgroup PID controller enabled in 16.04 kernel
(CONFIG_CGROUP_PIDS). You can refer its doc for how we can exploit it for
containers.

    https://www.kernel.org/doc/Documentation/cgroup-v1/pids.txt

Interestingly, docker daemon PID is not added in its parent group. Adding it
solves this problem.

root at ubuntu1604:/home/test# docker run --rm -it -p 600-700:600-700 ppc64le/ubuntu /bin/bash
root at 2359fabc9d9d:/# exit
exit

Over to build/docker team to fix bringing up docker daemon correctly.

== Comment: #8 - Kalpana Shetty <kalshett at in.ibm.com> - 2016-03-15 08:41:33 ==
Thanks  Gowri;  this helps.

root at u1604base:~# docker run --rm -it -p 200-500:200-500 xenial-base sh
# 

root at u1604base:~# docker run --rm -it -p 200-1000:200-1000 xenial-base
sh

# #

It works... :)

== Comment: #9 - Mel Bakhshi <melb at ca.ibm.com> - 2016-03-15 11:30:32 ==
 I also tested this on Ubuntu16.04 with docker 1.10.2 on ppc64le :

"docker daemon PID is not added in its parent group. Adding it
solves this problem."

 When should we expect this fix to be GA?

== Comment: #10 - Kalpana Shetty <kalshett at in.ibm.com> - 2016-03-15 11:52:02 ==
JFYI....
I have tried creating 1K containers with port option, without any issues. This works fine after I followed suggested work around of docker pid (see comment #6).

docker run cmd: docker run --rm -it -p 100-200:100-200 xenial-base ls

** Affects: docker.io (Ubuntu)
     Importance: Undecided
     Assignee: Taco Screen team (taco-screen-team)
         Status: Triaged

** Tags: architecture-ppc64le bot-comment bugnameltc-138945 severity-high targetmilestone-inin1604
-- 
port binding issues with docker-1.10.2 on Ubuntu 16.04
https://bugs.launchpad.net/bugs/1557669
You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to docker.io in Ubuntu.