Thoughts on Dense Container testing

[John A Meinel]

So I played around with manually assigning IP addresses to a machine, and
using BTRFS to make the LXC instances cheap in terms of disk space.

I had success bringing up LXC instances that I created directly, I haven't
gotten to the point where I could use Juju for the intermediate steps. See
the attached document for the steps I used to set up several addressable
containers on an instance.

However, I feel pretty good that Container Addressability would actually be
pretty straightforward to achieve with the new Networker. We need to make
APIs for requesting an Address for a new container available, but then we
can configure all of the routing stuff without too much difficulty.

Also of note, is that because we are using MASQUERADE in order to route the
traffic, it doesn't require putting the bridge (br0) directly onto eth0. So
it depends if MaaS will play nicely with routing rules if you assign an IP
address into a container on a machine, will the routes end up routing the
traffic there (I think it will, but we'd have to test to confirm it).

Ideally, I'd rather do the same thing everywhere, rather that have
containers routed one way in MaaS and a different way on EC2.

It may be that in the field we need to not Masquerade, so I'm open to
feedback here.

I wrote this up a bit like how I would want to use dense containers for
scale testing, since you can then deploy actual workloads into each of
these LXCs if you wanted (and had the horsepower :).

I succeeded in putting 6 IPs on a single m3.medium and running 5 LXC
containers and was able to connect to them from another machine running
inside the VPC.

John
=:->
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20140827/ee467cc8/attachment.html>
-------------- next part --------------
Steps for setting up high-density LXC machine

1) launch machine in a VPC (either bootstrap with defaultVPC or launch manually in VPC)
2) add additional IP addresses to the NIC
   in Dashboard this is EC2 dashboard/Network & Security/Network Interfaces, select interface, Manage Private IP Addresses, Assign new IP
   need to work out how to script this
3) on machine install lxc, btrfs and allow ip forwarding:
   $ sudo su -
	# apt-get update
	# apt-get install lxc btrfs-tools
	# sysctl -w net.ipv4.ip_forward=1
4) Create a BTRFS block device, and mount it into /var/lib/lxc
	# dd if=/dev/zero of=/var/lib/lxc-block bs=1M count=1024
	# losetup /dev/loop0 /var/lib/lxc-block
	# mkfs -t btrfs /dev/loop0
	# mount /dev/loop0 /var/lib/lxc
5) Create the first LXC container, making sure it is set up for BTRFS backing
	# lxc-create -B btrfs -n test-lxc-1 -t ubuntu-cloud
6) Before booting for the first time, configure the eth0.cfg with one of the static IP addresess from earlier. Read /etc/resolve.conf to find the address of your dns server.
	# vim /var/lib/lxc/test-lxc-1/rootfs/etc/networking/eth0.cfg
		auto eth0
		iface eth0 inet static
			address CONTAINER_IP
			netmask 255.255.255.255
			post-up ip route add HOST_IP dev eth0
			post-up ip route add default via HOST_IP
			dns-nameservers DNS_IP
7) Setup the host to masquerade traffic, and to have a route for all static addresses
	# ip route add CONTAINER_IP dev lxcbr0
	# iptables -t nat -I POSTROUTING -o eth0 -j MASQUERADE
8) Start the instance one time which will make sure the Ubuntu user is set up
	# lxc-start -n test-lxc-1
	  Login ubuntu/ubuntu
	  $ sudo shutdown -h now
9) Update .ssh with a authorized-keys so that we can use it. Perhaps step (8) can
be better done with a parameter to lxc-create?. Alternatively, maybe we can use
a mkdir -p, but that won't create the ubuntu user with the right skeleton
files.
	cp -r ~ubuntu/.ssh/ /var/lib/lxc/test-lxc-1/rootfs/home/ubuntu
10) For each new LXC:
	a) Clone the test LXC for a new LXC
		# lxc-clone -s -B btrfs test-lxc-1 test-lxc-2
	b) Update the CONTAINER_IP in eth0.cfg and set up a route for the new container
		# vim /var/lib/lxc/test-lxc-2/rootfs/etc/networking/eth0.cfg
			s/OLD_CONTAINER_IP/NEW_CONTAINER_IP/
		# ip route add NEW_CONTAINER_IP dev lxcbr0
	c) Start the container
		# lxc-start -n test-lxc-2 -d

	d) At this point, you should be able to SSH into the NEW_CONTAINER_IP
	as the 'ubuntu' user, which should let you use this with manual registration.


If you wanted to use 'juju bootstrap' and 'juju deploy --to lxc:' with this
setup, I believe you could if you had 'default-vpc'. The key is that you would
have to setup the BTRFS loopback mount before deploying anything in LXC, and
you would have to allocate and configure the IP addresses manually. (I believe
Juju is already aware that if /var/lib/lxc is BTRFS it will create the
juju-SERIES-template container in such a way that it can be trivially cloned.)