[Bug 994752] Re: lxc-start-ephemeral's use of dhcp lease table is fragile

[Gary Poster]

** Description changed:

  [Impact]
- <fill me in with explanation of severity and frequency of bug on users and justification for backporting the fix to the stable release>
+ This affects anyone using lxc-start-ephemeral as part of an automated process for which intermittent failures are a problem.  This includes the people who developed the initial version of the script, the Launchpad developers.  Our automated test suite will fail, stopping our landing tools, whenever this failure is triggered.
  
  [Development Fix]
- <fill me in with an explanation of how the bug has been addressed in the development branch, including the relevant version numbers of packages modified in order to implement the fix. >
+ 1. no longer look in the container's file system for a dhcp table to get the ip of the container; instead, look in the host's network information.  This is more reliable and ready sooner.  r101 of quantal lxc package has this change.
+ 2. increase the timeout waiting for the containers' network and sshd to be ready.
+ 
+ Note that the current increase, from 30 retries @ 1/sec to 60 retries @
+ 1/sec, is insufficient for the people who filed the bug, unfortunately.
+ Making the retry count configurable would be ideal.  Increasing it to
+ 300 would be sufficient, based on our experience so far.  stgraber
+ suggested using the `parallel -l maxload` construct to keep the starts
+ from being overwhelmed by load.  Unfortunately, we believe that this is
+ insufficient for at least two reasons.  First, the point of our effort
+ is to do a lot of work in parallel, with an lxc per core.  The work we
+ have to do takes more than half an hour.  Waiting for load to decrease
+ would miss the point of the effort.  Second, it doesn't seem that cpu
+ contention is always the problem, from watching top.
  
  [Stable Fix]
- <fill me in by pointing out a minimal patch applicable to the stable version of the package.>
+ [stgraber will need to specify]
  
  [Text Case]
- <fill me in with detailed *instructions* on how to reproduce the bug.  This will be used by people later on to verify the updated package fixes the problem.>
- 1.
- 2.
- 3.
- Broken Behavior: 
- Fixed Behavior: 
+ 
+ 1. Create an lxc container (which has sshd running and your home directory mounter, as is the default).  For the sake of these instructions, we will call it "lxctest".
+ 2. Run something like this.  Replace "username" with your user name.  You might need to do this more or fewer times; we've seen it most easily on a 32  core (16 core hyperthreaded) machine trying to run 32 concurrent callsTo make this less annoying, you could create a temporary passphraseless ssh key.
+ 
+ parallel -j 16 bash -c "lxc-start-ephemeral -u gary -o lpdev -- 'cat
+ /etc/hostname'" -- 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
+ 
+ Do this a few times.
+ 
+ Broken Behavior: At least one of the times, at least one of these fail (emitting an error message rather than the hostname) either because the code could not get the ip address in time, or because the container's sshd wasn't ready in time.
+ Fixed Behavior: You get all 16 hostnames.
  
  [Regression Potential]
- <fill me in with a discussion of likelihood and potential severity of regressions and how users could get inadvertently affected.>
+ The increased timeout might cause some code to wait longer than before to discover that something is wrong.  The improved ip code should have no negative effect.
  
  [Original Report]When lxc-start-ephemeral is given a command to run (--
  do_something) it wants to use lxc-attach to run the command, but lxc-
  attach is not ready yet.  Instead, it parses the dhcp leases to figure
  out the IP for the container, and then tries to use ssh to run the
  command.
  
  Twice today in tests involving lxc-start-ephemeral, the dhcp leases were
  unavailable and lxc-start-ephemeral failed.  The machine was under
  fairly heavy load and was virtualized (EC2).
  
  I'd like to try and make this less fragile.  As discussed on IRC, using
  lxcip
  (http://bazaar.launchpad.net/~launchpad/lpsetup/trunk/files/head:/lplxcip/)
  should make this more reliable.  Perhaps increasing the timeout in that
  code might be useful as well.

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to lxc in Ubuntu.
https://bugs.launchpad.net/bugs/994752

Title:
  lxc-start-ephemeral's use of dhcp lease table is fragile

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/994752/+subscriptions