Timeout added to kill-controller

[Tim Penhey]

Hi all,

NOTE: we do very much consider it a bug if the models don't die properly.

I have just landed a fix for a kill-controller issue where it would just 
sit there for a long time with nothing apparent happening.

Now kill-controller has a default timeout of 5 minutes. If there is no 
change in the timeout period, the command switches to a direct 
destruction mode where it will contact the cloud provider on behalf of 
each remaining model and destroy it that way.

The following examples all used LXD, and were a single controller 
machine with ubuntu deployed in the default model.

$ time juju kill-controller kill-test-fine -y
Destroying controller "kill-test-fine"
Waiting for resources to be reclaimed
Waiting on 1 model, 1 machine, 1 application
Waiting on 1 model, 1 machine, 1 application
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine
Waiting on 1 model
All hosted models reclaimed, cleaning up controller machines

real	0m27.443s


Nothing much changes here. Everything died nicely.
You can specify a timeout with --timeout (or -t). Valid formats are 
things like "2m" for two minutes, "30s" for thirty seconds.
Zero also works:

$ time juju kill-controller kill-test-no-delay -t 0 -y
Destroying controller "kill-test-no-delay"
Waiting for resources to be reclaimed
Killing admin at local/default directly
   done
All hosted models destroyed, cleaning up controller machines

real	0m2.492s


I had to throw a wrench in the works to get the provisioner to not kill 
the machine (wrench is a test facility we have). This allows me to show 
you a model that doesn't die like it should. I just specify a one minute 
timeout. The polling time was moved from two seconds to five seconds. 
Now you will see a countdown starting after 30 seconds of no change.

$ juju kill-controller kill-test -t 1m -y
Destroying controller "kill-test"
Waiting for resources to be reclaimed
Waiting on 1 model, 1 machine, 1 application
Waiting on 1 model, 1 machine, 1 application
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine
Waiting on 1 model, 1 machine, will kill machines directly in 29s
Waiting on 1 model, 1 machine, will kill machines directly in 24s
Waiting on 1 model, 1 machine, will kill machines directly in 19s
Waiting on 1 model, 1 machine, will kill machines directly in 14s
Waiting on 1 model, 1 machine, will kill machines directly in 9s
Waiting on 1 model, 1 machine, will kill machines directly in 4s
Killing admin at local/default directly
   done
All hosted models destroyed, cleaning up controller machines


Tim