Automated Testing for Flavors -- Update

[Nicholas Skaggs]

It's been an up and down cycle with mostly failing Autopilot tests for 
Ubiquity automated testing. I wanted to re-iterate Max's status updates 
as to what's going on so people understand what will exist for help with 
testing the final images for Xenial.

What's been happening:
Max has been working on various automated tests for images all cycle. In 
addition to these test for ubiquity, Max is also working on automated 
smoke tests, upgrade tests, etc. The AP tests for ubiquity are fragile 
and can often break, especially as Ubiquity changes. That's been 
happening for some time sadly. The goal is still to have them running 
for Final Beta, but time is running out.

  * You can see the runs here: http://162.213.34.238:8080/. They are not 
nice or pretty, but they do exist!

What we need:

* Folks to come and hack on the tests and fix them! Not just fix them, 
but look after them and expand them to keep them running. Everything you 
need to know is in the source tree. Branch lp:ubiquity and look at the 
README in the autopilot folder. Ask questions if needed. The tests 
themselves aren't too difficult to understand, but gtk2 is painful to 
automate at times. Dan has done wonderful work in writing and stepping 
in to fix them on occasion, but he's also writing the wonderful dekko 
mail client. It's time for others to jump in, if they are interested!

* More tests! Some have suggested simple tests such as merely attempting 
to boot the image. These tests are trivial to write since the hardwork 
of getting the environment ready and running them is already in place. 
Using autopilot or not, if you have a simple test idea it can probably 
be added and run.

* A nicer way to grok the test output. Simon has recently begun reviving 
the API for the tracker, and it would be lovely to somehow display the 
results on the tracker. Writing a scipt to add a test result submission 
against the daily test would be simple to do!

* We've now hit issues with scalability of hardware, so I'm looking to 
get a dedicated machine for this. I know we turned down hardware in the 
past, but it is now becoming a problem. Obviously it's lower priority as 
the reality is these tests have proven harder to keep running than 
imagined. As such, folks willing to stick with them would be wonderful!

Nicholas