Volunteers needed -- Automated Image Testing for flavors

[Nicholas Skaggs]

On 07/30/2015 04:14 PM, Nicholas Skaggs wrote:
> On 07/29/2015 04:01 PM, Nicholas Skaggs wrote:
>> Greetings everyone! I wanted to share some news about a renewed 
>> effort to restore automated installer testing of ubiquity using the 
>> daily images generated by cdimage. Up until last cycle, the images 
>> were being test automatically via a series of autopilot tests, 
>> written originally by the community (kudos to you Dan!). It was 
>> noticed the tests didn't run this cycle, and wxl accordingly filed an 
>> RT; https://rt.ubuntu.com/Ticket/Display.html?id=26570. In short, as 
>> part of some datacenter shuffling, we learned it's no longer possible 
>> for CI to run or maintain the tests. They recommended we host and run 
>> them ourselves as a community.
>>
>> With the directions from CI in hand, I initially asked DanChapman and 
>> dkessel to investigate setting up a jenkins to run these tests. But 
>> they need your help! The autopilot tests for ubiquity have a few bugs 
>> that need solving. You can see them here: 
>> https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bugs?field.tag=autopilot.
>>
>> To help solve these bugs or to learn more, check out this document: 
>> http://bazaar.launchpad.net/~ubuntu-installer/ubiquity/trunk/view/head:/autopilot/README.md. 
>> It will guide you through running the tests yourself locally. You 
>> should be able to replicate the tests following those instructions. 
>> From there, patches and comments on the bugs would be most welcome 
>> (as would additional bug reports should you find them).
>>
>> Even with the tests working, however, we still need to setup a 
>> jenkins to run them. We'll also need to maintain this server. Anyone 
>> with experience or desire in this area? Ideas for reporting results 
>> (on the isotracker for instance) also need to be explored. This is 
>> likely to involve some python, and potentially some web work. Is 
>> anyone interested?
>>
>> If you have some technical skills and want to help out, please do get 
>> in touch with myself, DanChapman or dkessel. The goal behind this 
>> effort is to see these tests be useful again this cycle for image 
>> testing and lowering the burden for manual testers.
>>
>> Thanks!
>>
>> Nicholas
> Thanks to everyone who expressed an interest! I've created a document 
> that will help answer your questions about what we are trying to do, 
> and perhaps generate some more. http://bit.ly/1IO103i 
> <http://t.co/Rj0w6yQefI>. Please leave your feedback and ideas inside 
> the document.
>
> In addition, for all those who are able and want to help out, we'll be 
> having a meeting tomorrow, July 31st, at 1900 UTC in #ubuntu-quality 
> on freenode. We'll go over the document, and finalize the plan of 
> work. We'll also start to divvy up the needed tasks and pick a good 
> day and time for future meetings as needed. If you miss the first one, 
> don't worry, stay involved.
>
> You can join the channel easily via webchat; 
> http://webchat.freenode.net/?channels=ubuntu-quality
>
> I hope to see you there!
>
> Nicholas


A big thank you to everyone who came out today to talk through the next 
steps. Our next meeting will be 1900 UTC on Thursday August 6th in 
#ubuntu-quality on freenode. Please join us if you are able!

We took some actions for the next meeting.

-svij and shrini agreed to setup a test jenkins instance to help answer 
our lingering questions on what we need. Specifically they'll be looking at
where should we host this?
can we test in the cloud?
what type of setup should we have (how many slaves, how many instances)?
and trying to get us all setup with a jenkins instance we can add jobs 
to and iterate on moving forward.

-flocculant took an action to investigate when the images came available 
and gave feedback on avoiding failing images and those being built only 
for LTS releases as part of the initial release.

-DanChapman agreed to create a readme for debugging autopilot and 
ubiquity in order to help fix and maintain the current autopilot test 
suite.

-shrini also volunteered to help out by looking into the failing tests


Finally, we discussed reporting and determined simple would be best at 
first, coming up with a couple easy ideas to choose between later.

Again, thanks to everyone who volunteered! That said, the are still some 
outstanding needs you can help us with!

We need folks willing to help fix and maintain the ubiqiuty autopilot 
tests. There is some learning curve, but you won't be alone. Dan has 
offered to help guide anyone who wants to undertake this work. The first 
step would be fixing the known bugs within the tests so they can run 
successfully.

We also needsomeone who knows python to have a look at the isotracker 
API, and create a script to update the notice board with the latest 
results.This was the favored idea by those in attendance at the meeting 
for reporting the build results to everyone in the community. The API is 
here: http://iso.qa.ubuntu.com/qatracker/api. Note, you need to login to 
see it!


Nicholas