Automated Testing for Flavors -- Update

[Nicholas Skaggs]

Sure. Each of the test runs are a 1 to 1 copy of a manual test. So, in 
the same way you would add a result, we'll have a bot account add a 
result with a Pass or Fail to the daily image. It should also leave a 
comment linking to the run so you can learn more if you are curious. 
Simon has actually agreed to hack on this, so I hope we'll start to see 
some bot results (though they will be failures!) on the tracker soon. We 
could still use some help with fixing the actual tests however, so they 
can provide value!


Nicholas

On 03/04/2016 11:31 PM, Istimsak Abdulbasir wrote:
>
> Could you explain more how the tracker will be able to store test 
> results for each daily test?
>
> On Mar 4, 2016 3:44 PM, "Nicholas Skaggs" 
> <nicholas.skaggs at canonical.com <mailto:nicholas.skaggs at canonical.com>> 
> wrote:
>
>     It's been an up and down cycle with mostly failing Autopilot tests
>     for Ubiquity automated testing. I wanted to re-iterate Max's
>     status updates as to what's going on so people understand what
>     will exist for help with testing the final images for Xenial.
>
>     What's been happening:
>     Max has been working on various automated tests for images all
>     cycle. In addition to these test for ubiquity, Max is also working
>     on automated smoke tests, upgrade tests, etc. The AP tests for
>     ubiquity are fragile and can often break, especially as Ubiquity
>     changes. That's been happening for some time sadly. The goal is
>     still to have them running for Final Beta, but time is running out.
>
>      * You can see the runs here: http://162.213.34.238:8080/. They
>     are not nice or pretty, but they do exist!
>
>     What we need:
>
>     * Folks to come and hack on the tests and fix them! Not just fix
>     them, but look after them and expand them to keep them running.
>     Everything you need to know is in the source tree. Branch
>     lp:ubiquity and look at the README in the autopilot folder. Ask
>     questions if needed. The tests themselves aren't too difficult to
>     understand, but gtk2 is painful to automate at times. Dan has done
>     wonderful work in writing and stepping in to fix them on occasion,
>     but he's also writing the wonderful dekko mail client. It's time
>     for others to jump in, if they are interested!
>
>     * More tests! Some have suggested simple tests such as merely
>     attempting to boot the image. These tests are trivial to write
>     since the hardwork of getting the environment ready and running
>     them is already in place. Using autopilot or not, if you have a
>     simple test idea it can probably be added and run.
>
>     * A nicer way to grok the test output. Simon has recently begun
>     reviving the API for the tracker, and it would be lovely to
>     somehow display the results on the tracker. Writing a scipt to add
>     a test result submission against the daily test would be simple to do!
>
>     * We've now hit issues with scalability of hardware, so I'm
>     looking to get a dedicated machine for this. I know we turned down
>     hardware in the past, but it is now becoming a problem. Obviously
>     it's lower priority as the reality is these tests have proven
>     harder to keep running than imagined. As such, folks willing to
>     stick with them would be wonderful!
>
>     Nicholas
>
>     -- 
>     Ubuntu-quality mailing list
>     Ubuntu-quality at lists.ubuntu.com
>     <mailto:Ubuntu-quality at lists.ubuntu.com>
>     Modify settings or unsubscribe at:
>     https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality
>