Automated Testing for Flavors -- Update

[flocculant at gmx.co.uk]

On 09/03/16 21:27, Nicholas Skaggs wrote:
> On 03/09/2016 04:11 PM, flocculant at gmx.co.uk wrote:
>> On 09/03/16 20:59, Nicholas Skaggs wrote:
>>> On 03/09/2016 03:51 PM, flocculant at gmx.co.uk wrote:
>>>> On 09/03/16 20:43, Nicholas Skaggs wrote:
>>>>> Sure. Each of the test runs are a 1 to 1 copy of a manual test. 
>>>>> So, in the same way you would add a result, we'll have a bot 
>>>>> account add a result with a Pass or Fail to the daily image. It 
>>>>> should also leave a comment linking to the run so you can learn 
>>>>> more if you are curious. Simon has actually agreed to hack on 
>>>>> this, so I hope we'll start to see some bot results (though they 
>>>>> will be failures!) on the tracker soon. We could still use some 
>>>>> help with fixing the actual tests however, so they can provide value!
>>>>>
>>>>>
>>>>> Nicholas
>>>>>
>>>> I would really really not want to see anything on the tracker from 
>>>> a bot - that's reporting a fail on the test not the reality :(
>>>>
>>>>
>>>>> [snip]
>>>>>
>>>>
>>>>
>>> Hmm, well any other opinions? You are correct at this point in that 
>>> it would be showing failures which are not true. However, we want 
>>> the tests to run and show proper pass/fails! So it should be a 
>>> temporary thing. That said, we could not post results until the 
>>> tests are working, but it's certainly possible to have a failed test 
>>> in the future that isn't a real failure.
>>>
>>> Nicholas
>> The trouble with posting any fail (assuming they run properly) from 
>> these tests to the tracker is there is absolutely no way of knowing 
>> what failed - just gobbledygook.
>>
>> So a flavour QA team would have to run the image to see if the fail 
>> is real or not, and 'where' it failed - at that point what have we 
>> gained?
>>
>> It won't be adding a bug will it - or I would assume not.
>>
>> Personally given the option to grab rss feeds from Jenkins - then 
>> people interested in whether an image has failed could do that. 
>> That's what I had intended way back when.
>>
>> Maybe a seperate area on the tracker for them?
>>
> I was trying to keep it simple, and integrating it was the simpliest 
> way I could think of. A post to the notice board? Would that work better?
>
> Nicholas
That might work - perhaps just put it there where a test fails?

So say Xubuntu failed - Notice board would say that - but say nothing 
about other flavours - the expectation being we always get passes :)