Automated Testing for Flavors -- Update

[Nicholas Skaggs]

On 03/09/2016 04:11 PM, flocculant at gmx.co.uk wrote:
> On 09/03/16 20:59, Nicholas Skaggs wrote:
>> On 03/09/2016 03:51 PM, flocculant at gmx.co.uk wrote:
>>> On 09/03/16 20:43, Nicholas Skaggs wrote:
>>>> Sure. Each of the test runs are a 1 to 1 copy of a manual test. So, 
>>>> in the same way you would add a result, we'll have a bot account 
>>>> add a result with a Pass or Fail to the daily image. It should also 
>>>> leave a comment linking to the run so you can learn more if you are 
>>>> curious. Simon has actually agreed to hack on this, so I hope we'll 
>>>> start to see some bot results (though they will be failures!) on 
>>>> the tracker soon. We could still use some help with fixing the 
>>>> actual tests however, so they can provide value!
>>>>
>>>>
>>>> Nicholas
>>>>
>>> I would really really not want to see anything on the tracker from a 
>>> bot - that's reporting a fail on the test not the reality :(
>>>
>>>
>>>> [snip]
>>>>
>>>
>>>
>> Hmm, well any other opinions? You are correct at this point in that 
>> it would be showing failures which are not true. However, we want the 
>> tests to run and show proper pass/fails! So it should be a temporary 
>> thing. That said, we could not post results until the tests are 
>> working, but it's certainly possible to have a failed test in the 
>> future that isn't a real failure.
>>
>> Nicholas
> The trouble with posting any fail (assuming they run properly) from 
> these tests to the tracker is there is absolutely no way of knowing 
> what failed - just gobbledygook.
>
> So a flavour QA team would have to run the image to see if the fail is 
> real or not, and 'where' it failed - at that point what have we gained?
>
> It won't be adding a bug will it - or I would assume not.
>
> Personally given the option to grab rss feeds from Jenkins - then 
> people interested in whether an image has failed could do that. That's 
> what I had intended way back when.
>
> Maybe a seperate area on the tracker for them?
>
I was trying to keep it simple, and integrating it was the simpliest way 
I could think of. A post to the notice board? Would that work better?

Nicholas