Automated Testing for Flavors -- Update

[flocculant at gmx.co.uk]

On 09/03/16 20:59, Nicholas Skaggs wrote:
> On 03/09/2016 03:51 PM, flocculant at gmx.co.uk wrote:
>> On 09/03/16 20:43, Nicholas Skaggs wrote:
>>> Sure. Each of the test runs are a 1 to 1 copy of a manual test. So, 
>>> in the same way you would add a result, we'll have a bot account add 
>>> a result with a Pass or Fail to the daily image. It should also 
>>> leave a comment linking to the run so you can learn more if you are 
>>> curious. Simon has actually agreed to hack on this, so I hope we'll 
>>> start to see some bot results (though they will be failures!) on the 
>>> tracker soon. We could still use some help with fixing the actual 
>>> tests however, so they can provide value!
>>>
>>>
>>> Nicholas
>>>
>> I would really really not want to see anything on the tracker from a 
>> bot - that's reporting a fail on the test not the reality :(
>>
>>
>>> [snip]
>>>
>>
>>
> Hmm, well any other opinions? You are correct at this point in that it 
> would be showing failures which are not true. However, we want the 
> tests to run and show proper pass/fails! So it should be a temporary 
> thing. That said, we could not post results until the tests are 
> working, but it's certainly possible to have a failed test in the 
> future that isn't a real failure.
>
> Nicholas
The trouble with posting any fail (assuming they run properly) from 
these tests to the tracker is there is absolutely no way of knowing what 
failed - just gobbledygook.

So a flavour QA team would have to run the image to see if the fail is 
real or not, and 'where' it failed - at that point what have we gained?

It won't be adding a bug will it - or I would assume not.

Personally given the option to grab rss feeds from Jenkins - then people 
interested in whether an image has failed could do that. That's what I 
had intended way back when.

Maybe a seperate area on the tracker for them?