Fwd: Ch-Ch-Changes for Core Apps QA

[Nicholas Skaggs]

Forwarding along for those that may interact with merging code on any of 
the community core apps projects.


-------- Forwarded Message --------
Subject: 	Ch-Ch-Changes for Core Apps QA
Date: 	Tue, 8 Mar 2016 12:24:19 -0500
From: 	Nicholas Skaggs <nicholas.skaggs at canonical.com>
To: 	Ubuntu CoreApps <ubuntu-touch-coreapps at lists.launchpad.net>



Our quest to test all the things 'Core Apps' has been tumultus to say
the least over the past year. We lost our jenkins and migrated to a new
one, all while experiencing growing pains. The platform as well has
continued to shift and innovate and our poor acceptance tests have
struggled to be run and keep up. This causes heartache for everyone, and
the sea of red test failures isn't helpful.

So I decided we needed to take a long-term approach to fixing this. I
drafted a plan with Alan and QA, along with some feedback from a couple
of the core app developers. The idea is to right-size our testing
strategy[1].

While we've advocated for using the testing pyramid to test
applications, we've written and depended on mostly autopilot tests. This
is partially because of our early adoption of autopilot, and because I
find them so fun to write! I'm sure there is another person or two who
might say the same, but by and large the autopilot tests have proven to
be not easy to write or maintain. Hence there place on the top of the
pyramid[2]. Acceptance tests such as autopilot should represent the
minority of our test cases, not the majority. And as we've seen, despite
our efforts, the platform has broken our unchanged autopilot tests and
applications.

So, while I encourage you to take a quick review of the summary and
action plans from the document[1], let me summarize the changes here as
well. In short, we are going to:

1) Remove unmaintainable autopilot tests
2) Cleanup the core apps test suites for best practices, old code. And
also add documentation and readme's.
3) Add new qml tests to enhance and expand our unit testing
4) Use manual testing for those tests that would be hard to automate

Writing qml tests should prove to be much easier to do, and easier to
maintain. They test only our internal application code, and they are
written in a language you already know! The takeaway for you is look for
MP's landing that do all of the above, and start thinking about and
including / expanding upon your unit test suites. I guarantee you'll
find writing them to be more fun than autopilot!

Finally, please do provide feedback on the changes. The plan has been
tweaked a few times already, and I'm sure it will see a few more tweaks.
Thanks for a great set of core applications, and I hope these changes
make your development experiences better.

Nicholas

P.S. You'll also note the workflow for release has changed, for one, AP
tests don't run on merges now -- they will never block you. Check it
out: https://wiki.ubuntu.com/Touch/CoreApps/Jenkins#Workflow

1.
https://docs.google.com/document/d/1ApGwM9iI9qArbu4fMMdDs4Irb9fULfSVIjm79E-UBpQ/edit#heading=h.davy9ssj4q67
2. https://developer.ubuntu.com/en/start/quality/