Testing report - Snappy Ubuntu Core 15.04 stable #3

[Leo Arias]

Hello!

I'm working with Federico Giménez in the quality assurance for snappy. For
every new stable release, we will be sending a testing report like this one.

This is our first release as part of the team; so the approach we took was
to spend as many days as we had available to run a small suite of manual
tests and an extensive session of exploratory testing on different
environments.

Summary
=======

We ran tests in BeagleBone Black Rev C, RaspberryPi 2 Model B, KVM,
Microsoft Azure and Google Compute Engine.
You can see our full testing journal here:
http://pad.ubuntu.com/testing-snappy

Results
======

We found no blockers for the release, only some bugs that can be resolved
in the following versions, mostly related to the documentation and guides.

- The snappy getting started guide doesn't mention the serial console
https://bugs.launchpad.net/developer-ubuntu-com/+bug/1457935
- Running set active without sudo gives lower level error than it should
https://bugs.launchpad.net/snappy/+bug/1461070
- snappy list error if executed during a snappy-autopilot update
https://bugs.launchpad.net/snappy/+bug/1461487
- raspberry pi downloads the update, asks for a reboot and then does
nothing.
https://trello.com/c/PJE5oYAF/84-disabling-updates-and-auto-update-when-using-sideloaded-kernel-snaps
- azure vm create command fails because the image doesn't exist
https://bugs.launchpad.net/developer-ubuntu-com/+bug/1463600
- azure vm create command fails because the specified DNS name is already
taken https://bugs.launchpad.net/developer-ubuntu-com/+bug/1463602
- gcloud config set project requires the id of the project, not the name
https://bugs.launchpad.net/developer-ubuntu-com/+bug/1463615
- the snappy guide for google compute engine doesn't mention that the api
needs to be enabled
https://bugs.launchpad.net/developer-ubuntu-com/+bug/1463620

To the developers: please help us triaging these bugs so we can start
resolving them in order of importance. I'm now an editor on the developer
site, so I can take care of most of the docs issues.

Future
=====

This was mostly a learning exercise for us. We collected many questions and
rough edges that make testing a little hard.
We have started planning the testing for the following release, that will
be more interesting, with more automation, more people and opportunities
for developers and community members. We'll send updates about our plan
during the month.

And in our snappified world, testing is a full team effort that happens
daily. So if you want to give us a hand and learn in the process, join
#snappy on freenode and ping elopio and fgimenez.

pura vida
-- 
¡paz y baile!
http://www.ubuntu.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/snappy-devel/attachments/20150610/ac6cd993/attachment-0001.html>