Error deploying charm on juju-core with CI options

[Francesco Banconi]

On 06/14/2013 09:19 PM, Gary Poster wrote:
> On 06/14/2013 11:23 AM, Nicola 'teknico' Larosa wrote:
>> Is anyone able to deploy with juju-core and staging enable?
> 
> I hope not.  You should not be able to, because staging == improv and
> there is no improv for Go.

Indeed.

> The question is whether you can simply run bin/test-charm with Juju
> Core.  Francesco has it working, AIUI, presumably by not running the
> tests dependent on staging/improv.  

Currently it is possible to run the charm tests (juju-test) with
juju-core: the staging tests are skipped in that case, and similarly we
avoid to run the force-machine ones if the Python implementation is
used. On the GUI side, bin/test-charm works only in pyJuju, and AFAIK
the main reason (excluding Canonistack set up) is that GUI CI tests are
mostly based on staging=true (we only have one testcase switching from
staging to sandbox). For the future, having juju-core sandbox enabled, I
propose to create our initial set up in sandbox, letting specific test
cases switch to other modes (staging if available, real backend).

Work needs to be done for merging charm and GUI tests, and a possible
strategy follow (please reply with your suggestions/corrections):

1) We add to the GUI the ability to run charm tests, e.g.: the charm
trunk is checked out and then juju-test is used to run the charm tests.
The charm test suite already knows how to bootstrap an environment,
download tests dependencies, run unit and functional tests, collect the
results. This should strongly simplify the creation of a testing
environment (for the perspective of the GUI user/contributor), and
reduce the amount of Python requirements in the GUI.

2) Currently charm tests don't support Saucelabs (they just run the
local Firefox driver) and don't have the notion of multiple browsers
tests. Moreover, at this time it is not possible to pass to the suite a
customized juju-gui-source (in order to test an arbitrary proposed
branch). We need to find a way to propagate this info through juju-test,
so that the suite can be properly configured: browsers and branches.

3) In the charm functional tests, the GUI is deployed (juju deploy) and
removed (juju destroy-service) for each test in the test case. Different
initial configuration are tested (juju deploy --config ...). In the GUI
integration tests, instead, the GUI is deployed one time at the
beginning of the test run, and destroyed at the end. Different
configuration are tested using juju set, isolation is guaranteed
restarting the agent with ssh/juju ssh when required.
Merging the two suites also means making a decision about the direction
we want to follow (i.e. finding a tradeoff). For example, we want to
test and support co-location (in the current force-machine incarnation)
but IMHO we also want the suite to be fast and focused on the GUI
behavior. We want to exercise the stop hook (juju destroy-service) but
also the config-changed one (juju set).

4) Migrate the CI tests from the GUI to the charm. Also get rid of all
the Python dependencies currently required to test the GUI.

Sorry, this was longer than planned. Thoughts?

-- 
Francesco Banconi