[Blueprint servercloud-s-juju-charm-testing] Juju Charm Testing

[Marco Ceppi]

Blueprint changed by Marco Ceppi:

Whiteboard changed:
  Discussion:
   *Bug submission on charm failure.
   *Define a process around how charm maintainers respond to test failures and subsequent bugs. Can a user run a manual test and submit the test back to the bug report to update testing status to green.
   *Enable Autocharm tester to be more resilient against provider failures, and Jenkins usage.
       simulate provider failure, and be able to recover: $ juju ssh <MACHINE> sudo shutdown now
   * Define WIs to execute auto charm testing on Go.
   * Continuous Integration (also will help with gating on charm commits)
   * Juju Testing Blogging
   * Juju testing communication to Juju lists.
   * Work on integrating/fixing Charm runner (graph testing/ dependency/env set up testing).
  
  Add a Jenkins workflow to run a charm or a set of charms in the following LXC environments:
   -raring container on raring host
   -raring container on precise host
   -precise container on raring host
   -precise container on precise host
  
  Two modes of testing:
   -Unit (does the charm start, and report ready)
   -Workload (test the charms relations, and pushing data)
  
  Reference Links:
   *Charm Test Spec [html] https://juju.ubuntu.com/docs/charm-tests.html
   * Charm Test Spec [source] http://bazaar.launchpad.net/~juju/juju/docs/view/head:/source/charm-tests.rst
   * CharmTester Charm http://jujucharms.com/~mark-mims/oneiric/charmtester
   * Charm Runner: https://launchpad.net/charmrunner
   * Jenkins Charm Testing: https://jenkins.qa.ubuntu.com/view/Charms/
  
  [USER STORIES]
  
  William is a juju user who wishes to know a charms current stability
  
- Saul is patching a charm and wants to in sure his changes are work with
+ Saul is patching a charm and wants to insure his changes work with the
  current tests
  
- Laura is a charm maintainer and wants to add tests to in sure her charm
+ Laura is a charm maintainer and wants to add tests to insure her charm
  is stable
  
  Kara is a charm maintainer and needs to know when her charm is broken
  
  Lee is a charmer who, while reviewing charm submissions, needs to know
  if these changes break backwards compatibility with currently deployed
  services
  
  Gaius is a charm maintainer from an upstream project and needs an easy
  way to learn how to write tests for his charm
  
  [ASSUMPTIONS]
  
  - Charm tester/charm tester control will work with gojuju for at least
  graph testing
  
  [RISKS]
  
  - Relying soley on "graph" testing may result in inaccurate test results due to lack of embedded tests
  - Making tests too complicated may result in low adoption rate of embedded testing
  
  [IN SCOPE]
  
  [OUT OF SCOPE]
  
  [USER ACCEPTANCE]
  
  [RELEASE NOTE/BLOG]
  
  [NOTES]
  
  UDS 1305 notes: http://pad.ubuntu.com/ep/pad/view/uds-1305-servercloud-s
  -juju-charm-testing/latest
  
  === UDS 1303 Notes ===
  Pad: http://pad.ubuntu.com/uds-1303-servercloud-r-juju-charm-testing
  
  Question:
  Is there a way in meta-data to explicitly state provider support.
      -Example: Ceph: Does cloud provider have block support
      -More broadly stated does the cloud provider have the capabilities the charm needs
  
  Idea:
    -In charm testing status be able to show that a charm failure can be a result of the provider not providing the needed capabilities, ie the Ceph charm fails on a provider because it does not support object store.
    -Make interface usage more verbose in the charm description.
    -Need a rule/spec on how a interface should be implemented
      -Need to investigate possible enforment of interfaces
   -**Have the testing framework iterate through the operational deployment requirments.
  
  Interfaces doc link broken:
    -Example: http://jujucharms.com/interfaces/ceph-client  Interface doc link broken:
        https://juju.ubuntu.com/Interfaces/ceph-client <-- broken
  
  Meta-language testing (http://paste.ubuntu.com/5588570/):
  
  Lanugage suggestions:
  http://lettuce.it/
  http://cukes.info/
  
  Charm-Runner integration:
   - https://launchpad.net/juju-deployer
  
  Wrap Go/Py juju client status:
   - https://launchpad.net/python-jujuclient
  
  ---
  
  [notes from cloudsprint 2013-05]
  Topics to cover
  Current Testing
  Current todos
  Experiences from IS
  Ideas
  
  Review charm policy to include:
  Test must pass tests
  Charm must have tests
  
  We want embedded tests!
  All tests live in the charm
  Functional Tests
  /test (in charm)
  Integration
  /test.d (in charm)
  How to make it low entry for charmers to add tests
  charm create tests (charm-tools make a stub _simple_ test)
  leverage libraries, and possibly a deployment (dare I say declarative) testing lang.
  Sidnei mocks all the juju calls (U1 testing)
  have a library that stubs this for you.
  pull this into charm-helper library
  leverage Go-watch
  leverage charm testing with charm upgrade

-- 
Juju Charm Testing
https://blueprints.launchpad.net/ubuntu/+spec/servercloud-s-juju-charm-testing