[Blueprint servercloud-q-juju-charm-unit-tests] Juju Charm Unit Tests

[Antonio Rosales]

Blueprint changed by Antonio Rosales:

Whiteboard changed:
  charm unit tests
  
  recap... we automatically run:
    - installation
    - graph test
  we'd like to additionally run
    - charm tests
  
  read the spec lp:juju in the "docs" dir... look for something like charmtests.rst
  http://bazaar.launchpad.net/~clint-fewbar/juju/charm-tests-spec/view/head:/source/charm-tests.rst
  
  policy stuff?
  tests required?  just encouraged for now
  
  additional testing plugins possible:
    - nessus, etc against the running service
    - app-armor complain against running service
    - lint-type 'charm proof'
    - upgrade-charm tests
    - testing spread with add-unit
    - coverage ???
    - watching relation settings??????
  
  please put all projects in one of:
  charm-tools
  juju-jitsu
  charms
  juju
  
  environments - include all of them, including MAAS
  
  support MAAS, other distro needs
  
  enable manual kicks from ~charmers (charmbot, but we prob wanna do that
  through openid over the web) - parameterized jenkins tests, both
  official and against specific distro needs
  
  charmtester is a *charm* in the charm store - test my custom stack of
  services is readily doable (contributions are welcome as always)
  
  jenkins file bugs when charms fail (close them when they pass again?)
  try for "maintainer" notification through launchpad
  
  how do long-failing tests relate to "series" freeze???
  
  coverage tools possible?  check out next cycle
  
  load testing for individual charms?  for example, siege all charms with
  http interface
  
  scale testing?  weekly large runs?  really a _juju_ test and less a
  _charm_ test
  
  ---
  User Stories:
-   *Levi has just submitted and gotten his new Charm approved for inclusion in the Juju Charm store. Levi and other Juju community members would like to ensure his new Charm is staying functional with updates, and properly interacting with related services. Furthermore, Levi would like additional tools to give him further insights on how to improve his Charm.
-   *Andrew has submitted his charm and gotten it accepted in a previous Ubuntu release. Andrew would like to regularly confirm his charm is running appropriately in the current release under development.
-   *Naomi is interested in using MAAS/OpenStack/EC2 to deploy services via Juju. She would like to review charm testing results for a given Ubuntu release and have insights into how charms are performing in that release.
-   *Sarah is are regular charm and has a few charms she is maintaining. She would like to get Launchpad notifications when one of her charms fails in CharmTester. Sarah would like to easily look at a testing reporting interface and drill down to the logs to start the debugging process.
+   *Levi has just submitted and gotten his new Charm approved for inclusion in the Juju Charm store. Levi and other Juju community members would like to ensure his new Charm is staying functional with updates, and properly interacting with related services. Furthermore, Levi would like additional tools to give him further insights on how to improve his Charm.
+   *Andrew has submitted his charm and gotten it accepted in a previous Ubuntu release. Andrew would like to regularly confirm his charm is running appropriately in the current release under development.
+   *Naomi is interested in using MAAS/OpenStack/EC2 to deploy services via Juju. She would like to review charm testing results for a given Ubuntu release and have insights into how charms are performing in that release.
+   *Sarah is are regular charm and has a few charms she is maintaining. She would like to get Launchpad notifications when one of her charms fails in CharmTester. Sarah would like to easily look at a testing reporting interface and drill down to the logs to start the debugging process.
  
  Test Plans:
-   *need formal plan [a.rosales]
+   *Run charm tester against new and current charms in supported Ubuntu releases against supported Juju providers.
  
  Assumptions:
-   * Juju providers are functional to test in
+   * Juju providers are functional to test in
  
  Release Notes:
-   * Add availability notice and information on CharmTester.
+   * Add availability notice and information on CharmTester.

-- 
Juju Charm Unit Tests
https://blueprints.launchpad.net/ubuntu/+spec/servercloud-q-juju-charm-unit-tests