[Blueprint servercloud-p-complex-deployment-testing] Automated Complex Server Deployment Testing

[James Page]

Blueprint changed by James Page:

Whiteboard changed:
  Scope: Deployment and Testing of Ubuntu Server to support complex requirements i.e. OpenStack + others
  Coverage:
-   OpenStack
-   
+   - OpenStack
+   - Wiki + Database + Memcache + Load Balancer
+   - Networking File Systems
+   - Samba + LDAP or Kerberos Backend
+   - Backup solutions - bacula?
+   - Ruby on Rails stack
+ 
+ UDS-O - Complex Deployment Scenarios:
+ 
  
  Method: orchestra+juju AND cloud+juju
  
  james-page proposed a method for testing juju charms on juju at l.u.c - see
  https://lists.ubuntu.com/archives/juju/2011-October/000837.html
  
  Broadley this involves a new charm when can test other charms - results are then collated and can be reported on.
  This is then executed by a framework (TBN) which set's up the juju environment and services, executes the charm testing and then can execute tests on the deployed environment.
  I think this is the right way todo this testing - it can be used with any juju provider so can be applied in both physical and cloud environments.
  
  Frequency: TBC
  
  Daily Test with current development release
  
  for OpenStack:
  
  Daily Test of upstream trunk?
  
- 
  Questions/Proposals:
  [james-page] We need to verify the list of deployments and get volunteers to develop charms for them where gaps exist.
  
  Work Items:
  Develop juju charm testing framework to support all requirements: TODO
  Writing testing hooks for all charms which we want to test this way: TODO
  Setup OpenStack test lab: TODO
  Identify OpenStack testing requirements on the deployed service: TODO
  
- >>>>>>>>>>>>>>>>>>>>>>>
  
- UDS-O Session Notes:
- 
- 1) Complex Deployment Testing
- 
- Alot of Ubuntu server deployments are much more than an ISO install on a
- single node; however this is pretty much the limit of the automated ISO
- testing that we currently complete.
- 
- We should aim to target some complex Ubuntu server deployment scenarios
- for regular automated testing during the release cycle.
- 
- This could be implemented either in the cloud or using virtualization;
- ensemble and cobbler+puppet might form part of the solution.
- 
- 2) Automated Boot testing
- 
- As we make changes to the boot/shutdown sequence of the server, and
- migrate init.d scripts to upstart jobs, we need some continuous
- integration testing to make sure it continues to work. We should take
- advantage of the automated testing framework to install, reboot,
- shutdown, start, remove, and repeat with all packages that are changed.
- We should also automate this testing with all of the available tasks on
- the install CDs. In addition, we should collect metrics on boot time and
- how each package affects it.
- 
- hggdh:
-  * we need to test iso install
-  * want to do a single install and create gold copy
-  * need test do-release-upgrade
- 
- High Level Test Objectives:
- -------------------------------------
- 
- a) Deployment Testing:
- 
- i) ISO Image Proving
- 
- RAID testing (N.B. For raid testing also see: https://blueprints.launchpad.net/ubuntu/+spec/foundations-o-software-raid and/or etherpad until blueprint updated)
- LVM testing
- Multipath and encryption
- 
- Existing ISO testing framework should deal with this.
- 
- ii) AMI Image Proving
- 
- Existing ec2 testing framework should deal with this.
+ Stuff left from UDS-O not covered in other specs
+ --------------------------------------------------------------------
  
  b) Package Removal Testing:
  
  - install services
  - start services
  - check services
  - stop services
  - uninstall services
  - re-install same services
  - start & re-check services
  
- c) Phase 2 - Service Proving
- 
- Deployment of service and pluggable testing
- 
- Deploy through orchestra and/or ensemble - which would allow testing of
- recipes and formulas as well (+1)
- 
- Framework required for executing tests once deployment complete;
- mcollective could be used to distribute and execute tests; maybe still
- use python unittests;
- 
  d) Automated Boot Testing:
  
  Minimal install  - startup and shutdown OK
  Full install - startup and shutdown OK
  Complete upgrade and then re-test.
  
- e) Upgrade Testing:
- 
- Lucid->Maverick->Natty for example.
- 
- Need to catchup with mvo in this area - suspect some sort of gold-copy
- type setup might make sense.
- 
  f) Recovery Testing:
  
  Recovery Mode Testing:
- 
- Complex Deployment Scenarios:
- 
- Wiki + Database + Memcache + Load Balancer
- Bridging and bonding of network interfaces
- Networking File Systems
- Openstack Deployment test
- Launchpad
- Samba + LDAP or Kerberos Backend
- Backup solutions.
- Ruby on Rails stack?
- 
- Requirements:
- 
- Remote testing of services.
- 
- Possibly ask community to donate resources for optional tests

-- 
Automated Complex Server Deployment Testing
https://blueprints.launchpad.net/ubuntu/+spec/servercloud-p-complex-deployment-testing