[Blueprint server-o-complex-deployment-testing] Automated Complex Server Deployment Testing

[Robbie Williamson]

Blueprint changed by Robbie Williamson:

Whiteboard changed:
+ Work items:
+ Base framework to support complex deployment testing: TODO
+ Configurations for complex deployment scenarios (may overlap with formulas and recipes): TODO
+ Update ISO testing to deal with server ISO refactoring work: TODO
+ ec2 testing configuration for kernel team patch testing work: TODO
+ 
  UDS-O Session Notes:
  
  1) Complex Deployment Testing
  
  Alot of Ubuntu server deployments are much more than an ISO install on a
  single node; however this is pretty much the limit of the automated ISO
  testing that we currently complete.
  
  We should aim to target some complex Ubuntu server deployment scenarios
  for regular automated testing during the release cycle.
  
  This could be implemented either in the cloud or using virtualization;
  ensemble and cobbler+puppet might form part of the solution.
  
  2) Automated Boot testing
  
  As we make changes to the boot/shutdown sequence of the server, and
  migrate init.d scripts to upstart jobs, we need some continuous
  integration testing to make sure it continues to work. We should take
  advantage of the automated testing framework to install, reboot,
  shutdown, start, remove, and repeat with all packages that are changed.
  We should also automate this testing with all of the available tasks on
  the install CDs. In addition, we should collect metrics on boot time and
  how each package affects it.
  
  hggdh:
   * we need to test iso install
   * want to do a single install and create gold copy
   * need test do-release-upgrade
  
  High Level Test Objectives:
  -------------------------------------
  
  a) Deployment Testing:
  
  i) ISO Image Proving
  
  RAID testing (N.B. For raid testing also see: https://blueprints.launchpad.net/ubuntu/+spec/foundations-o-software-raid and/or etherpad until blueprint updated)
  LVM testing
  Multipath and encryption
  
  Existing ISO testing framework should deal with this.
  
  ii) AMI Image Proving
  
  Existing ec2 testing framework should deal with this.
  
  b) Package Removal Testing:
  
  - install services
  - start services
  - check services
  - stop services
  - uninstall services
  - re-install same services
  - start & re-check services
  
  c) Phase 2 - Service Proving
  
  Deployment of service and pluggable testing
  
  Deploy through orchestra and/or ensemble - which would allow testing of
  recipes and formulas as well (+1)
  
  Framework required for executing tests once deployment complete;
  mcollective could be used to distribute and execute tests; maybe still
  use python unittests;
  
  d) Automated Boot Testing:
  
  Minimal install  - startup and shutdown OK
  Full install - startup and shutdown OK
  Complete upgrade and then re-test.
  
  e) Upgrade Testing:
  
  Lucid->Maverick->Natty for example.
  
  Need to catchup with mvo in this area - suspect some sort of gold-copy
  type setup might make sense.
  
  f) Recovery Testing:
  
  Recovery Mode Testing:
  
  Complex Deployment Scenarios:
  
  Wiki + Database + Memcache + Load Balancer
  Bridging and bonding of network interfaces
  Networking File Systems
  Openstack Deployment test
  Launchpad
  Samba + LDAP or Kerberos Backend
  Backup solutions.
  Ruby on Rails stack?
  
  Requirements:
  
  Remote testing of services.
  
  Possibly ask community to donate resources for optional tests
- 
- Work items:
- Base framework to support complex deployment testing: TODO
- Configurations for complex deployment scenarios (may overlap with formulas and recipes): TODO
- Update ISO testing to deal with server ISO refactoring work: TODO
- ec2 testing configuration for kernel team patch testing work: TODO

-- 
Automated Complex Server Deployment Testing
https://blueprints.launchpad.net/ubuntu/+spec/server-o-complex-deployment-testing