[ubuntu-cloud] [Openstack] Update on Ubuntu automated testing and CI of Openstack

[Adam Gandelman]

On 02/27/2012 09:22 AM, Duncan McGreggor wrote:
> It's been an invaluable source for not only information, but also
> planning for the cloud work here at DreamHost.

This is great to hear!  I'm happy to hear other people are benefiting 
from this as much as Ubuntu

>
> To the point of this email, though, I have a question for you that I
> wasn't able to parse an explicit answer to from your post: For the
> packages that are built in the PPA linked above, are they only built
> after all the components of OpenStack have been confirmed working as a
> whole? Or are they built just after individual testing?

The jenkins trunk build jobs are the ones responsible for uploading to 
the PPA.  It basically stuffs the newly built package in the local 
repository and uploads it to the PPA (if it fails to build, it doesnt 
upload).  The nova trunk build also triggers the deployment + testing.   
So, to answer, those PPA packages are *pre* deployment + smoke testing.  
The idea with that is that we can use the PPA externally to debug 
specific issues that might turn up during testing without blocking the 
deployment/smoke testing queue.  We're limited by the number of machines 
we have and can not currently run deployment testing in parallel.

> My question comes from this concern: if we're building out a product
> based on this PPA, (before Precise is delivered) we want to make sure
> that when we bring up new systems by installing the packages from the
> PPA, all of those work together properly. If the latest code from
> keystone, for example, hasn't been building due to testing errors, we
> want to make sure that the presence of the older keystone package in
> the PPA won't be causing issues with the newer builds of the rest of
> OpenStack.

So you might have noticed the version of Keystone we are testing is 
getting a bit dusty.  We decided to freeze the version of Keystone we're 
testing just before the KSL/redux branch was merged.   There are still 
some features that need to land before we can modify our Juju charms to 
deploy the new version (specifically SQL persistence for the service 
catalog).  I'm hoping these will land before e4 (tomorrow) so we can 
begin testing the new branch this week.

>
> To clarify: in your blog post, you explicitly mention the validation
> process per component, starting with the upstream git repos. In the
> deploy phase, you verify that the system as a whole (all of OpenStack)
> works as expected. But what happens when one or more of those
> components don't work? Are packages rolled back in the PPA until the
> PPA only provides packages that will result in, once installed, a
> complete working system?

So far, we haven't rolled anything back in the PPA or the local 
repository.  I believe Keystone is the only package we've had to freeze 
while things stabilize upstream.  Fortunately, we've found the CI (even 
with the limited test coverage we currently run) is picking up new bugs 
almost instantaneously.  In most cases we're able to either get a fix 
into gerrit same day or find a proposed fix that we can cherry pick into 
our debian/patches/ repository and carry temporarily in our 
lp:~openstack-ubuntu-testing packages branches until its been fixed 
proper upstream.

Please understand the PPA is for testing purposes only, and we're not 
making any promise that what installs from there is 
stable/usable/working.  We're using the work around that PPA and this CI 
to essentially gate what goes into the Ubuntu archive.  This cycle, 
Chuck has been uploading a weekly snapshot of Openstack (usually every 
Friday).  With the CI in place, we can essentially verify that those 
packages build clean and install okay, and provide something usable 
given the last weeks Gerrit churn.

Keep in mind this is still a developing process.  I expect this will 
evolve over the next few months.  We've just begun trigger pre-commit 
testing to the Diablo stable branch against Oneiric.  Still working out 
some kinks, but we'll hopefully be going into Folsom+Precise with 
coverage of both trunk and stable updates to Essex/Precise LTS.

Also note that, at this point in the Ubuntu development cycle, it can 
often take *a long time* before an upload to a PPA is built and ready 
for use.  Its probably safe to assume that what you're using from the 
PPA is what we were testing yesterday (check the version of the package 
for a timestamp)

>
> Keep up the great work, guys -- you have fans out here in the wild,
> wild world of OpenStack :-)

:)

Cheers,
Adam