[ubuntu-cloud] [Openstack] Update on Ubuntu automated testing and CI of Openstack
adamg at canonical.com
Tue Feb 28 03:36:18 UTC 2012
On 02/27/2012 09:22 AM, Duncan McGreggor wrote:
> It's been an invaluable source for not only information, but also
> planning for the cloud work here at DreamHost.
This is great to hear! I'm happy to hear other people are benefiting
from this as much as Ubuntu
> To the point of this email, though, I have a question for you that I
> wasn't able to parse an explicit answer to from your post: For the
> packages that are built in the PPA linked above, are they only built
> after all the components of OpenStack have been confirmed working as a
> whole? Or are they built just after individual testing?
The jenkins trunk build jobs are the ones responsible for uploading to
the PPA. It basically stuffs the newly built package in the local
repository and uploads it to the PPA (if it fails to build, it doesnt
upload). The nova trunk build also triggers the deployment + testing.
So, to answer, those PPA packages are *pre* deployment + smoke testing.
The idea with that is that we can use the PPA externally to debug
specific issues that might turn up during testing without blocking the
deployment/smoke testing queue. We're limited by the number of machines
we have and can not currently run deployment testing in parallel.
> My question comes from this concern: if we're building out a product
> based on this PPA, (before Precise is delivered) we want to make sure
> that when we bring up new systems by installing the packages from the
> PPA, all of those work together properly. If the latest code from
> keystone, for example, hasn't been building due to testing errors, we
> want to make sure that the presence of the older keystone package in
> the PPA won't be causing issues with the newer builds of the rest of
So you might have noticed the version of Keystone we are testing is
getting a bit dusty. We decided to freeze the version of Keystone we're
testing just before the KSL/redux branch was merged. There are still
some features that need to land before we can modify our Juju charms to
deploy the new version (specifically SQL persistence for the service
catalog). I'm hoping these will land before e4 (tomorrow) so we can
begin testing the new branch this week.
> To clarify: in your blog post, you explicitly mention the validation
> process per component, starting with the upstream git repos. In the
> deploy phase, you verify that the system as a whole (all of OpenStack)
> works as expected. But what happens when one or more of those
> components don't work? Are packages rolled back in the PPA until the
> PPA only provides packages that will result in, once installed, a
> complete working system?
So far, we haven't rolled anything back in the PPA or the local
repository. I believe Keystone is the only package we've had to freeze
while things stabilize upstream. Fortunately, we've found the CI (even
with the limited test coverage we currently run) is picking up new bugs
almost instantaneously. In most cases we're able to either get a fix
into gerrit same day or find a proposed fix that we can cherry pick into
our debian/patches/ repository and carry temporarily in our
lp:~openstack-ubuntu-testing packages branches until its been fixed
Please understand the PPA is for testing purposes only, and we're not
making any promise that what installs from there is
stable/usable/working. We're using the work around that PPA and this CI
to essentially gate what goes into the Ubuntu archive. This cycle,
Chuck has been uploading a weekly snapshot of Openstack (usually every
Friday). With the CI in place, we can essentially verify that those
packages build clean and install okay, and provide something usable
given the last weeks Gerrit churn.
Keep in mind this is still a developing process. I expect this will
evolve over the next few months. We've just begun trigger pre-commit
testing to the Diablo stable branch against Oneiric. Still working out
some kinks, but we'll hopefully be going into Folsom+Precise with
coverage of both trunk and stable updates to Essex/Precise LTS.
Also note that, at this point in the Ubuntu development cycle, it can
often take *a long time* before an upload to a PPA is built and ready
for use. Its probably safe to assume that what you're using from the
PPA is what we were testing yesterday (check the version of the package
for a timestamp)
> Keep up the great work, guys -- you have fans out here in the wild,
> wild world of OpenStack :-)
More information about the Ubuntu-cloud