simple streams failure - no 12.04 amd64 image in ap-southeast-2

[Ian Booth]

The latest simplestreams metadata upload is broken. The latest data as of 2
hours ago is the culprit (around 10-Jul-2013 04:14 UTC)
The expected format is something like:

{
 "index": {
   "com.ubuntu.cloud:custom": {
    "updated": "...",
    "clouds": [
     {
       "region": "...",
       "endpoint": "..."
     }
    ],
    "cloudname": "custom",
    "datatype": "image-ids",
    "format": "products:1.0",
    "products": [
      "com.ubuntu.cloud:server:...:..."
    ],
    "path": ".../..."
   }
  },
  "updated": "...",
  "format": "index:1.0"
}

Instead we have something like:

{
   "com.ubuntu.cloud:custom": {
    "updated": "...",
    "clouds": [
     {
       "region": "...",
       "endpoint": "..."
     }
    ],
    "cloudname": "custom",
    "datatype": "image-ids",
    "format": "products:1.0",
    "products": [
      "com.ubuntu.cloud:server:...:..."
    ],
    "path": ".../..."
   }
  "updated": "...",
  "format": "index:1.0"
}

This issue needs to be resolved by uploading the image metadata in the expected
format.

This incident has highlighted some problems with our image metadata deployment
processes:
1. We need to have access to archived simplestreams files so a bad deploy can
easily be compared to known working copies and rolled back if required
2. We need to not upload new metadata to *production* without first putting it
in some sort of staging area where acceptance tests can be run
3. We need to have processes in place to initiate a rollback since as of right
now, juju is broken for EC2

We'll obviously work to get the immediate problem fixed ASAP and the above
issues addressed.

On 10/07/13 14:57, David Cheney wrote:
> Please ignore the debugging info in my build - what has gone wrong ?
> 
> % juju bootstrap -v
> 2013-07-10 04:54:42 INFO juju ec2.go:128 environs/ec2: opening environment
> "ap-southeast-2"
> 2013-07-10 04:54:43 INFO juju ec2.go:238 environs/ec2: bootstrapping
> environment "ap-southeast-2"
> dialing connection 1 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 2 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 3 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 4 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 5 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 6 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 7 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 8 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 9 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 10 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 11 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 12 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 13 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 14 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 15 s3-ap-southeast-2.amazonaws.com:443
> dialing connection 16 s3-ap-southeast-2.amazonaws.com:443
> 2013-07-10 04:54:49 INFO juju tools.go:25 environs: reading tools with
> major version 1
> dialing connection 17 s3-ap-southeast-2.amazonaws.com:443
> 2013-07-10 04:54:49 INFO juju tools.go:29 environs: falling back to public
> bucket
> dialing connection 18 s3.amazonaws.com:443
> 2013-07-10 04:54:51 INFO juju tools.go:52 environs: filtering tools by
> series: precise
> 2013-07-10 04:54:51 INFO juju tools.go:75 environs: picked newest version:
> 1.11.2
> findInstanceSpec: baseURLs [http://cloud-images.ubuntu.com/releases]
> fetchData: http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson
> dialing connection 19 cloud-images.ubuntu.com:80
> 2013-07-10 04:54:51 ERROR juju supercommand.go:234 command failed: cannot
> start bootstrap instance: no "precise" images in ap-southeast-2 with arches
> [amd64 i386]
> error: cannot start bootstrap instance: no "precise" images in
> ap-southeast-2 with arches [amd64 i386]
> 
> 
>