[Blueprint servercloud-p-move-ec2-mirrors-to-s3] Move EC2 mirrors to S3

[Ben Howard]

Blueprint changed by Ben Howard:

Whiteboard changed:
- Questions:
- - 'LAN' bandwidth access to the region mirrors is currently free already, no? -- Daviey
+ Q. 'LAN' bandwidth access to the region mirrors is currently free already, no? -- Daviey
+ A. All upload and intra-zone transit is free. 
+ 
+ Q. How do you keep people out of EC2 from access?
+ A. Bucket policies enable restricting access to the Amazon CIDR address. 
  
  Comments:
  - Can this whiteboard be pre-filed with some examples of how others have implemented this?
+    *  AFAIK, there is only one implementation of S3 buckets. Most examples that I know of us S3 as a storage backend, while having an EC2 instance front the S3 storage. Amazon Linux AMI is the only pure S3 backend/frontend solution. 
+ 
+ - How would this work?
+   * There are a number of ways to push out a repository using existing tools. s3fuse (albeit rather buggy) allows you to mount a s3 bucket as file system. s3cmd allows for local to remote synchronization. However, in order to build something that scales well, you need something that uses multiple connections and pushes multiple files at once (i.e. threading and boto python library)
+ 
+ --
+ 
+ Examples: 
+ Amazon Linux AMI currently uses S3 for backing the mirrors. The design is more or less:
+     - One bucket per region. 
+     - Buckets are named off of the DNS CNAME, i.e. "us-east-1.ec2.archive.ubuntu.com"
+     - a method of pushing pristine repositories up

-- 
Move EC2 mirrors to S3
https://blueprints.launchpad.net/ubuntu/+spec/servercloud-p-move-ec2-mirrors-to-s3