Changes to StorageReader Get() and List() semantics

[Ian Booth]

he Hi folks

TL;DR;

The semantics of StorageReader Get() and List() have changed. Depending on the
capabilities of the implementation, calling these methods directly no longer
retries if the objects are not found. Instead, use these new helper methods
defined in environs.storage

// Get gets the named file from stor using the stor's default consistency strategy.
func Get(stor StorageReader, name string) (io.ReadCloser, error)

// GetWithRetry gets the named file from stor using the specified attempt strategy.
func GetWithRetry(stor StorageReader, name string, attempt
utils.AttemptStrategy) (r io.ReadCloser, err error)

// List lists the files matching prefix from stor using the stor's default
consistency strategy.
func List(stor StorageReader, prefix string) ([]string, error)

// ListWithRetry lists the files matching prefix from stor using the specified
attempt strategy.
func ListWithRetry(stor StorageReader, prefix string, attempt
utils.AttemptStrategy) (list []string, err error)

Additionally, the StorageReader, StorageWriter and Storage interfaces are moved
from environs to environs.storage


A longer explanation........

The StroageReader interface defines Get() and List() methods. On some clouds
like EC2, the objects being retrieved may not be there immediately, and so the
operation is retried. There were 2 places which were providing the retry logic:
1. the goamz library s3 code
2. the juju environ provider

It can be argued either way whether having 2 places do the retries is sensible.
IMO it's not. But a real problem is that the s3 code *always* retries and this
is not desirable when the object being retrieved may genuinely not be there.
This manifested itself in the time taken to bootstrap when searching for tools
using simplestreams.

So, the s3 retry logic has been deactivated, and all retries are done solely by
the juju storage implementations, as and when needed. Additionally, callers can
specify a different retry strategy if needed to suit the particular use case.