Fwd: Charm store API proposal, new version

[roger peppe]

[resending with correct From address]

On 15 July 2014 21:07, Aaron Bentley <aaron.bentley at canonical.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 14-07-15 11:43 AM, Richard Harding wrote:
>> On Tue, 15 Jul 2014, Aaron Bentley wrote:
>>
>>> On 14-07-15 10:17 AM, roger peppe wrote:
>>>> The most significant change is that all endpoints refer just
>>>> to a single charm or bundle, rather than being "bulk" calls as
>>>> they were before.
>>>
>>> That sounds like the opposite of what juju-reports wants.  Does
>>> it really mean that we would need to make N requests to retrieve
>>> info on N charms?  What was the reason for changing this?
>>
>> The change is because we went through the known clients and could
>> not come up with use cases where some client had a known list of
>> charm ids they wanted data for, but they knew those ids ahead of
>> time.
>
> I am surprised that juju-reports was not considered a known client.  I
> certainly made many comments on the first draft of the original proposal.
>
>> Aaron, I'd like to see if we can get your full use case details
>> and make sure we address them.
>
> Our use case is we want to be able to synchronize public data from the
> charm store into our DB.
>
> We need to provide the statistics that show adoption rates to Mark
> Ramm and Mark Shuttleworth.  We cannot forsee exactly which questions
> will be asked.  We need to be able to add new statistics without
> friction, so we need our own copy of this data.
>
> One way to do this would be to have an API that accepts a timestamp
> and returns the metadata for all charm and bundle revisions created on
> or after that timestamp.
>
> Right now we do this instead:
> 1. Retrieve a list of all charms from charmworld
> 2. Determine the tip revisions of all charms from the charm store
> 3. Determine which revisions exist but we haven't seen (including
> non-tip revisions)
> 4. Determine the revision ids of all the new revisions
> 5. Use the revision-ids to determine the publication dates of all the
> new revisions
>
> Steps 2, 4, 5 use multiple-id queries.
>
> Right now, we're showing number of charms published here:
> http://reports.vapour.ws/scorecard
>
> We'll need to do the same for bundles.
>
> We've also been asked to display the number of running environments.
>
>> You had requested an all charms/bundles api call in the last
>> revision of the doc and we're looking at addresssing that through
>> search results.
>
> The doc doesn't say that search can be used to list all charms and
> bundles.  If you meant it to be used that way, please say so in the
> doc ;-)
>
> Search provides results for only the most recent revisions.  It can
> stand in for step 1 & 2 of our algorithm, but in order to implement 4
> and 5 efficiently, we'd need to do multi-id queries.
>
> Aaron
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJTxYnvAAoJEK84cMOcf+9hr04IAM9SwL4xZv0Dez33KiA8/SJf
> ahApZSuiWfDKR5h2DSKuuW+ZgzMwHOJwtMUd+wmFu6s5W4+2rI5IU60EQBd2M+3u
> JG2ZkK3C0qvibxnlDwY0gyy/atwdEnOf4X5/5sx/gI08ZtqnmpMpUohDMGMFHTHc
> 191i9ZZJVz/FvC+FxIzyd0p79veAZf1lvEWK2tDunX+H5VTwhGvfGnnjjraNxXTE
> bzFNwIUV8uuRiIETGKDNBbgncdMfNW48bw76TB+sQoy4ElL3Nl9pnA4o/i9zwzX/
> HcvA6W3DlknLgfH6odHL+Ot70txt8v9DObRSMifGSlXMYHuox3gZT/bs4XRVRS4=
> =GRZs
> -----END PGP SIGNATURE-----
>
> --
> Juju-dev mailing list
> Juju-dev at lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev

FWIW, it would be easy to provide a bulk endpoint for charm/bundle
metadata in addition to the existing path-based calls.

We could define:

    meta/$endpoint?id=$id0[&id=$id1...][$otherflags]

to be equivalent to the aggregation of the results from:

    {$id0/meta/$endpoint$otherflags, $id1/meta/$endpoint$otherflags, ...}

for id0, id1, ... across all the ids from the original request.

The additional amount of code to implement this should not be great.