RFC: mongo "_id" fields in the multi-environment juju server world

[Gustavo Niemeyer]

On Mon, Jul 7, 2014 at 12:26 PM, roger peppe <roger.peppe at canonical.com> wrote:
> On 7 July 2014 14:27, Gustavo Niemeyer <gustavo at niemeyer.net> wrote:
>> On Mon, Jul 7, 2014 at 10:09 AM, roger peppe <roger.peppe at canonical.com> wrote:
>>> I had assumed that because every client needs to see every transaction
>>> there would likely be no benefit to sharding the log, although
>>> technically you could shard on transaction id. I'd be
>>
>> Clients don't need to see every transaction. Only those that affect
>> the documents they are acting on.
>
> Is it actually possible to shard the transaction log based on the documents
> the transactions are acting on?

That's unrelated to what you said above, or to my response.

Either way, we can shard transaction documents, and we can add a shard
key to them if necessary.

>>> Thanks for pointing this out. If we manage to hugely scale juju using mongodb
>>> I will be very happy. I still think we should do some measurements to
>>> convince us that we actually have some hope of doing so though.
>>> My own measurements left me less than convinced of the
>>> possibility, although it's been a while since I did them.
>>
>> When you measured a sharded setup, what was the outcome?
>
> I simply measured operation rate (of some actual juju operations)
> on a non-sharded setup.

Okay, so the measurements that left you unconvinced that sharding
might help to scale up were not using sharding.

> I saw around 60 operations per second.
> It may well have been that I was testing an inefficient setup, or
> that my mongo settings were inadequate.

I cannot really comment on that. What I can say is:

1. The txn package can run transactions on the order of a few hundred
per second on my measurements on MongoDB 2.2

2. Sharding allows sending load to independent replica sets

3. MongoDB performance is improving release over release, and there's
more coming (http://goo.gl/qPE9LB)

4. Nothing will work without effort.


gustavo @ http://niemeyer.net