Bug 1365828 - mongo log spam

[Ian Booth]

Before I sent the original email I put up a PR to reduce the noise by
eliminating the warnings. eg we only apply the cluster admin role to the admin
database, not the presence or juju databases.

https://github.com/juju/juju/pull/714

This cuts down on the warnings but we still get repetitive syslog entries for
mongo authentication on the connections.

I also put in logging to see if we were calling login over and over and it
appears we are not. The agent is not bouncing. From what I can tell, Juju only
calls Login a finite number of times when the agent starts. Everything is
working as expected.

We do call session.Copy() when we perform a db interaction, but that to my
understanding just reuses the credentials from the original session. Maybe that
also causes mongo to create these log entries?

>From what I can see, this is a mongo logging issue that we have little control
over. But I'd love to be wrong so we can fix it.


On 09/09/14 18:53, John Meinel wrote:
> On Tue, Sep 9, 2014 at 12:27 PM, Andrew Wilkins <
> andrew.wilkins at canonical.com> wrote:
> 
>> On Tue, Sep 9, 2014 at 4:14 PM, John Meinel <john at arbash-meinel.com>
>> wrote:
>>
>>> I don't think these are from authentication between mongo instances.
>>>
>>
>> Agreed, looks like the machine-0 jujud connecting over and over again.
>> Perhaps the agent is unhappy and is bouncing.
>>
> 
> It is possible, but its also possible that the new session copying means we
> just have multiple connections (conn11) to the database, and each one is
> going to be authenticated.
> 
> 
>>
>>
>>> This looks more like a Mongo 2.4 vs 2.6 difference. We added stuff like
>>> "clusterAdmin" to priviledges that we give because of 2.6 support (I
>>> thought).. But if 2.4 doesn't know anything about "clusterAdmin" then
>>> whenever we log in, it goes through the list of roles and noisily ignores
>>> some of them.
>>>
>>
>> clusterAdmin is for replica sets, and is applicable to 2.4. In particular,
>> in grants replSetInitiate. I went through and removed the extraneous stuff
>> when I did the 2.6 support updates.
>> See http://docs.mongodb.org/v2.4/reference/user-privileges/
>>
> 
> Digging into it a bit more, it is only complaining about not having those
> roles on the "juju" and "presence" databases, they are probably present on
> the "admin" database.
> 
> Looking at the code, the only place I see RoleClusterAdmin is while
> connected to the Admin database.
> "clusterAdmin
> <http://docs.mongodb.org/manual/reference/built-in-roles/#clusterAdmin> is
> only applicable on the admin database, and does not confer any access to
> the local or config databases."
> 
> I wonder if this is either:
> 1) Mongo just being overly verbose in warnings that aren't actually
> warnings. (You logged in with a user that has clusterAdmin, but that
> doesn't apply to this other database you are accessing.)
> 2) We somehow created a User on the Juju/Presence databases and then also
> gave it "clusterAdmin" rights there.
> 
> The only place I see any of this is via the mongo.SetAdminMongoPassword
> call, though, and that does appear to forcibly only do the UpsertUser
> rights on "admin".
> So I'm *guessing* we are running into (1). We are connecting directly to
> the "juju" database, and we are connecting as a user that has clusterAdmin
> but it just doesn't do anything in this context and mongo warns about it.
> An idea is that we always connect to "admin" and then switch databases once
> connected? I think that's how I've done it for connecting when using
> "mongo" the CLI client.
> 
> John
> =:->
> 
> 
> 
> 
>> Presumably the real fix is to only give roles that are 2.6 specific when
>>> we know we're running against 2.6 ?
>>>
>>> (I could be wrong on this, but I thought we had earlier bugs that we
>>> couldn't do the cluster reconfiguration on 2.6 because we had to add the
>>> "clusterAdmin" role.)
>>>
>>> John
>>> =:->
>>>
>>>
>>> On Tue, Sep 9, 2014 at 11:22 AM, Ian Booth <ian.booth at canonical.com>
>>> wrote:
>>>
>>>> https://bugs.launchpad.net/juju-core/+bug/1365828
>>>>
>>>> Hi mongo experts
>>>>
>>>> Mongo is filling up syslog with lots of spam.
>>>> eg, as reported by a customer, https://clbin.com/XZ63i
>>>>
>>>> I have reduced this a little by tweaking the roles used to log in to the
>>>> "juju"
>>>> and "presence" databases, so the warnings are gone. But the spam remains.
>>>>
>>>> It seems to me these Mongo bugs might be relevant:
>>>>
>>>> https://jira.mongodb.org/browse/SERVER-5847
>>>> https://jira.mongodb.org/browse/SERVER-5952
>>>>
>>>> I can't see from a brief look what Juju might be doing to directly cause
>>>> the
>>>> spam, nor what we can do to remove it.
>>>>
>>>> Does anyone with more Mongo knowledge have any insight as to what we can
>>>> do to
>>>> address this problem?
>>>>
>>>>
>>>> --
>>>> Juju-dev mailing list
>>>> Juju-dev at lists.ubuntu.com
>>>> Modify settings or unsubscribe at:
>>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>>
>>>
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev at lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>>
>>
>