Bug 1365828 - mongo log spam

[John Meinel]

I'm pretty sure that is the wrong fix, as the API servers need
"clusterAdmin" role so they can set up the Replica Sets when users use
"juju ensure-availability". The reconfiguring is done by the machine agent,
not the user admin.

John
=:->


On Tue, Sep 9, 2014 at 2:12 PM, Ian Booth <ian.booth at canonical.com> wrote:

> Before I sent the original email I put up a PR to reduce the noise by
> eliminating the warnings. eg we only apply the cluster admin role to the
> admin
> database, not the presence or juju databases.
>
> https://github.com/juju/juju/pull/714
>
> This cuts down on the warnings but we still get repetitive syslog entries
> for
> mongo authentication on the connections.
>
> I also put in logging to see if we were calling login over and over and it
> appears we are not. The agent is not bouncing. From what I can tell, Juju
> only
> calls Login a finite number of times when the agent starts. Everything is
> working as expected.
>
> We do call session.Copy() when we perform a db interaction, but that to my
> understanding just reuses the credentials from the original session. Maybe
> that
> also causes mongo to create these log entries?
>
> From what I can see, this is a mongo logging issue that we have little
> control
> over. But I'd love to be wrong so we can fix it.
>
>
> On 09/09/14 18:53, John Meinel wrote:
> > On Tue, Sep 9, 2014 at 12:27 PM, Andrew Wilkins <
> > andrew.wilkins at canonical.com> wrote:
> >
> >> On Tue, Sep 9, 2014 at 4:14 PM, John Meinel <john at arbash-meinel.com>
> >> wrote:
> >>
> >>> I don't think these are from authentication between mongo instances.
> >>>
> >>
> >> Agreed, looks like the machine-0 jujud connecting over and over again.
> >> Perhaps the agent is unhappy and is bouncing.
> >>
> >
> > It is possible, but its also possible that the new session copying means
> we
> > just have multiple connections (conn11) to the database, and each one is
> > going to be authenticated.
> >
> >
> >>
> >>
> >>> This looks more like a Mongo 2.4 vs 2.6 difference. We added stuff like
> >>> "clusterAdmin" to priviledges that we give because of 2.6 support (I
> >>> thought).. But if 2.4 doesn't know anything about "clusterAdmin" then
> >>> whenever we log in, it goes through the list of roles and noisily
> ignores
> >>> some of them.
> >>>
> >>
> >> clusterAdmin is for replica sets, and is applicable to 2.4. In
> particular,
> >> in grants replSetInitiate. I went through and removed the extraneous
> stuff
> >> when I did the 2.6 support updates.
> >> See http://docs.mongodb.org/v2.4/reference/user-privileges/
> >>
> >
> > Digging into it a bit more, it is only complaining about not having those
> > roles on the "juju" and "presence" databases, they are probably present
> on
> > the "admin" database.
> >
> > Looking at the code, the only place I see RoleClusterAdmin is while
> > connected to the Admin database.
> > "clusterAdmin
> > <http://docs.mongodb.org/manual/reference/built-in-roles/#clusterAdmin>
> is
> > only applicable on the admin database, and does not confer any access to
> > the local or config databases."
> >
> > I wonder if this is either:
> > 1) Mongo just being overly verbose in warnings that aren't actually
> > warnings. (You logged in with a user that has clusterAdmin, but that
> > doesn't apply to this other database you are accessing.)
> > 2) We somehow created a User on the Juju/Presence databases and then also
> > gave it "clusterAdmin" rights there.
> >
> > The only place I see any of this is via the mongo.SetAdminMongoPassword
> > call, though, and that does appear to forcibly only do the UpsertUser
> > rights on "admin".
> > So I'm *guessing* we are running into (1). We are connecting directly to
> > the "juju" database, and we are connecting as a user that has
> clusterAdmin
> > but it just doesn't do anything in this context and mongo warns about it.
> > An idea is that we always connect to "admin" and then switch databases
> once
> > connected? I think that's how I've done it for connecting when using
> > "mongo" the CLI client.
> >
> > John
> > =:->
> >
> >
> >
> >
> >> Presumably the real fix is to only give roles that are 2.6 specific when
> >>> we know we're running against 2.6 ?
> >>>
> >>> (I could be wrong on this, but I thought we had earlier bugs that we
> >>> couldn't do the cluster reconfiguration on 2.6 because we had to add
> the
> >>> "clusterAdmin" role.)
> >>>
> >>> John
> >>> =:->
> >>>
> >>>
> >>> On Tue, Sep 9, 2014 at 11:22 AM, Ian Booth <ian.booth at canonical.com>
> >>> wrote:
> >>>
> >>>> https://bugs.launchpad.net/juju-core/+bug/1365828
> >>>>
> >>>> Hi mongo experts
> >>>>
> >>>> Mongo is filling up syslog with lots of spam.
> >>>> eg, as reported by a customer, https://clbin.com/XZ63i
> >>>>
> >>>> I have reduced this a little by tweaking the roles used to log in to
> the
> >>>> "juju"
> >>>> and "presence" databases, so the warnings are gone. But the spam
> remains.
> >>>>
> >>>> It seems to me these Mongo bugs might be relevant:
> >>>>
> >>>> https://jira.mongodb.org/browse/SERVER-5847
> >>>> https://jira.mongodb.org/browse/SERVER-5952
> >>>>
> >>>> I can't see from a brief look what Juju might be doing to directly
> cause
> >>>> the
> >>>> spam, nor what we can do to remove it.
> >>>>
> >>>> Does anyone with more Mongo knowledge have any insight as to what we
> can
> >>>> do to
> >>>> address this problem?
> >>>>
> >>>>
> >>>> --
> >>>> Juju-dev mailing list
> >>>> Juju-dev at lists.ubuntu.com
> >>>> Modify settings or unsubscribe at:
> >>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >>>>
> >>>
> >>>
> >>> --
> >>> Juju-dev mailing list
> >>> Juju-dev at lists.ubuntu.com
> >>> Modify settings or unsubscribe at:
> >>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >>>
> >>>
> >>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju-dev/attachments/20140909/e6c3fcef/attachment-0001.html>