Cleansing Mongo data

[Katherine Cox-Buday]

I didn't understand at first either. The code has a comment[1] with a link providing justification[2].

TLDR: There are 2 issues:

1. Something in our stack juju->mgo->mongo (it hasn't been explained to me what is doing this) rewrites keys which contain dots into new nested structures. E.g.: {foo.bar: baz} -> {foo: {bar: baz}}.
2. Storing keys which begin with $ signs can lead to injection attacks if you're using that data while executing these statements: {$where, mapReduce, group}. This is better explained in [2].

I hope that helps. Please, keep asking questions until this is better understood. It looks like a huge blind-spot for most of our developers.

[1] - https://github.com/juju/juju/blob/master/mongo/utils/data_cleansing.go#L31-L32
[2] - http://docs.mongodb.org/manual/faq/developers/#faq-dollar-sign-escaping

Matthew Williams <matthew.williams at canonical.com> writes:

> I seem to be missing something. Why do we need this?
>
> Matty
>
> On 24 Jun 2016 17:14, "Nate Finch" <nate.finch at canonical.com> wrote:
>
>     It seems as though we should be cleansing all the keys.... since
>     we never know what queries we might want to make in the future. 
>
>     
>     
>     On Fri, Jun 24, 2016 at 12:04 PM Katherine Cox-Buday
>     <katherine.cox-buday at canonical.com> wrote:
>     
>     
>         As I have only just discovered the need to cleanse mongo data,
>         I can't say for sure, but it looks like we may have been
>         cleansing things in the parts of Juju that need it. William
>         may know more.
>         
>         If not, I imagine a small upgrade step would make short work
>         of any problems.
>         
>         roger peppe <roger.peppe at canonical.com> writes:
>         
>         > This is useful, thanks.
>         >
>         > Note that's it's not necessary to cleanse *all* keys that go
>         into Mongo,
>         > just the ones that might be used in queries.
>         >
>         > But one thought... what about keys that already contain
>         full-width
>         > dollar and dot?
>         >
>         > cheers,
>         > rog.
>         >
>         > On 23 June 2016 at 21:09, Katherine Cox-Buday
>         > <katherine.cox-buday at canonical.com> wrote:
>         >> Hey all,
>         >>
>         >> William gave me a good review and it came up that I wasn't
>         cleansing
>         >> some of
>         >> the data being placed in Mongo. I wasn't aware this had to
>         be done,
>         >> and
>         >> after talking to a few other folks it became apparent that
>         maybe not
>         >> many
>         >> people know we should be doing this.
>         >>
>         >> At any rate, William also pointed me to some existing code
>         which did
>         >> this.
>         >> I've pulled it out into the mongo/utils package for general
>         >> consumption. The
>         >> comments do a pretty good job of elucidating why this is
>         necessary.
>         >>
>         >>
>         https://github.com/juju/juju/blob/master/mongo/utils/data_cleansing.go
>        
>         >>
>         >> -
>         >> Katherine
>         >>
>         >> --
>         >> Juju-dev mailing list
>         >> Juju-dev at lists.ubuntu.com
>         >> Modify settings or unsubscribe at:
>         >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>         >>
>         
>         --
>         Katherine
>         
>         --
>         Juju-dev mailing list
>         Juju-dev at lists.ubuntu.com
>         Modify settings or unsubscribe at:
>         https://lists.ubuntu.com/mailman/listinfo/juju-dev
>         
>
>     --
>     Juju-dev mailing list
>     Juju-dev at lists.ubuntu.com
>     Modify settings or unsubscribe at:
>     https://lists.ubuntu.com/mailman/listinfo/juju-dev
>     
>     
>
>

-- 
Katherine