[Bug 1433924] Re: "Too much data for sort() with no index." While Getting Resources

[James Page]

b3 has been uploaded to wily - setting to Fix Released.

** Also affects: ceilometer (Ubuntu Wily)
   Importance: Undecided
       Status: New

** Changed in: ceilometer (Ubuntu Wily)
   Importance: Undecided => Medium

** Also affects: ceilometer (Ubuntu Vivid)
   Importance: Undecided
       Status: New

** Also affects: ceilometer (Ubuntu Trusty)
   Importance: Undecided
       Status: New

** Changed in: ceilometer (Ubuntu Wily)
       Status: New => Fix Released

** Changed in: ceilometer (Ubuntu Vivid)
       Status: New => Triaged

** Changed in: ceilometer (Ubuntu Trusty)
       Status: New => Triaged

** Changed in: ceilometer (Ubuntu Trusty)
   Importance: Undecided => Medium

** Changed in: ceilometer (Ubuntu Vivid)
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to ceilometer in Ubuntu.
https://bugs.launchpad.net/bugs/1433924

Title:
  "Too much data for sort() with no index." While Getting Resources

Status in Ceilometer:
  Fix Released
Status in ceilometer package in Ubuntu:
  Fix Released
Status in ceilometer source package in Trusty:
  Triaged
Status in ceilometer source package in Vivid:
  Triaged
Status in ceilometer source package in Wily:
  Fix Released

Bug description:
  We have 47GB of mongo data. When I run "ceilometer resource-list", I
  get the error in the subject. I isolated the problem down to
  storage/impl_mongodb.py:746 _get_floating_resources(). It builds an
  empty query with sort options "{user_id: -1, project_id: -1,
  last_sample_timestamp: -1}" and it fails.

  I ran the query in mongodb to see if it was generating the same error
  and confirmed it. After a bit digging, I learned that mongodb needs
  compound index for that kind of sort. I added the following index via
  mongocl:

  db.resource.createIndex({user_id: -1, project_id: -1,
  last_sample_timestamp: -1}, {background: true})

  After this index, the problem went away. I can run "ceilometer
  resource-list" and "ceilometer meter-list". However, those commands
  take ~1minute to complete with the index. The performance may be
  another issue that needs investigation.

  I need to mention that this error is different than
  https://bugs.launchpad.net/ceilometer/+bug/1193906. The timestamp
  index is in place but we need compound indexes as I explained above.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ceilometer/+bug/1433924/+subscriptions