[Merge] lp:~sjakthol/update-manager/fix-slow-calculation into lp:update-manager

[Sami Jaktholm]

Sami Jaktholm has proposed merging lp:~sjakthol/update-manager/fix-slow-calculation into lp:update-manager.

Commit message:
Implement various solutions to speed up the group calculation.

1. Create a set of recursive dependencies for each app group.
   The previous implementation traversed the dependency tree until a dependency
   was found (if there was one) and repeated the same procedure for every
   package. Software Updater spent most of it's time doing the same thing again
   and again.

   This branch traverses the dependency tree once and stores the data in a set
   (for automatic deduplication). The cache is initiated during first call to
   is_dependency and expanded with new items as new packages are added to the
   group. Once the cache is created, it's just a matter of simple "pkg in deps"
   test to see if the package is a dependency of an application in that group.

2. Don't glob /usr/share/app-install/desktop/ for every package.
   Instead, glob the directory once and save the URIs of desktop files that
   have an upgradeable package associated with them. These URIs are later added
   to the list of desktop_files in _get_application_for_package.

3. Cache package dependencies.
   When the package dependencies are calculated for the first time, the data is
   saved to a dict shared between groups. The cached data is used in the
   consequent queries for that same package (in other app groups). It's faster
   to fetch a list of strings from a dict than it's to iterate Dependency
   objects to get the package names.

4. Speed up _file_is_application.
   Make _file_is_application to return false right away if the file name 
   doesn't have a .desktop extension.

   The old version was spending a lot of time checking whether file was located
   in a known application directory and then returned false because it didn't
   have a .desktop extension. As this is called for every file in every
   upgradeable package, the performance impact was substantial.

5. Don't spin main loop on every call to _add_deps.
   _add_deps is invoked thousands of times in a few seconds and the UI
   doesn't need to be updated every time. As _add_deps doesn't take ages to
   complete it's enough to spin the loop once in a while. That'll be enough to
   keep the UI alive.

6. Remove useless sorting by source package in _make_groups.
   The list of packages was first sorted by source but that information was
   never used. The useless nested loops were removed.

Requested reviews:
  Ubuntu Core Development Team (ubuntu-core-dev)
Related bugs:
  Bug #1167277 in update-manager (Ubuntu): "Unresponsive for a long time after checking for updates"
  https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/1167277

For more details, see:
https://code.launchpad.net/~sjakthol/update-manager/fix-slow-calculation/+merge/161284

The tests are still passing and the packages are grouped the same way as they are now.

Inside VirtualBox it took nearly five minutes to calculate groups for 269 updates. This version was able to do it in 0.67 seconds (varied between 0.67 and 2 second for a cold start). 

Without VirtualBox overhead the time it took to calculate groups for 308 upgrades went from 2 minutes to 0.75 seconds. However, this was measured before some final changes were made and I don't have access to that system anymore. Inside VirtualBox the changes shaved nearly 1 second off the group calculation time so those should also cut that 0.75 seconds down considerably.

The total time for update calculation is still long, but group calculation shouldn't be a problem anymore. This version spends most of its time opening the cache, which is a different story...
-- 
https://code.launchpad.net/~sjakthol/update-manager/fix-slow-calculation/+merge/161284
Your team Ubuntu Core Development Team is requested to review the proposed merge of lp:~sjakthol/update-manager/fix-slow-calculation into lp:update-manager.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: review-diff.txt
Type: text/x-diff
Size: 12786 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-reviews/attachments/20130427/36b4e277/attachment-0001.diff>