[Bug 1013446] Re: Uncached grp and pwd calls make duplicity slow with large group and passwd maps

[Brian Murray]

Hello Steve, or anyone else affected,

Accepted duplicity into precise-proposed. The package will build now and
be available at
http://launchpad.net/ubuntu/+source/duplicity/0.6.18-0ubuntu3.1 in a few
hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
enable and use -proposed.  Your feedback will aid us getting this update
out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-needed to verification-done. If it does not fix the
bug for you, please add a comment stating that, and change the tag to
verification-failed.  In either case, details of your testing will help
us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: duplicity (Ubuntu Precise)
       Status: In Progress => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1013446

Title:
  Uncached grp and pwd calls make duplicity slow with large group and
  passwd maps

Status in Duplicity - Bandwidth Efficient Encrypted Backup:
  Fix Committed
Status in “duplicity” package in Ubuntu:
  Fix Released
Status in “duplicity” source package in Precise:
  Fix Committed
Status in “duplicity” source package in Quantal:
  Fix Committed

Bug description:
  With large /etc/group and /etc/passwd files, duplicity spends the vast
  majority of its time mapping group and user names to gids and uids.
  This is with duplicity 0.6.18-0ubuntu3 on Ubuntu 12.04 and Python
  2.7.3.

  I have a largish group map (7MB, with many group entries having lines
  in the 100KB - 200KB range.)  A zero change incremental backup of
  ~200k files takes about 23 minutes.

  Profiling the duplicity run shows that 92% of the time is spent
  calling grp.getgrnam!  Another ~2% is spent calling grp.getgrgid,
  pwd.getpwnam, and pwd.getpwuid.

  Some simple caching of caching of this information would make a big
  improvement.

  --

  SRU Justification:
  [Impact] 
   * This fix should be backported into Precise LTS because users of duplicity with large group/passwd files will encounter performance issues.

  [Test Case]
   * Using a large group/passwd file configuration, perform a backup and record the time it takes to backup. Profile this as well and determine how much time is spent in grp.getgrnam. Compare this with the patched version and ensure improvement in performance.

  [Regression Potential] 
   * This is a backported patch, and is currently in the latest duplicity. However; because it is a new feature we will need to ensure backups work as expected with this patch.

To manage notifications about this bug go to:
https://bugs.launchpad.net/duplicity/+bug/1013446/+subscriptions