[Bug 1013446] Re: Uncached grp and pwd calls make duplicity slow with large group and passwd maps

[Michael Terry]

Shouldn't this fix be proposed upstream to python?

Before this change, we had an exact copy of python 2.7's tarfile.py
(with some changes to support python 2.4).  Now we can't ever go back to
a pure upstream tarfile.py.

And since the change seems to make sense for python itself too...

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1013446

Title:
  Uncached grp and pwd calls make duplicity slow with large group and
  passwd maps

Status in Duplicity - Bandwidth Efficient Encrypted Backup:
  Fix Committed
Status in “duplicity” package in Ubuntu:
  In Progress
Status in “duplicity” source package in Precise:
  In Progress

Bug description:
  With large /etc/group and /etc/passwd files, duplicity spends the vast
  majority of its time mapping group and user names to gids and uids.
  This is with duplicity 0.6.18-0ubuntu3 on Ubuntu 12.04 and Python
  2.7.3.

  I have a largish group map (7MB, with many group entries having lines
  in the 100KB - 200KB range.)  A zero change incremental backup of
  ~200k files takes about 23 minutes.

  Profiling the duplicity run shows that 92% of the time is spent
  calling grp.getgrnam!  Another ~2% is spent calling grp.getgrgid,
  pwd.getpwnam, and pwd.getpwuid.

  Some simple caching of caching of this information would make a big
  improvement.

  --

  SRU Justification:
  [Impact] 
   * This fix should be backported into Precise LTS because users of duplicity with large group/passwd files will encounter performance issues.

  [Test Case]
   * Using a large group/passwd file configuration, perform a backup and record the time it takes to backup. Profile this as well and determine how much time is spent in grp.getgrnam. Compare this with the patched version and ensure improvement in performance.

  [Regression Potential] 
   * This is a backported patch, and is currently in the latest duplicity. However; because it is a new feature we will need to ensure backups work as expected with this patch.

To manage notifications about this bug go to:
https://bugs.launchpad.net/duplicity/+bug/1013446/+subscriptions