[Bug 1268326] Re: locate spawns very CPU hungry find process

[Menachem Shapiro]

Related: 
https://bugs.launchpad.net/ubuntu/+source/findutils/+bug/332790
https://bugs.launchpad.net/ubuntu/+source/mlocate/+bug/1190696

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to findutils in Ubuntu.
https://bugs.launchpad.net/bugs/1268326

Title:
  locate spawns very CPU hungry find process

Status in “findutils” package in Ubuntu:
  Confirmed

Bug description:
  /etc/cron.daily/locate job spawns find process which very aggressively
  uses CPU. Full scan of filesystem of HDD on my laptop takes about 40
  minutes and almost all of that time one core is 100% occupied by find.

  Here is the command that runs

  /usr/bin/find / -ignore_readdir_race      \( -fstype NFS -o -fstype
  nfs -o -fstype nfs4 -o -fstype afs -o -fstype binfmt_misc -o -fstype
  proc -o -fstype smbfs -o -fstype autofs -o -fstype iso9660 -o -fstype
  ncpfs -o -fstype coda -o -fstype devpts -o -fstype ftpfs -o -fstype
  devfs -o -fstype mfs -o -fstype shfs -o -fstype sysfs -o -fstype cifs
  -o -fstype lustre_lite -o -fstype tmpfs -o -fstype usbfs -o -fstype
  udf -o -fstype ocfs2 -o      -type d -regex
  '\(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/afs$\)\|\(^/amd$\)\|\(^/alex$\)\|\(^/var/spool$\)\|\(^/sfs$\)\|\(^/media$\)\|\(^/var/lib/schroot/mount$\)'
  \) -prune -o -print0

  After brief investigation I found that CPU strain is produced by
  filesystem type checks. For example, here is original command
  benchmark (with additionally excluded /home/snapshots directory which
  contains dozens of BTRFS snapshots with too many files):

  time sudo /usr/bin/find / -ignore_readdir_race      \( -fstype NFS -o -fstype nfs -o -fstype nfs4 -o -fstype afs -o -fstype binfmt_misc -o -fstype proc -o -fstype smbfs -o -fstype autofs -o -fstype iso9660 -o -fstype ncpfs -o -fstype coda -o -fstype devpts -o -fstype ftpfs -o -fstype devfs -o -fstype mfs -o -fstype shfs -o -fstype sysfs -o -fstype cifs -o -fstype lustre_lite -o -fstype tmpfs -o -fstype usbfs -o -fstype udf -o -fstype ocfs2 -o      -type d -regex '\(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/afs$\)\|\(^/amd$\)\|\(^/alex$\)\|\(^/var/spool$\)\|\(^/sfs$\)\|\(^/media$\)\|\(^/var/lib/schroot/mount$\)\|\(^/home/snapshots$\)' \) -prune -o -print0 | pv > /dev/null
  57,1MB 0:13:17 [73,3kB/s] [     <=>                                                                                                                                    ]

  real	13m17.647s
  user	6m2.983s
  sys	6m41.837s

  And here is benchmark of command with severely shortened filesystem
  check list:

  time sudo /usr/bin/find / -ignore_readdir_race      \( -fstype afs -o -fstype binfmt_misc -o -fstype proc -o -fstype autofs -o -fstype devpts -o -fstype devfs -o -fstype sysfs -o -fstype tmpfs -o -fstype usbfs -o      -type d -regex '\(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/afs$\)\|\(^/amd$\)\|\(^/alex$\)\|\(^/var/spool$\)\|\(^/sfs$\)\|\(^/media$\)\|\(^/var/lib/schroot/mount$\)\|\(^/home/snapshots$\)' \) -prune -o -print0 | pv > /dev/null
  57,1MB 0:05:04 [ 192kB/s] [             <=>                                                                                                                            ]

  real	5m4.092s
  user	2m24.112s
  sys	2m39.951s

  Each excluded filesystem greatly speeds up the process.

  This is really annoying, as somehow daily cron jobs sometimes start
  even if my laptop is running on battery o_O (AFAIK anacron avoids
  running periodic jobs ).

  I checked the same find command on onther PC with arch linux, and it
  used less than 10 seconds of CPU time with comparable number of files,
  so it is possible, that something is wrong with ubuntu find version or
  somehow related to ubuntu VFS specifics.

  Also locate may check mounted filesystem and do not include unused
  filesystems when compiling command for find.

  ProblemType: Bug
  DistroRelease: Ubuntu 13.10
  Package: locate 4.4.2-6ubuntu1
  ProcVersionSignature: Ubuntu 3.11.0-15.23-generic 3.11.10
  Uname: Linux 3.11.0-15-generic x86_64
  ApportVersion: 2.12.5-0ubuntu2.2
  Architecture: amd64
  Date: Sun Jan 12 17:27:24 2014
  Dependencies:
   findutils 4.4.2-6ubuntu1
   gcc-4.8-base 4.8.1-10ubuntu9
   libc6 2.17-93ubuntu4
   libgcc1 1:4.8.1-10ubuntu9
   multiarch-support 2.17-93ubuntu4
  InstallationDate: Installed on 2012-07-01 (559 days ago)
  InstallationMedia: Xubuntu 12.04 LTS "Precise Pangolin" - Release amd64 (20120425)
  MarkForUpload: True
  SourcePackage: findutils
  UpgradeStatus: Upgraded to saucy on 2014-01-02 (10 days ago)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/findutils/+bug/1268326/+subscriptions