[Bug 1123272] Re: high cpu usage on download (not optimised SHA256, SHA512, SHA1)

[Julian Andres Klode]

Results from my 5 year old laptop (X230, Core i5-3320M with 2.6 GHz),
for a 1 GB file:

MD5+SHA1+SHA256: 11s => 90 MB/s => 727 Mbit/s
MD5+SHA1+SHA256+SHA512: 15s => 67 MB/s => 533 Mbit/s

That seems like an acceptable baseline performance. You are not likely
to hit GBs worth of packages, so you have like less than 10 seconds of
maximum CPU use (likely you'll hit 100-200 MB, so 2-3 seconds of CPU
burst). I think 2-3 seconds is completely acceptable.

With nettle which is hardware-accelerated, we achieve:

MD5+SHA1+SHA256: 9s => 111 MB/s => 888 Mbit/s
MD5+SHA1+SHA256+SHA512: 12.8s => 78 MB/s => 625 Mbit/s

(Tests might be skewed a bit in favor of apt, as it avoids duplicate
reads - I had to run multiple nettle-hash after each other, rather than
after each other per block, the overhead is about 0.2 / 0.4s)

The improvement does not seem huge.

Obviously, if you find faster implementations to contribute to nettle,
things might improve further.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/1123272

Title:
  high cpu usage on download (not optimised SHA256, SHA512, SHA1)

Status in apt package in Ubuntu:
  Confirmed

Bug description:
  http process takes about 20% on intel i5-3317. Here is output of ps aux:
  root      4309 21.7  0.0  37108  2344 pts/2    S+   18:17   0:01 /usr/lib/apt/methods/http

  it can be easy reproduces with "apt-get download firefox-dbg" or any
  other upgrade realted command.

  The reason of this  overhead are this functions:
  22,34%  http  libapt-pkg.so.4.12.0  [.] SHA256_Transform(_SHA256_CTX*, unsigned int const*)
  14,81%  http  libapt-pkg.so.4.12.0  [.] SHA512_Transform(_SHA512_CTX*, unsigned long const*)
  8,11%  http  libapt-pkg.so.4.12.0  [.] SHA1Transform(unsigned int*, unsigned char const*) 
  5,62%  http  libapt-pkg.so.4.12.0  [.] MD5Transform(unsigned int*, unsigned int const*)

  libapt uses own implementation of this hasches without any optimisation.  Are there any reason why libapt do not use
  openssl? Beside current version of openssl has AVX optimised SHA* implementations.

  ProblemType: Bug
  DistroRelease: Ubuntu 13.04
  Package: apt 0.9.7.7ubuntu1
  ProcVersionSignature: Ubuntu 3.8.0-4.9-generic 3.8.0-rc6
  Uname: Linux 3.8.0-4-generic x86_64
  ApportVersion: 2.8-0ubuntu4
  Architecture: amd64
  Date: Tue Feb 12 18:12:33 2013
  InstallationDate: Installed on 2012-09-13 (152 days ago)
  InstallationMedia: Ubuntu 12.10 "Quantal Quetzal" - Alpha amd64 (20120913.1)
  MarkForUpload: True
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=de_DE.UTF-8
   SHELL=/bin/bash
  SourcePackage: apt
  UpgradeStatus: Upgraded to raring on 2013-02-06 (5 days ago)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1123272/+subscriptions