[Bug 1794957] Re: pipelining on archive.u.c aborts after 101 packages

[Łukasz Zemczak]

Hello Julian, or anyone else affected,

Accepted apt into xenial-proposed. The package will build now and be
available at https://launchpad.net/ubuntu/+source/apt/1.2.28 in a few
hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-xenial to verification-done-xenial. If it does not
fix the bug for you, please add a comment stating that, and change the
tag to verification-failed-xenial. In either case, without details of
your testing we will not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: apt (Ubuntu Xenial)
       Status: Triaged => Fix Committed

** Tags added: verification-needed-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/1794957

Title:
  pipelining on archive.u.c aborts after 101 packages

Status in apt package in Ubuntu:
  Fix Released
Status in apt source package in Xenial:
  Fix Committed
Status in apt source package in Bionic:
  Fix Committed

Bug description:
  [Impact]
  Downloading many packages on archive.ubuntu.com or some other mirrors seems to close the connection after every 100 or so packages. APT prior to 1.7.0~rc1 (commit df696650b7a8c58bbd92e0e1619e956f21010a96), treats a connection closure with a 200 response as meaning that the server does not support pipelining, hence disabling it for any further downloads.

  With high speed connections at higher latency, this can cause a severe
  reduction in usable bandwidth. For example, I saw speeds drop from 40
  MB/s to 15 MB/s due to this.

  The fix ensures that we continue pipelining if the previous connection
  to the server successfully retrieved at least 3 files with pipelining
  enabled.

  [Test case]
  Pick a package that would cause a large number (200/300 packages of packages to be installed). I used plasma-desktop and xubuntu-desktop, for example. Run apt install -d $package. Ensure that after the first 101 packages the progress does not slow down - you should not see a lot "working" in the progress output. The speed should be substantially higher.

  Don't run in a container setup by cloud-init, as cloud-init disables
  pipelining; or remove /etc/apt/apt.conf.d/90cloud-init-pipelining (see
  bug 1794982).

  Requirements:
  * High speed, medium-high latency connection (e.g. 400 Mbit/s at 30 ms RTT is enough)
  * Not a terribly slow CPU, as we'd get slowed down by hashing otherwise

  [Regression potential]
  This fix is isolated to code enabling/disabling pipelining on subsequent connections. It could cause more pipelining to be tried on servers who are not particularly good at it, but can deal with 3 items correctly.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1794957/+subscriptions