[Bug 996151] Re: disable apt http pipelining in quantal

[David Kalnischkies]

> A much better suggestion would be to detect a proxy during install
(most proxies add headers to their reply) and drop a pipeline depth
config into /etc/apt/apt.conf.d/ ... Or perhaps not pipeline the first
request and look for a proxy in the reply, then turn it on.

Its a bit unfair to disable it for all proxies (and as said I am not
even sure squid is still fully buggy) - and after all S3 and the other
"dumb storage" entities which triggered this are not proxies, so we
would gain nothing – and they seem to be in no way interested to follow
a standard because they are "the cloud" and "the cloud rulez": "Hey,
there is a + in the URI, that must be a space!" …

Looking just at the first response from the server is not going to work
as we are frequently lucky enough to get the expected response just by
chance. I can find frequent reports that reducing the Pipeline-Depth to
2 makes it work "all the time" (which is not too surprising, but still
fails once in a blue moon).

I guess our only option is to check the hashsum of the file in the
methods already (we calculate them there, so we "just" have to tell them
what to expect), so that on a mismatch we can see if another file we
requested in the pipeline matches the hashsum and if so do a data
shuffle and disable pipelining for this server in this run (in the hope
that it will be fixed soon - or in fear that a different one will answer
our request next time).

That is quiet a bit of work though and not only on the code, but also on
the test front – and has very ugly corner cases (partial downloads, …)
just to work around the problems in "the cloud" … (I should just have
grown some balls and tell the world "<censored>!! Fix Pipelining
<censored>!" I guess, but I am so weak, I even censor myself …)

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/996151

Title:
  disable apt http pipelining in quantal

Status in “apt” package in Ubuntu:
  Fix Released
Status in “apt” package in Debian:
  New

Bug description:
  Per UDS session on Apt improvements, it has been proposed to remove
  apt http pipelining

  The reasons:
  1. HTTP Pipelining has issue with certain proxy implementation
  2. Some new object stores, like S3, or Google's APT repositories have problems with HTTP Pipelining

  Running a test shows that disabling apt-pipelining has no perceptable
  diffferenvce, and disabling apt pipeling actually performed slightly
  better with an average of 31.899s versuses 32.456s. I tested an "apt-
  get -y update" with and without apt HTTP pipelining turned on.

  For more information on apt-pipelining, here are 2 threads to read:
   http://old.nabble.com/APT-do-not-work-with-Squid-as-a-proxy-because-of-pipelining-default-td28579596.html
   http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=565555

  
  Pipelining on (apt-get -y upgrade):
  33.92
  31.37
  31.64
  31.63
  33.29
  33.08
  32.92
  32.88
  31.73
  31.98
  32.01
  32.96
  31.51
  32.68
  33.25

  Pipelining off (apt-get -o Acquire::http::Pipeline-Depth="0" -y upgrade):
  31.66
  31.59
  31.24
  31.30
  31.29
  32.85
  32.75
  31.50
  31.18
  32.26
  31.43
  33.28
  31.67
  32.45
  32.04

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/996151/+subscriptions