[MERGE] [BUG #165061] Force http.readv() to a maximum number of range requests.
Vincent Ladeuil
v.ladeuil+lp at free.fr
Thu Nov 29 06:54:12 GMT 2007
>>>>> "robert" == Robert Collins <robertc at robertcollins.net> writes:
robert> Robert Collins has voted resubmit.
robert> Status is now: Resubmit
robert> Comment:
robert> I think its better to make many requests that to ask for more
robert> data than the fudge factor requires.
Agreed.
robert> That is:
robert> http._readv
robert> for group in xrange(len(offsets), max_ranges):
robert> for result in self._do_readv(offsets[group *
robert> max_ranges:(group+1 * max_ranges)):
robert> yield result
Except two details invalidate this simple solution:
- out of order offsets (but jam pointed me to sftp to handle
that, with a risk of buffering too much but the consensus is
that packs should not suffer from that),
- transient errors handling (this bug showed us that the whole
file can and will be downloaded which should avoided when
possible).
So yes, your proposal is the right direction and I'll try to
implement it in one or two steps:
- reuse the sftp_readv with slight adaptations*,
- rewrite the http response handling to deliver data as soon as
they arrive. This is needed and will massively reduce the
memory consumption.
I don't have a lot a free time these days so I'll look at
implementing the first step quickly.
Vincent
*: needs adaptation since after aggregation the big chunks are
split again to accommodate sftp limits.
More information about the bazaar
mailing list