[MERGE] [BUG #165061] Force http.readv() to a maximum number of range requests.

Vincent Ladeuil v.ladeuil+lp at free.fr
Thu Nov 29 06:54:12 GMT 2007


>>>>> "robert" == Robert Collins <robertc at robertcollins.net> writes:

    robert> Robert Collins has voted resubmit.
    robert> Status is now: Resubmit
    robert> Comment:
    robert> I think its better to make many requests that to ask for more
    robert> data than the fudge factor requires.

Agreed.

    robert> That is:
    robert> http._readv
    robert> for group in xrange(len(offsets), max_ranges):
    robert>     for result in self._do_readv(offsets[group *
    robert> max_ranges:(group+1 * max_ranges)):
    robert>          yield result

Except two details invalidate this simple solution:

- out of order offsets (but jam pointed me to sftp to handle
  that, with a risk of buffering too much but the consensus is 
  that packs should not suffer from that), 

- transient errors handling (this bug showed us that the whole
  file can and will be downloaded which should avoided when
  possible).

So yes, your proposal is the right direction and I'll try to
implement it in one or two steps:

- reuse the sftp_readv with slight adaptations*,

- rewrite the http response handling to deliver data as soon as
  they arrive. This is needed and will massively reduce the
  memory consumption.

I don't have a lot a free time these days so I'll look at
implementing the first step quickly.

             Vincent

*: needs adaptation since after aggregation the big chunks are
 split again to accommodate sftp limits.



More information about the bazaar mailing list