[Merge] lp:~cjwatson/ubuntu-archive-publishing/parallel-germinate into lp:ubuntu-archive-publishing

[Colin Watson]

Colin Watson has proposed merging lp:~cjwatson/ubuntu-archive-publishing/parallel-germinate into lp:ubuntu-archive-publishing.

Commit message:
Parallelise generate-extra-overrides by architecture, saving about two minutes off each publisher run.

Requested reviews:
  William Grant (wgrant)
  Ubuntu Package Archive Administrators (ubuntu-archive)
Related bugs:
  Bug #1248867 in ubuntu-archive-publishing: "Investigate parallelising cron.germinate by arch"
  https://bugs.launchpad.net/ubuntu-archive-publishing/+bug/1248867

For more details, see:
https://code.launchpad.net/~cjwatson/ubuntu-archive-publishing/parallel-germinate/+merge/222607

Parallelise generate-extra-overrides by architecture, saving about two minutes off each publisher run.

I considered using the multiprocessing module, but gave up after six hours of fighting with obscure errors; it's simple enough to do by hand.  Note that attempts to do this using threading instead are unlikely to scale well, as germinate is CPU-heavy and Python threads all have to share the GIL.

(This is ported from https://code.launchpad.net/~cjwatson/launchpad/parallel-germinate/+merge/220610, after extracting cron.germinate and friends from Launchpad.)
-- 
https://code.launchpad.net/~cjwatson/ubuntu-archive-publishing/parallel-germinate/+merge/222607
Your team Ubuntu Package Archive Administrators is requested to review the proposed merge of lp:~cjwatson/ubuntu-archive-publishing/parallel-germinate into lp:ubuntu-archive-publishing.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: review-diff.txt
Type: text/x-diff
Size: 6676 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-archive/attachments/20140610/72cf9ef0/attachment.diff>