[Bug 1067934] Re: spends 10+ minutes deduplicating Package lists

[Stuart Longland]

That seems to have fixed it at long last.  We've got a rather
complicated boot menu system so out of those netboot tarballs, I just
grab initrd.gz and linux files, and place them in a common directory
tree so we can deploy various i686 and AMD64 distributions by picking
them from a menu.

The broken files had the checksums:
59fcdc4f29fdc3ff08d24b96ef28151f3988a9ff  initrd.gz
fba53798707c7278334a5402453586bed216de20  linux

The working ones had the checksums:
54c80dd94b0f22f4a51df70502bcfff7c29d42b4  initrd.gz
76a613634ff1c595b2e279af99bd88bff4a46404  linux

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to net-retriever in Ubuntu.
https://bugs.launchpad.net/bugs/1067934

Title:
  spends 10+ minutes deduplicating Package lists

Status in “net-retriever” package in Ubuntu:
  Fix Released
Status in “net-retriever” source package in Lucid:
  Fix Released
Status in “net-retriever” source package in Precise:
  Fix Released
Status in “net-retriever” source package in Quantal:
  Won't Fix
Status in “net-retriever” source package in Raring:
  Won't Fix
Status in “net-retriever” source package in Saucy:
  Fix Released

Bug description:
  [Impact] The netboot installer is unreasonably slow to deduplicate udeb Packages files retrieved from the network, scaling (at least) quadratically in the number of packages with a bad constant factor.
  [Test Case] Run a netboot installer built with the new net-retriever (we'll put one in -proposed at some point after this SRU is reviewed and accepted, and post the URLs to this bug).  It should work and should not take unreasonable time around the "Download installer components" stage.
  [Regression Potential] Cannot affect anything other than netboot installations.  I've compared the output against the old code for a sample Packages file provided by Michael and made sure it's identical; but if this goes wrong then the most likely result is a partitioning failure, or perhaps a complaint that no suitable kernel versions are available (although this can happen for other reasons, e.g. forgetting "apt-setup/proposed=true" for an image built against the kernel in -proposed).

  Original report follows;

  At some point after Lucid's initial release, the ubuntu alternate
  installer seems to stall for a very long time (10+ minutes) before the
  partitioning menu comes up. I don't think it was happening for the
  first year or so after Lucid's release, but I'm not real clear on when
  it started happening (it was only recently brought to my attention).

  I think it is happening in /usr/lib/debian-installer/net-retriever's deduplicate function...
  When it is stalled out, it keeps adding content to /tmp/net-retriever-NNNN-deduplicate/, eventually it moves on, but it takes forever. The speed of the install machine doesn't appear to matter much.
  There's nothing obvious in the log as to why it's stalling at that point...

  I thought it might be some issue with our preseeding, but it appears
  to happen on an install without preseeding as well.

  Newer versions of Ubuntu (at least Xubuntu 12.04) don't seem to
  experience this problem, as far as I can tell.

  We're using the netboot installer, apparently last updated on
  2011-04-29:

  2011-04-29 03:32:27-07:00 checking for updates: http://us.archive.ubuntu.com/ubuntu/dists/lucid/main/installer-i386/current/images/netboot/netboot.tar.gz
  2011-04-29 03:33:13-07:00 updated: /var/lib/tftpboot/ubuntu-installer/lucid.i386.netboot.tar.gz

  though as far as i can tell, that file hasn't been updated on the
  mirrors since 2010-04-25.

  live well,
    vagrant

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/net-retriever/+bug/1067934/+subscriptions