[Bug 1240652] Re: maas-import-ephemerals crashes with "unexpected checksum 'sha256'" when using a proxy

[Scott Moser]

This bug is believed to be fixed in simplestreams in version 0.1.0. If
this is still a problem for you, please make a comment and set the state
back to New

Thank you.

** Changed in: simplestreams
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1240652

Title:
  maas-import-ephemerals crashes with "unexpected checksum 'sha256'"
  when using a proxy

Status in Ubuntu Cloud Archive:
  Fix Released
Status in MAAS:
  Invalid
Status in simplestreams:
  Fix Released
Status in simplestreams package in Ubuntu:
  Fix Released
Status in simplestreams source package in Saucy:
  Fix Released
Status in simplestreams source package in Trusty:
  Fix Released

Bug description:
  == Begin SRU Info ==
  [Impact]
  maas-import-ephemerals is the program that is executed to download images from http://maas.ubuntu.com which are then used for the commissioning, enlistment, and curtin install environment.

  This bug causes that download to fail in most cases with an http proxy
  between the user and maas.ubuntu.com.  It is also reported to fail
  under cases where there is no http_proxy set (but may be a proxy in
  between).

  [Test Case]
  The bug could be reproduced by simply doing:
    PROXY=http://localhost:3128 # or some valid proxy
    http_proxy=$PROXY python /usr/bin/sstream-mirror -vvv http://maas.ubuntu.com/images/ephemeral/daily/streams/v1/index.json out.d arch=amd64 release=quantal

  You would see the download start and then hang.  A suitable proxy can
  be installed just by 'apt-get install squid3' and then using
  http://localhost:3128.

  [Regression Potential]
  Regression potential should be very small.  Previously each connection we made was doing an open and close and then open again.  This was wreaking havoc with urllib3 as used by python-requests.  Now, we're only opening and closing a single time during the mirror, and even then,
   a.) the url that is being downloaded will change (by having a '/' inserted into its path) where previously it would be the same url
   b.) step 'a' will only occur if the base url provided by the user did not already end in '/'.
  == End SRU Info ==

  Since 2013-10-11 we started seeing errors running the MAAS integration
  tests for saucy[1].

  The checksum of the downloaded image wasn't matching the checksum
  expected[2]. maas-import-pxe-files failed with the following
  traceback:

  Traceback (most recent call last):
    File "/usr/sbin/maas-import-ephemerals", line 26, in <module>
      main(args)
    File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/ephemerals_script.py", line 450, in main
      target.sync(source, args.path)
    File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 85, in sync
      return self.sync_index(reader, path, data, content)
    File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 230, in sync_index
      self.sync(reader, path=epath)
    File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 83, in sync
      return self.sync_products(reader, path, data, content)
    File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 304, in sync_products
      self.insert_item(item, src, target, pgree, ipath_cs)
    File "/usr/lib/python2.7/dist-packages/provisioningserver/import_images/ephemerals_script.py", line 249, in insert_item
      data, src, target, pedigree, contentsource)
    File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 388, in insert_item
      size=data.get('size'))
    File "/usr/lib/python2.7/dist-packages/simplestreams/objectstores/__init__.py", line 147, in insert
      cksum.hexdigest(), cksum.expected))
  Exception: unexpected checksum 'sha256' on precise/release-20131010/precise-daily-maas-amd64.tar.gz (found: 3d99e069e34d031f924689a3c28ea43d4046d8e9a79a2ea9014d35bcbcf57a16 expected: 2c7ab801ffd7e8673b77be828335bac29538f102dbc6baacd4ccc4c0d309a11b)
  }}}

  Initially it was thought this could be caused by a misconfigured cache
  in the QA lab, but after re-installing and reconfiguring (and checking
  with wget that the cache was returning the correct objects for a given
  URL, as well as checking the checksum of the file matched the one in
  the json file), the error was still present.

  Further investigation[3] raised the suspicion that the downloaded
  image is being handled by multiple processes. Turns out the code for
  simplestreams and import_ephemerals.py is not multi process safe, and
  multiple import-ephemerals processes working with the same datadir can
  cause problems like this.

  [1] http://10.189.74.2:8080/view/MAAS/job/saucy-adt-maas-daily/220/ARCH=amd64,label=lenovo-RD230-01/console
  [2] http://10.189.74.2:8080/view/MAAS/job/saucy-adt-maas-manual/95/ARCH=amd64,label=lenovo-RD230-01/console
  [3] http://pastebin.ubuntu.com/6247125/

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1240652/+subscriptions