[Bug 1934849] Re: s3 backend takes time exponentially

Tue Jul 27 16:33:24 UTC 2021

[focal-victoria]

All of the uploads succeeded. And -proposed shortened time for the
larger sizes.

$ sudo apt-get install python3-glance-store/focal-proposed
$ sudo systemctl restart glance-api

$ apt-cache policy python3-glance-store
python3-glance-store:
  Installed: 2.3.0-0ubuntu1~cloud1
  Candidate: 2.3.0-0ubuntu1~cloud1
  Version table:
 *** 2.3.0-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/victoria/main amd64 Packages
        100 /var/lib/dpkg/status
     2.3.0-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/victoria/main amd64 Packages
     2.0.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

size	ceph	swift	s3	s3 -proposed
0MiB	0m10.780s	0m2.762s	0m3.819s	0m3.235s
3MiB	0m3.334s	0m2.541s	0m3.019s	0m2.389s
5MiB	0m2.731s	0m2.244s	0m2.687s	0m2.417s
9MiB	0m2.725s	0m2.334s	0m2.243s	0m2.069s
10MiB	0m2.752s	0m2.315s	0m3.055s	0m2.120s
128MiB	0m4.404s	0m3.604s	0m56.566s	0m4.728s
512MiB	0m8.411s	0m6.021s	13m45.246s	0m17.453s
1024MiB	0m14.707s	0m11.430s	54m22.978s	0m55.937s

** Tags removed: verification-needed verification-victoria-needed
** Tags added: verification-done verification-victoria-done

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1934849

Title:
  s3 backend takes time exponentially

Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive ussuri series:
  Fix Committed
Status in Ubuntu Cloud Archive victoria series:
  Fix Committed
Status in Ubuntu Cloud Archive wallaby series:
  Fix Committed
Status in glance_store:
  Fix Released
Status in python-glance-store package in Ubuntu:
  Fix Released
Status in python-glance-store source package in Focal:
  Fix Committed
Status in python-glance-store source package in Hirsute:
  Fix Committed

Bug description:
  [Impact]

  Glance with S3 backend cannot accept image uploads in a realistic time
  frame. For example, an 1GB image upload takes ~60 minutes although other
  backends such as swift can complete it with 10 seconds.

  [Test Plan]

  1. Deploy a partial OpenStack with multiple Glance backends including S3
    (zaza test bundles can be used with "ceph" which will set up "rbd", "swift", and "s3" backends - https://opendev.org/openstack/charm-glance/src/branch/master/tests/tests.yaml)
  2. Upload multiple images with variety of sizes
  3. confirm the duration of uploading images are shorter in general after applying the updated package
    (expected duration of 1GB is from ~60 minutes to 1-3 minutes)

  for backend in ceph swift s3; do
      echo "[$backend]"
      for i in {0,3,5,9,10,128,512,1024}; do
          dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
          echo "${i}MiB"
          time glance image-create \
              --store $backend \
              --file my-image.img --name "my-image-${backend}-${i}MiB" \
              --disk-format raw --container-format bare \
              --progress
      done
  done

  [Where problems could occur]

  Since we bump WRITE_CHUNKSIZE from 64KiB to 5MiB, there might be a case where image uploads fail if the size of the image is less than WRITE_CHUNKSIZE. Or there might be an unexpected latency in the worst case scenario. We will try to address the concerns by testing multiple images uploads with multiple sizes including some corner cases as follows:
  - 0 - zero
  - 3MiB - less than the new WRITE_CHUNKSIZE(5MiB)
  - 5MiB - exactly same as the new WRITE_CHUNKSIZE(5MiB)
  - 9MiB - bigger than new WRITE_CHUNKSIZE(5MiB) but less than twice
  - 10MiB - exactly twice as the new WRITE_CHUNKSIZE(5MiB)
  - 128MiB, 512MiB, 1024MiB - some large images

  ====

  I have a test Ceph cluster as an object storage with both Swift and S3
  protocols enabled for Glance (Ussuri). When I use Swift backend with
  Glance, an image upload completes quickly enough. But with S3 backend
  Glance, it takes much more time to upload an image and it seems to rise
  exponentially.

  It's worth noting that when uploading an image with S3 backend, a single
  core is consumed 100% by glance-api process.

  [swift]
  8MB   -  2.4s
  16MB  -  2.8s
  32MB  -  2.6s
  64MB  -  2.7s
  128MB -  3.1s
  ...
  512MB -  5.9s

  [s3]
  8MB   -  2.2s
  16MB  -  2.9s
  32MB  -  5.5s
  64MB  - 16.3s
  128MB - 54.9s
  ...
  512MB - 14m26s

  Btw, downloading of 512MB image with S3 backend can complete with less
  than 10 seconds.

  $ time openstack image save --file downloaded.img 917c5424-4350-4bc5-98ca-66d40e101843
  real    0m5.673s

  $ du -h downloaded.img
  512M    downloaded.img

  [/etc/glance/glance-api.conf]

  enabled_backends = local:file, ceph:rbd, swift:swift, s3:s3

  [swift]
  auth_version = 3
  auth_address = http://192.168.151.131:5000/v3
  ...
  container = glance
  large_object_size = 5120
  large_object_chunk_size = 200

  [s3]
  s3_store_host = http://192.168.151.137:80/
  ...
  s3_store_bucket = zaza-glance-s3-test
  s3_store_large_object_size = 5120
  s3_store_large_object_chunk_size = 200

  ProblemType: Bug
  DistroRelease: Ubuntu 20.04
  ProblemType: BugDistroRelease: Ubuntu 20.04
  Package: python3-glance-store 2.0.0-0ubuntu1
  ProcVersionSignature: Ubuntu 5.4.0-77.86-generic 5.4.119
  Uname: Linux 5.4.0-77-generic x86_64
  ApportVersion: 2.20.11-0ubuntu27.18
  Architecture: amd64
  CasperMD5CheckResult: skip
  Date: Wed Jul  7 04:46:05 2021
  PackageArchitecture: all
  ProcEnviron:
   TERM=screen-256color
   PATH=(custom, no user)
   LANG=C.UTF-8
   SHELL=/bin/bash
  SourcePackage: python-glance-store
   SHELL=/bin/bashSourcePackage: python-glance-store
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions