[Bug 1834875] Re: cloud-init growpart race with udev

[Ryan Harper]

> it will prevent udevd from running the rules against it. Thus
effectively the event will be fired and done, but nothing actually
executed for it.

Interesting, I suspect this is the race we see.  The events emitted but
no actions taken (ie we didn't get our by-partuuid symlink created.

> I somehow wonder if we even need partx call, if we properly flock the
device and trigger udev after everything is done.

Growpart is modifying the partition table of the root device which is
already mounted.  We will not be able to flock the root device since
it's open and mounted.  This is precisely why we need to use the partx
call as it updates the kernel partition table mapping without requiring
an exclusive lock on the device.

> So does growpart create partition? move it? delete/recreate one? i.e
does ADD happen? or like REMOVE & ADD? or maybe it's like just MOVE or
CHANGE? Do we have logs of the emitted events already?

growpart on gpt, uses sgdisk which, deletes and then recreates the
existing partition but with a larger size.

The logs are above, see comment #22 -> #24 and #38.

> Don't like flags, as then we'll have to supported forever =) maybe env
variable? or like simply change in focal and compare focal vs eoan?

This may not matter if we can't flock.  I suspect we'll need to use the
ugly re-trigger events for the disk.  growpart could take the disk it's
pointed at, examine the existing udevadm symlinks associated with this
(via udevadm info --export), perform it's operation, and then post-
operation, trigger the add event on the disk, settle, and confirm that
udevadm info --export contains the same set of links (partuuid ,etc).

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1834875

Title:
  cloud-init growpart race with udev

Status in cloud-init:
  Incomplete
Status in cloud-utils:
  New
Status in linux-azure package in Ubuntu:
  New
Status in systemd package in Ubuntu:
  Incomplete

Bug description:
  On Azure, it happens regularly (20-30%), that cloud-init's growpart
  module fails to extend the partition to full size.

  Such as in this example:

  ========================================

  2019-06-28 12:24:18,666 - util.py[DEBUG]: Running command ['growpart', '--dry-run', '/dev/sda', '1'] with allowed return codes [0] (shell=False, capture=True)
  2019-06-28 12:24:19,157 - util.py[DEBUG]: Running command ['growpart', '/dev/sda', '1'] with allowed return codes [0] (shell=False, capture=True)
  2019-06-28 12:24:19,726 - util.py[DEBUG]: resize_devices took 1.075 seconds
  2019-06-28 12:24:19,726 - handlers.py[DEBUG]: finish: init-network/config-growpart: FAIL: running config-growpart with frequency always
  2019-06-28 12:24:19,727 - util.py[WARNING]: Running module growpart (<module 'cloudinit.config.cc_growpart' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py'>) failed
  2019-06-28 12:24:19,727 - util.py[DEBUG]: Running module growpart (<module 'cloudinit.config.cc_growpart' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py'>) failed
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 812, in _run_modules
      freq=freq)
    File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
      return self._runners.run(name, functor, args, freq, clear_on_fail)
    File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 187, in run
      results = functor(*args)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 351, in handle
      func=resize_devices, args=(resizer, devices))
    File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2521, in log_time
      ret = func(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 298, in resize_devices
      (old, new) = resizer.resize(disk, ptnum, blockdev)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 159, in resize
      return (before, get_size(partdev))
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 198, in get_size
      fd = os.open(filename, os.O_RDONLY)
  FileNotFoundError: [Errno 2] No such file or directory: '/dev/disk/by-partuuid/a5f2b49f-abd6-427f-bbc4-ba5559235cf3'

  ========================================

  @rcj suggested this is a race with udev. This seems to only happen on
  Cosmic and later.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1834875/+subscriptions