[Bug 1934933] Re: cloud-init dhclient apparmor denied with noexec on /var/tmp

[Launchpad Bug Tracker]

Status changed to 'Confirmed' because the bug affects multiple users.

** Changed in: isc-dhcp (Ubuntu)
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to isc-dhcp in Ubuntu.
https://bugs.launchpad.net/bugs/1934933

Title:
  cloud-init dhclient apparmor denied with noexec on /var/tmp

Status in isc-dhcp package in Ubuntu:
  Confirmed

Bug description:
  Hello - we are seeing an issue on multiple Azure hosts where there is
  a long delay during bootup.  This appears to be related to an apparmor
  issue with dhclient executed via cloud-init when /var is mounted
  noexec.  Because /var is noexec, the original dhclient is executed
  rather than the copy in /var/tmp/cloud-init, which causes the AppArmor
  profile to be applied.

  This prevents the instance from being able to record the DHCP lease
  information to /var/tmp/cloud-init/cloud-init-dhcp-*, which prevents
  the instance from being able to obtain goalstate information, and with
  cloud-init 21.2-3 or later, results in an extended delay during boot
  due to a recent change in azure.py
  (https://github.com/canonical/cloud-init/pull/842).

  This issue does not occur in default Ubuntu installations (including
  the Ubuntu 20.04 default Azure image), as the dhcp.py script in cloud-
  init behaves differently, copying /usr/sbin/dhclient to
  /var/tmp/cloud-init/cloud-init-dhcp-xxxxx/dhclient when /var allows
  executables, and the apparmor profiles then do not apply to the copied
  executable.

  The syslog will show the following entry when the instance boots up: 
  cloud-init[820]: 2021-07-07 14:50:40,661 - dhcp.py[WARNING]: dhclient did not produce expected files: dhcp.leases, dhclient.pid

  The cloud-init.log file will show this entry when this issue is occurring. Since the instance has no IP address at this stage of the boot process, an unreachable network is to be expected:
  azure.py[DEBUG]: Failed HTTP request with Azure endpoint http://168.63.129.16/machine/?comp=goalstate during attempt 240 with exception: HTTPConnectionPool(host='168.63.129.16', port=80): Max retries exceeded with url: /machine/?comp=goalstate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9e89ee190>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

  With the timeouts in azure.py described above, the instance will not
  boot for around 20 minutes until all 240 connection attempts are
  completed.

  This is logged in /var/log/audit/audit.log, showing that the dhclient process executed from cloud-init is unable to write the dhclient.pid and dhcp.leases files that are needed to continue the datasource retrieval process:
  type=AVC msg=audit(1625678140.496:1898): apparmor="DENIED" operation="open" profile="/{,usr/}sbin/dhclient" name="/proc/8537/task/8540/comm" pid=8537 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0

  Adding the following file resolves the issue: /etc/apparmor.d/local/sbin.dhclient
  /var/tmp/cloud-init/cloud-init-dhcp-*/dhclient.pid lrw,
  /var/tmp/cloud-init/cloud-init-dhcp-*/dhcp.leases lrw,

  This allows dhclient executed via cloud-init to write the dhclient.pid
  and dhcp.leases files to /var/tmp/cloud-init and the instance to boot
  normally.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1934933/+subscriptions