[Bug 1581200] Re: Ubuntu cloud-init expects trailing dot on GCE metadata FQDN

[Chris J Arges]

Hello Max, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and
be available at https://launchpad.net/ubuntu/+source/cloud-
init/0.7.5-0ubuntu1.20 in a few hours, and then in the -proposed
repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
enable and use -proposed.  Your feedback will aid us getting this update
out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-needed to verification-done. If it does not fix the
bug for you, please add a comment stating that, and change the tag to
verification-failed.  In either case, details of your testing will help
us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: cloud-init (Ubuntu Trusty)
       Status: In Progress => Fix Committed

** Tags added: verification-needed

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1581200

Title:
  Ubuntu cloud-init expects trailing dot on GCE metadata FQDN

Status in cloud-init package in Ubuntu:
  Fix Released
Status in cloud-init source package in Trusty:
  Fix Committed
Status in cloud-init source package in Xenial:
  Fix Released

Bug description:
  [Impact]

   * If bind9 is installed and configured as a local DNS server on an
  ubuntu instance on GCE then on every reboot cloud-init will fail to
  retrieve instance metadata from GCE due to the lookup hostname not
  resolving.

   * Backporting of this is necessary as instances with bind9 installed
  can no longer take advantage of cloud-init

   * The patch fixes this bug by updating the hostname used in the
  metadata lookup to one that is included in /etc/hosts. As such it will
  resolve, even if bind9 hasn't started yet.

  [Test Case]

  #launch an instance of ubuntu 14.04 on GCE
  sudo apt-get update
  sudo apt-get install -y bind9
  #Add the Google DNS servers as global forwarders and configure bind9 for the GCE environment
  cat << EOF | sudo tee /etc/bind/named.conf.options
  options {
      directory "/var/cache/bind";

      forwarders {
          169.254.169.254;
      };  

      recursion yes;
      dnssec-validation no; 
      dnssec-enable no; 
      auth-nxdomain no;
      listen-on { 127.0.0.1; };
    };
  EOF  
  sudo service bind9 restart
  #setup your instance to use bind9 instead of the Google server
  echo "supersede domain-name-servers 127.0.0.1;" | sudo tee -a /etc/dhcp/dhclient.conf
  sudo dhclient -pf /run/dhclient.eth0.pid -x
  sudo dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
  if grep -q "nameserver 127.0.0.1" "/etc/resolv.conf"; then echo "resolv.conf has been updated"; fi
  if host -t A metadata.google.internal | grep '169.254.169.254' > /dev/null; then echo "host lookup succeeded as expected"; fi
  sudo service bind9 stop
  if host -t A metadata.google.internal | grep 'connection timed out' > /dev/null; then echo "host lookup failed as expected"; fi

  #Now reboot the instance
  sudo reboot
  #Once rebooted run the following
  if grep -q "http://metadata.google.internal./computeMetadata/v1/ is not resolvable" "/var/log/cloud-init.log"; then echo "cloud-init failed to lookup metadata as expected"; else echo "cloud-init did _not_ fail to lookup metadata as expected";  fi

  A patched ubuntu14.04 has been built. To test the patch run the above but after reboot run
  #launch a patched instance
  gcloud compute instances create ubuntu1404-patched-cloudinit --image daily-ubuntu-proche-cloudinit-1404-trusty-v20160627 --image-project ubuntu-os-cloud-devel

  #on a patched instance run the following after reboot
  if grep -q "http://metadata.google.internal/computeMetadata/v1/ is not resolvable" "/var/log/cloud-init.log"; then echo "cloud-init failed to retrieve metadata"; else echo "cloud-init did successfully retrieve metadata as expected";  fi

  [Regression Potential]

   * GCE are questing this change. 
   * The reported issue only affects GCE users and only a small set of those users will be using a local DNS server. 
   * The change is a single character change and has been tested and as such has limited regression potential. 

  
  [Original Bug Report]

  cloud-init hostname breaks because /etc/hosts does not have the
  trailing dot on metadata FQDN.

  Background:
  On Ubuntu, cloud-init sets the hostname using our metadata service. To do this, it hits "metadata.google.internal." (note trailing dot) via HTTP.

  We have entries in /etc/hosts for the metadata service to ensure that
  we can access it at boot time (if DNS is not yet up) as we have other
  init scripts which block bootup when metadata cannot be reached.
  However, these /etc/hosts entries only have "metadata.google.internal"
  (no trailing dot) entries.

  When a customer runs their own bind9 daemon, it starts *after* cloud-
  init, meaning that cloud-init must use /etc/hosts to find the metadata
  service. When it cannot, it incorrectly sets the hostname to
  "$hostname.localdomain" instead of just $hostname.

  Proposed fix:
  Update:
  http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceGCE.py

  Line 28 should read:
  'metadata_url': 'http://metadata.google.internal/computeMetadata/v1/'

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1581200/+subscriptions