[Bug 1581200] [NEW] Ubuntu cloud-init expects trailing dot on GCE metadata FQDN

[Launchpad Bug Tracker]

You have been subscribed to a public bug by Philip Roche (philroche):

[Impact]

 * If bind9 is installed and configured as a local DNS server on an
ubuntu instance on GCE then on every reboot cloud-init will fail to
retrieve instance metadata from GCE due to the lookup hostname not
resolving.

 * Backporting of this is necessary as instances with bind9 installed
can no longer take advantage of cloud-init

 * The patch fixes this bug by updating the hostname used in the
metadata lookup to one that is included in /etc/hosts. As such it will
resolve, even if bind9 hasn't started yet.

[Test Case]

#launch an instance of ubuntu 14.04 on GCE
sudo apt-get update
sudo apt-get install -y bind9
#Add the Google DNS servers as global forwarders and configure bind9 for the GCE environment
cat << EOF | sudo tee /etc/bind/named.conf.options
options {
    directory "/var/cache/bind";

    forwarders {
        169.254.169.254;
    };  

    recursion yes;
    dnssec-validation no; 
    dnssec-enable no; 
    auth-nxdomain no;
    listen-on { 127.0.0.1; };
  };
EOF  
sudo service bind9 restart
#setup your instance to use bind9 instead of the Google server
echo "supersede domain-name-servers 127.0.0.1;" | sudo tee -a /etc/dhcp/dhclient.conf
sudo dhclient -pf /run/dhclient.eth0.pid -x
sudo dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
if grep -q "nameserver 127.0.0.1" "/etc/resolv.conf"; then echo "resolv.conf has been updated"; fi
if host -t A metadata.google.internal | grep '169.254.169.254' > /dev/null; then echo "host lookup succeeded as expected"; fi
sudo service bind9 stop
if host -t A metadata.google.internal | grep 'connection timed out' > /dev/null; then echo "host lookup failed as expected"; fi

#Now reboot the instance
sudo reboot
#Once rebooted run the following
if grep -q "http://metadata.google.internal./computeMetadata/v1/ is not resolvable" "/var/log/cloud-init.log"; then echo "cloud-init failed to lookup metadata as expected"; else echo "cloud-init did _not_ fail to lookup metadata as expected";  fi

A patched ubuntu14.04 has been built. To test the patch run the above but after reboot run
#launch a patched instance
gcloud compute instances create ubuntu1404-patched-cloudinit --image daily-ubuntu-proche-cloudinit-1404-trusty-v20160627 --image-project ubuntu-os-cloud-devel

#on a patched instance run the following after reboot
if grep -q "http://metadata.google.internal/computeMetadata/v1/ is not resolvable" "/var/log/cloud-init.log"; then echo "cloud-init failed to retrieve metadata"; else echo "cloud-init did successfully retrieve metadata as expected";  fi

[Regression Potential]

 * GCE are questing this change. 
 * The reported issue only affects GCE users and only a small set of those users will be using a local DNS server. 
 * The change is a single character change and has been tested and as such has limited regression potential. 


[Original Bug Report]

cloud-init hostname breaks because /etc/hosts does not have the trailing
dot on metadata FQDN.

Background:
On Ubuntu, cloud-init sets the hostname using our metadata service. To do this, it hits "metadata.google.internal." (note trailing dot) via HTTP.

We have entries in /etc/hosts for the metadata service to ensure that we
can access it at boot time (if DNS is not yet up) as we have other init
scripts which block bootup when metadata cannot be reached. However,
these /etc/hosts entries only have "metadata.google.internal" (no
trailing dot) entries.

When a customer runs their own bind9 daemon, it starts *after* cloud-
init, meaning that cloud-init must use /etc/hosts to find the metadata
service. When it cannot, it incorrectly sets the hostname to
"$hostname.localdomain" instead of just $hostname.

Proposed fix:
Update:
http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/DataSourceGCE.py

Line 28 should read:
'metadata_url': 'http://metadata.google.internal/computeMetadata/v1/'

** Affects: cloud-init (Ubuntu)
     Importance: Medium
     Assignee: Philip Roche (philroche)
         Status: Fix Released

** Affects: cloud-init (Ubuntu Trusty)
     Importance: Undecided
     Assignee: Philip Roche (philroche)
         Status: New

** Affects: cloud-init (Ubuntu Xenial)
     Importance: Undecided
         Status: New

-- 
Ubuntu cloud-init expects trailing dot on GCE metadata FQDN
https://bugs.launchpad.net/bugs/1581200
You received this bug notification because you are a member of Ubuntu Sponsors Team, which is subscribed to the bug report.