[Bug 1078926] Re: raring instance failed to find EC2 datasource

[Steve Langasek]

** Description changed:

+ [Impact]
+ The previous SRU, while fixing the problem it was intended to fix, partially reintroduced the problem from before 2.41 where some filesystem events would end up blocking on one another when they shouldn't.  This is particularly noticeable for filesystems that have been mounted by the initramfs and there are jobs started on the 'mounted' event for one of these.  In the particular case of cloud-init, the jobs that start on mounted MOUNTPOINT=/ block waiting for the network to come up, which needs the 'virtual-filesystems' event which isn't happening because the 'mounted MOUNTPOINT=/run' event hasn't finished processing - it's behind the 'mounted /' in the event queue.  This intermittently causes cloud boot times in excess of 5 minutes.
+ 
+ [Test case]
+ 1. Boot the current quantal daily cloud images.
+ 2. Confirm that at least sometimes, the images take 5 minutes to boot.
+ 3. Upgrade mountall to the quantal-proposed version.
+ 4. Confirm that the images now boot without hitting the "wait for network" timeout.
+ 
+ [Regression potential]
+ Minimal, as this is correcting a regression from the previous SRU.
+ 
  This seems sporadic failure at best.  I had seen it on openstack fail similarly.
  Today I launched 7 instances of us-east-1 t1.micro from each of :
   ami-be70f5d7 ebs/ubuntu-raring-daily-amd64-server-20121109
   ami-de27a2b7 ebs/ubuntu-raring-daily-amd64-server-20121110
   ami-f6c94d9f ebs/ubuntu-raring-daily-amd64-server-20121111
   ami-cc8307a5 ebs/ubuntu-raring-daily-amd64-server-20121112
   ami-5c43c735 ebs/ubuntu-raring-daily-amd64-server-20121113
  and then another one of 20121109 and 20121113.
  
  One of the 2 20121109 failed to boot, and I will attach its console log
  and cloud-init.log.  The log was obtained from stopping the instance,
  attaching to another system, and getting it.  Then I restarted the
  instance, and it booted fine.
  
  Things that were of interest:
   * from cloud-init's perspective, the thing that failed was finding the ec2 metadata service.
   * In the failure case, there were no messages to the console log after initramfs, while
  cloud-init's log shows it WARNing, which should go to console.
  
  Related bugs:
   * mountall: bug 1059471  2.41 fails to mount root partition
-  * plymouth: bug 1086072  some output to /dev/console does not reach /dev/console 
+  * plymouth: bug 1086072  some output to /dev/console does not reach /dev/console
  
  ProblemType: Bug
  DistroRelease: Ubuntu 13.04
  Package: cloud-init 0.7.0-0ubuntu2
  ProcVersionSignature: User Name 3.5.0-17.28-generic 3.5.5
  Uname: Linux 3.5.0-17-generic x86_64
  ApportVersion: 2.6.2-0ubuntu3
  Architecture: amd64
  Date: Wed Nov 14 21:30:05 2012
  Ec2AMI: ami-be70f5d7
  Ec2AMIManifest: (unknown)
  Ec2AvailabilityZone: us-east-1b
  Ec2InstanceType: t1.micro
  Ec2Kernel: aki-825ea7eb
  Ec2Ramdisk: unavailable
  MarkForUpload: True
  PackageArchitecture: all
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  SourcePackage: cloud-init
  UpgradeStatus: No upgrade log present (probably fresh install)

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to mountall in Ubuntu.
https://bugs.launchpad.net/bugs/1078926

Title:
  raring instance failed to find EC2 datasource

Status in “mountall” package in Ubuntu:
  In Progress
Status in “mountall” source package in Quantal:
  In Progress
Status in “mountall” source package in Raring:
  In Progress

Bug description:
  [Impact]
  The previous SRU, while fixing the problem it was intended to fix, partially reintroduced the problem from before 2.41 where some filesystem events would end up blocking on one another when they shouldn't.  This is particularly noticeable for filesystems that have been mounted by the initramfs and there are jobs started on the 'mounted' event for one of these.  In the particular case of cloud-init, the jobs that start on mounted MOUNTPOINT=/ block waiting for the network to come up, which needs the 'virtual-filesystems' event which isn't happening because the 'mounted MOUNTPOINT=/run' event hasn't finished processing - it's behind the 'mounted /' in the event queue.  This intermittently causes cloud boot times in excess of 5 minutes.

  [Test case]
  1. Boot the current quantal daily cloud images.
  2. Confirm that at least sometimes, the images take 5 minutes to boot.
  3. Upgrade mountall to the quantal-proposed version.
  4. Confirm that the images now boot without hitting the "wait for network" timeout.

  [Regression potential]
  Minimal, as this is correcting a regression from the previous SRU.

  This seems sporadic failure at best.  I had seen it on openstack fail similarly.
  Today I launched 7 instances of us-east-1 t1.micro from each of :
   ami-be70f5d7 ebs/ubuntu-raring-daily-amd64-server-20121109
   ami-de27a2b7 ebs/ubuntu-raring-daily-amd64-server-20121110
   ami-f6c94d9f ebs/ubuntu-raring-daily-amd64-server-20121111
   ami-cc8307a5 ebs/ubuntu-raring-daily-amd64-server-20121112
   ami-5c43c735 ebs/ubuntu-raring-daily-amd64-server-20121113
  and then another one of 20121109 and 20121113.

  One of the 2 20121109 failed to boot, and I will attach its console
  log and cloud-init.log.  The log was obtained from stopping the
  instance, attaching to another system, and getting it.  Then I
  restarted the instance, and it booted fine.

  Things that were of interest:
   * from cloud-init's perspective, the thing that failed was finding the ec2 metadata service.
   * In the failure case, there were no messages to the console log after initramfs, while
  cloud-init's log shows it WARNing, which should go to console.

  Related bugs:
   * mountall: bug 1059471  2.41 fails to mount root partition
   * plymouth: bug 1086072  some output to /dev/console does not reach /dev/console

  ProblemType: Bug
  DistroRelease: Ubuntu 13.04
  Package: cloud-init 0.7.0-0ubuntu2
  ProcVersionSignature: User Name 3.5.0-17.28-generic 3.5.5
  Uname: Linux 3.5.0-17-generic x86_64
  ApportVersion: 2.6.2-0ubuntu3
  Architecture: amd64
  Date: Wed Nov 14 21:30:05 2012
  Ec2AMI: ami-be70f5d7
  Ec2AMIManifest: (unknown)
  Ec2AvailabilityZone: us-east-1b
  Ec2InstanceType: t1.micro
  Ec2Kernel: aki-825ea7eb
  Ec2Ramdisk: unavailable
  MarkForUpload: True
  PackageArchitecture: all
  ProcEnviron:
   TERM=xterm
   PATH=(custom, no user)
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  SourcePackage: cloud-init
  UpgradeStatus: No upgrade log present (probably fresh install)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1078926/+subscriptions