[Bug 1474030] Re: amulet _get_proc_start_time has a race which causes service restart checks to fail

[Liam Young]

** Changed in: heat (Juju Charms Collection)
    Milestone: 16.07 => 16.10

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to cinder in Juju Charms Collection.
Matching subscriptions: charm-bugs
https://bugs.launchpad.net/bugs/1474030

Title:
  amulet _get_proc_start_time has a race which causes service restart
  checks to fail

Status in Charm Helpers:
  Fix Released
Status in cinder package in Juju Charms Collection:
  Fix Released
Status in glance package in Juju Charms Collection:
  Confirmed
Status in heat package in Juju Charms Collection:
  Confirmed
Status in neutron-api package in Juju Charms Collection:
  Fix Released
Status in neutron-gateway package in Juju Charms Collection:
  Fix Released
Status in neutron-openvswitch package in Juju Charms Collection:
  Fix Released
Status in openstack-dashboard package in Juju Charms Collection:
  Fix Released

Bug description:
  Although most frequently observed in the openstack-dashboard, neutron-
  api and neutron-gateway charms, this affects all OpenStack charm
  amulet tests which utilize a service restart check.

  The test is to make a charm config change, then inspect the unit for
  process start time, and compare that start time with file modification
  time of the relevant config file.  In some cases, that arbitrary sleep
  results in the _get_proc_start_time checking for a service start time
  before the service is started, yielding OSError: [Errno 2] No such
  file or directory: '/proc/1242'.

  2015-07-13 14:57:39,934 test_900_restart_on_config_change DEBUG: Making config change on neutron-api...
  2015-07-13 14:57:40,527 test_900_restart_on_config_change DEBUG: Checking that service restarted: neutron-server
  2015-07-13 14:59:29,632 _get_proc_start_time DEBUG: CMDout: (u'1242', 0)
  2015-07-13 14:59:29,632 _get_proc_start_time DEBUG: Pid for neutron-server 1242
  Traceback (most recent call last):
    File "tests/019-basic-vivid-kilo", line 9, in <module>
      deployment.run_tests()
    File "/home/ubuntu/bzr/next/neutron-api/tests/charmhelpers/contrib/amulet/deployment.py", line 93, in run_tests
      getattr(self, test)()
    File "/home/ubuntu/bzr/next/neutron-api/tests/basic_deployment.py", line 516, in test_900_restart_on_config_change
      pgrep_full=True):
    File "/home/ubuntu/bzr/next/neutron-api/tests/charmhelpers/contrib/amulet/utils.py", line 290, in service_restarted
      if (self._get_proc_start_time(sentry_unit, service, pgrep_full) >=
    File "/home/ubuntu/bzr/next/neutron-api/tests/charmhelpers/contrib/amulet/utils.py", line 279, in _get_proc_start_time
      return self._get_dir_mtime(sentry_unit, proc_dir)
    File "/home/ubuntu/bzr/next/neutron-api/tests/charmhelpers/contrib/amulet/utils.py", line 260, in _get_dir_mtime
      return sentry_unit.directory_stat(directory)['mtime']
    File "/usr/lib/python2.7/dist-packages/amulet/sentry.py", line 88, in directory_stat
      return self._fs_data(path)
    File "/usr/lib/python2.7/dist-packages/amulet/sentry.py", line 75, in _fs_data
      return self._run_unit_script("filesystem_data.py {}".format(path))
    File "/usr/lib/python2.7/dist-packages/amulet/sentry.py", line 126, in _run_unit_script
      raise IOError(output)
  IOError: Traceback (most recent call last):
    File "/tmp/amulet/filesystem_data.py", line 7, in <module>
      s = os.stat(sys.argv[1])
  OSError: [Errno 2] No such file or directory: '/proc/1242'
  ERROR subprocess encountered error code 1

  
  real    17m21.555s

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-helpers/+bug/1474030/+subscriptions