[Bug 1977320] Re: libcloud: autopkgtest is flaky due to racy "test_retry_*" tests

[Nick Rosbrook]

** Patch added: "libcloud-lp-1977320-v2.debdiff"
   https://bugs.launchpad.net/ubuntu/+source/libcloud/+bug/1977320/+attachment/5594714/+files/libcloud-lp-1977320-v2.debdiff

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1977320

Title:
  libcloud: autopkgtest is flaky due to racy "test_retry_*" tests

Status in libcloud package in Ubuntu:
  In Progress

Bug description:
  This issue has appeared recently on ppc64el autopkgtest [1]. The issue
  is that some of the tests from libcloud/test/test_connection.py
  incorrectly assume the mock side effects will always finish within the
  specified timeouts, which may not happen depending on the execution
  environment.

  The following autopkgtest log snippets demonstrate this:

  Take from [2]:

  =================================== FAILURES ===================================
  __ ConnectionClassTestCase.test_retry_on_all_default_retry_exception_classes ___

  self = <test.test_connection.ConnectionClassTestCase testMethod=test_retry_on_all_default_retry_exception_classes>
  mock_connect = <MagicMock name='request' id='134056393449408'>

      @patch('libcloud.common.base.Connection.request')
      def test_retry_on_all_default_retry_exception_classes(self, mock_connect):
          con = Connection()
          con.connection = Mock()
          connect_method = 'libcloud.common.base.Connection.request'
      
          self.retry_counter = 0
      
          def mock_connect_side_effect(*args, **kwargs):
              self.retry_counter += 1
      
              if self.retry_counter < len(RETRY_EXCEPTIONS):
                  raise RETRY_EXCEPTIONS[self.retry_counter]
      
              return 'success'
      
          mock_connect.__name__ = 'mock_connect'
          mock_connect.side_effect = mock_connect_side_effect
          retry_request = Retry(timeout=0.6, retry_delay=0.1,
                                  backoff=1)
  >       result = retry_request(con.request)(action='/')

  test/test_connection.py:583: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  /usr/lib/python3/dist-packages/libcloud/utils/retry.py:127: in retry_loop
      raise last_exc
  /usr/lib/python3/dist-packages/libcloud/utils/retry.py:103: in retry_loop
      return transform_ssl_error(func, *args, **kwargs)
  /usr/lib/python3/dist-packages/libcloud/utils/retry.py:93: in transform_ssl_error
      raise exc
  /usr/lib/python3/dist-packages/libcloud/utils/retry.py:88: in transform_ssl_error
      return function(*args, **kwargs)
  /usr/lib/python3/dist-packages/mock/mock.py:1100: in __call__
      return _mock_self._mock_call(*args, **kwargs)
  /usr/lib/python3/dist-packages/mock/mock.py:1104: in _mock_call
      return _mock_self._execute_mock_call(*args, **kwargs)
  /usr/lib/python3/dist-packages/mock/mock.py:1167: in _execute_mock_call
      result = effect(*args, **kwargs)
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

  args = (), kwargs = {'action': '/'}

      def mock_connect_side_effect(*args, **kwargs):
          self.retry_counter += 1
      
          if self.retry_counter < len(RETRY_EXCEPTIONS):
  >           raise RETRY_EXCEPTIONS[self.retry_counter]
  E           libcloud.utils.retry.TransientSSLError: ()

  test/test_connection.py:575: TransientSSLError
  ---

  Taken from [3]:

  =================================== FAILURES ===================================
  _______________ ConnectionClassTestCase.test_retry_with_timeout ________________

  self = <test.test_connection.ConnectionClassTestCase testMethod=test_retry_with_timeout>
  mock_connect = <MagicMock name='request' id='127398439136048'>

      @patch('libcloud.common.base.Connection.request')
      def test_retry_with_timeout(self, mock_connect):
          con = Connection()
          con.connection = Mock()
      
          mock_connect.side_effect = socket.gaierror('')
          retry_request = Retry(timeout=0.2, retry_delay=0.1,
                                  backoff=1)
          self.assertRaises(socket.gaierror,
                            retry_request(con.request), action='/')
      
  >       self.assertGreater(mock_connect.call_count, 1,
                              'Retry logic failed')
  E       AssertionError: 1 not greater than 1 : Retry logic failed

  test/test_connection.py:457: AssertionError
  ---

  [1] https://autopkgtest.ubuntu.com/packages/libc/libcloud/kinetic/ppc64el
  [2] https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/ppc64el/libc/libcloud/20220528_194230_765c9@/log.gz
  [3] https://autopkgtest.ubuntu.com/results/autopkgtest-kinetic/kinetic/ppc64el/libc/libcloud/20220526_090523_a6c63@/log.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libcloud/+bug/1977320/+subscriptions