[Bug 1262575] [NEW] Autopilot: Failed test results giving non-helpful/no-info messages for error dialogs

[Dan Chapman]

Public bug reported:

As an example bug 1261745, on jenkins it displays the error as the page
title hasn't changed, meaning we didn't get to the next step of the
install but actually the problem was we were stuck at an error dialog
displaying the installer couldn't create a JFS partition.

This type of problem can also be seen on the lubuntu tests where the
test result is reporting that the progress bar didn't reset but is in
fact halted by an error dialog. So these results provide no useful
information before further investigation as to what the actual cause for
test failure was.

The key problem is the tests check for error dialogs at anticipated
points throughout the install usually between providing input and
switching to the next step or while polling on the progress bar,  which
in reality the dialogs could pop up at any point of the install.

It would be a better idea to only check for visible error dialogs at the
end of the test whether its a fail or pass, and try and grab as much
info from the dialog as possible and add it to the test result, this way
we will always get the info of any error dialogs open at the end of a
test, and not need to try and guess where to catch them during.

** Affects: ubiquity (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: autopilot

** Description changed:

  As an example bug 1261745, on jenkins it displays the error as the page
  title hasn't changed, meaning we didn't get to the next step of the
  install but actually the problem was we were stuck at an error dialog
  displaying the installer couldn't create a JFS partition.
  
  This type of problem can also be seen on the lubuntu tests where the
  test result is reporting that the progress bar didn't reset but is in
  fact halted by an error dialog. So these results provide no useful
  information before further investigation as to what the actual cause for
  test failure was.
  
  The key problem is the tests check for error dialogs at anticipated
- points throughout the install usually providing input and switching to
- the next step or while polling on the progress bar,  which in reality
- the dialogs could pop up at any point of the install.
+ points throughout the install usually between providing input and
+ switching to the next step or while polling on the progress bar,  which
+ in reality the dialogs could pop up at any point of the install.
  
  It would be a better idea to only check for visible error dialogs at the
  end of the test whether its a fail or pass, and try and grab as much
  info from the dialog as possible and add it to the test result, this way
  we will always get the info of any error dialogs open at the end of a
  test, and not need to try and guess where to catch them during.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ubiquity in Ubuntu.
https://bugs.launchpad.net/bugs/1262575

Title:
  Autopilot: Failed test results giving non-helpful/no-info messages for
  error dialogs

Status in “ubiquity” package in Ubuntu:
  New

Bug description:
  As an example bug 1261745, on jenkins it displays the error as the
  page title hasn't changed, meaning we didn't get to the next step of
  the install but actually the problem was we were stuck at an error
  dialog displaying the installer couldn't create a JFS partition.

  This type of problem can also be seen on the lubuntu tests where the
  test result is reporting that the progress bar didn't reset but is in
  fact halted by an error dialog. So these results provide no useful
  information before further investigation as to what the actual cause
  for test failure was.

  The key problem is the tests check for error dialogs at anticipated
  points throughout the install usually between providing input and
  switching to the next step or while polling on the progress bar,
  which in reality the dialogs could pop up at any point of the install.

  It would be a better idea to only check for visible error dialogs at
  the end of the test whether its a fail or pass, and try and grab as
  much info from the dialog as possible and add it to the test result,
  this way we will always get the info of any error dialogs open at the
  end of a test, and not need to try and guess where to catch them
  during.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1262575/+subscriptions