Default Desktop Experience for 11.04 - User testing results

[Matthew Paul Thomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Rick Spencer wrote on 08/04/11 02:38:
>...
> Back at UDS for 11.04 in Orlando, Mark set the goal of using Unity by
> default on the Ubutu desktop. Given the current course of development,
> it appears that we are going to achieve this goal, and Unity will stay
> the default for 11.04.
> 
> I'm following up on this list at the suggestion of the Tech Board to
> give folks a chance to respond or escelate any concerns.
>...

Last week, Charline Poirier ran a user test of Unity, with 11 individual
participants. This week, I have helped Charline analyze the results.

In this summary, I have numbered the participants:
- - P1, 19, a student and Mac user
- - P2, 33, an administrator and Mac user
- - P3, 25, a student and Windows user
- - P4, 32, a teacher and Windows user
- - P5, 27, a compliance officer who uses both Windows and Mac
- - P7, 44, a life coach who uses both Windows and Ubuntu
- - P8, 30, an IT network manager and Windows user
- - P9, 22, a student and Windows user
- - P10, 21, a student and Windows user
- - P11, 47, a teacher and Windows user
- - P12, 34, an operations manager and Windows user.

The test machine was a Lenovo ThinkPad T410i running Ubuntu Natty with
unity 3.8.2-0ubuntu1 and compiz 1:0.9.4git20110322-0ubuntu5.

Charline asked each participant to try several tasks. Not every
participant had time to try every task.

*   Every participant who was asked understood most of the launcher
    items. P7 and P11 thought that "LibreOffice Calc" was a calculator,
    and P7 and P9 thought Ubuntu Software Center was the Recycle Bin.
    Nobody understood Ubuntu One. (The Classic session has much smaller
    icons for everything, but has a visible-by-default label plus an
    extra tooltip for each app.)

*   Almost everyone understood most of the indicators, but 4/11 people
    (P7, P9, P11, P12) thought the Me menu icon might be a close button.

*   11/11 people easily launched Firefox to check Web mail.

*   8/10 people could find a window's menus, but 7/8 of them learned to
    access them by hovering over maximized close/minimize/unmaximize
    buttons then moving horizontally -- which was extremely slow, and
    failed whenever the window wasn't maximized.

*   10/11 worked out how to open a new Firefox window, though 5/10 first
    tried clicking Firefox in the launcher again, which didn't work.

*   Only 4/11 worked out how to change the background picture. This is
    not as bad as it looks: for some of the others, Charline had asked
    them *not* to right-click on the desktop, because she was testing
    access to settings in general. Nevertheless, no-one found System
    Settings, in the session menu or anywhere else.

*   Only 5/11 could easily rearrange items in the launcher. For the
    other six, the main problems were that the launcher scrolled when
    they were trying to drag an item, or that it didn't accept a drop.
    (P10 was particularly unlucky in doing a Dash search for "menu" and
    finding the Main Menu editor, which is useless in Unity but still
    present by default.)

*   6/10 could easily find and launch a game that wasn't in the
    launcher. (P2 deserves special mention for finding and launching the
    game's .desktop file amongst piles of detritus in Nautilus's "File
    System" search results.)

*   Only 1/9 (P4) easily added that game to the launcher. For the other
    eight, the main problems were that the launcher disappeared when a
    window was maximized or at the left of the screen, that Dash items
    didn't have context menus, and that the launcher often didn't accept
    a drop.

*   2/2 successfully removed an item from the launcher.

*   Only 2/6 noticed an XChat Gnome notification, despite (1) a
    notification bubble appearing, (2) the Ubuntu button going blue,
    (3) the messaging menu envelope going blue, and (4) an emblem
    appearing on XChat Gnome's launcher.

*   9/9 easily launched LibreOffice Writer to write a letter.

*   5/5 easily found today's date.

*   9/9 easily saved their LibreOffice Writer document. (P1 recovered
    amazingly well after trying to save "Letter to Mr Smith 08/04/11",
    and getting the vile response "Error stating file '/home/ubuntu
    /Documents/Letter to Mr Smith 08/04': No such file or directory").

*   9/11 people could easily close a window. The other two (P2, P7)
    were not the only ones to be attacked by a bug that hid the title
    bar for a window underneath the menu bar; they were just the only
    participants for whom that bug really cramped their style.

*   9/9 easily found and opened an existing document.

*   8/9 easily copied text from one document into another. The other one
    (P2) managed it eventually, but the missing title bar for one of the
    document windows was again a major stumbling block.

*   Only 1/7 (P9, a Windows 7 user) easily arranged windows side by side
    by discovering the window snap feature. (That's probably not really
    a problem; it's a power user feature.)

*   Only 5/10 could easily delete a document, and two of those five (P3,
    P5) weren't sure that it had actually worked. For those who failed,
    the major problems were that files in the Dash didn't have context
    menus and weren't selectable, and that they couldn't see the Rubbish
    Bin (this was a UK test) folded away at the bottom of the launcher.

*   6/10 could easily see how many applications were running. (This was
    something people learned by experiment, and again probably isn't
    important.)

*   Only 3/6 could easily reveal the launcher when a window was touching
    the left of the screen (for example, a maximized window). For those
    who failed, the problems were that the effect of hovering over the
    Ubuntu button was unpredictable (P7), that the launcher popped up
    unwanted when aiming for a close button (P9), and that it popped up
    unwanted when trying to drag a file to a different target near the
    left of the screen (P11).

*   0/2 people could play MP3 songs stored on a USB key. This appeared
    to be entirely because the "Search for suitable plugin?" is too
    geeky, and was not Unity's fault at all.

During these tasks, the participants also revealed many other
interesting quirks and bugs. For example:

*   Nobody seemed to understand what the Ubuntu button was for, or the
    distinction between the Dash main screen and the Applications
    screen.

*   P1 and P12 both clicked in the Dash search field several times, but
    both concluded that the field could not be typed in (probably
    because the caret didn't blink and the hint text didn't disappear).

*   4/11 participants (P2, P3, P7, P10) tried double-clicking on
    "Applications" or "Files & Folders" in the launcher, but that just
    made the screen flicker.

*   5/11 participants (P2, P3, P5, P9, P10, P11) crashed Unity during
    their hour of testing. And towards the end of her test, P11 opened
    a zombie quicklist that stayed on top of everything and didn't
    respond to clicks.

I'll post more about the design issues (including those not specific to
Unity) in a separate message to the ayatana@ list.

- -- 
mpt
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2npr8ACgkQ6PUxNfU6ecqBWQCgulHgSGo4anzShhPtLFubUtDE
fRkAn24p/kDBbbWZ4t42e9RJ4DUYQF78
=GdiZ
-----END PGP SIGNATURE-----