Usability Testing [was: Re: FAQ page]

[Andreas Lloyd]

Duncan Lithgow wrote:
> If someone could make a spec with differently graded tasks, I'd be happy
> to record desktop sessions and get some friends and family to carry out
> the listed tasks. But writing the tasks and instructions should be done
> by someone with good understanding.
>   

At the GNOME conference this year, there was a talk by Novell's
Usability person, Anna Dirks, about doing your own Usability testing. I
couldn't find the slides, but there's a truckload of material from
Novell's usability testing of the GNOME desktop available at the
http://www.betterdesktop.org website.

Traditionally, usability testing involves multiple cameras, to record
not only what happens onscreen but also how the user uses her hands and
how facial expressions change as she attempts to complete the various
tasks. This is quite extensive, and generates a lot of data - both
quantitative (number of successes vs. failures at each task) and
qualitative (how the users react, their emotions and stress at the given
situation), as well as general behavioural data - which may be dependent
on the users' level of experience with similar products.

Dirks recommended that when you introduce a person to the usability
test, make them feel comfortable, small-talk, give them something to
drink (alcohol is not bad to settle nerves), and give them the money up
front (Novell pays their usability testers, so giving the money up front
makes sure that they don't feel that their pay is dependent on how well
they do on the test).

Then she recommends telling the user a simple background story, giving
the user a context for performing the tasks that she wants to test. It
needs to be an appealing background story - though what appeals may be
culturally specific depending on where and with whom you do the testing.

It is important to avoid all kinds of unpleasantness and stressful
situations in the background story, because the user will feel
unnecessarily stressed and will consequently perform a lot worse. For
instance, Dirks told a story of how the tester needed to get some
information from one of their co-workers' computer. This co-worker
happened to run Linux, which made the tester extremely angry - not at
the computer or the Linux system, but at the imaginary co-worker for
indirectly forcing him to deal with something he wasn't familiar with.

Instead, go with positive, relaxed, appealing scenarios like "You're a
gourmet chef, besides running a successful restaurant in New York, you
have also written several cookbooks to great critical acclaim, and you
have now been approached by a television network to do a series on how
to do good cooking easy. Preparing for the TV-series, you've been given
an office to use at the TV station, complete with a desktop PC.
Leisurely, you slink into the office one Thursday afternoon to do a
little research on the Internet. You learn back in the comfortable
office chart and sip from a long-stemmed glass of well-aged Bordeaux
while the computer boots up. The desktop appears to be called Ubuntu."

Apparently, people in the Boston area where Dirks does most of her
testing like to think that they're gourmet chefs. :-P

Once you have gotten people relaxed and in a good mood, they're much
more likely to give good (as in qualitatively useful) feedback on the
elements you want them to test.

For usability testing, you want as much feedback as possible. Ask the
testers to talk about every step. Have them think aloud: Their
rationalizations, frustrations and successes. What did they expect to
happen? What happened? Why do they think it happened?

Be cautious with asking too many questions, better explain what you want
them to do in detail before you start the testing.
If you're doing this with friends and family, this may seem somewhat
silly, but it does help to distance the already-existing relationship
between the two of you. Since most of this testing I guess will be
without cameras, forcing you to be standing next to the tester, you may
have to consider how to best present yourself in that space. Clearly,
you do not want to feel that they're embarrassing themselves, nor that
they're not doing their best because they know you'll help them if they
fail.

Okay, that was a fair bit. But that's just some general pointers that
won't fit in a spec. I'll try to formalize this in a wiki page on
usability testing at some point. Maybe we can just put some test suites
on the wiki, I don't know if we really need a spec for this at all.

Cheers,

Andreas

-- 
https://launchpad.net/people/lloydinho