Behind the scenes: A remote usability test

Behind the scenes: A remote usability test

Posted On:
February 21, 2011
By:
Warren Anthony
Behind the scenes of a remote usability test

The Challenge

We wanted to conduct a usability test for a major client, with test participants in the US, Canada and Europe.

Their target users are in a very specific audience, so we knew we would be recruiting from our client’s own customer database.

Participants were known to be relatively computer literate, but tough to pin down, and unlikely to spare time to travel specifically to help us test our client’s website.

Solution

We chose to conduct a moderated remote usability test for several key reasons:

  • Low cost - No travel required by participants, observers or our test team
  • Better feedback - Users at home or in their office are more relaxed
  • Faster - We can set up and test when our users are available
  • More depth - Moderated testing allows you to test more complex tasks, digging deeper into specific features over un-moderated testing approaches



Usability Testing Tools

We’ve used several tools and approaches to remote testing, and with new tools coming to market almost daily, it’s easier, cheaper and faster than ever to run usability tests with users around the globe.

A great reference guide for all things regarding remote usability testing is the Rosenfeld book Remote Research: http://rosenfeldmedia.com/books/remote-research/.

(The technology suggestions at the back are already a little out of date. We evaluated these and several of our own options before coming up with our plan.)

We’re fans of the new wave of usability testing sites:

These are all unmoderated solutions, however, which are great for quick, simple testing with basic scenarios, but don’t allow you to dig deeper with your users to explore more complex and realistic tasks.

WebEx

We use WebEx daily here at FCV for our meetings as we have clients across Canada and in the US. It’s not a dedicated usability tool per se, but for this type of test, it was a strong choice.

Pros

  • It’s reliable, easy to setup
  • It works on Mac and PC browsers (and the iPhone / iPad)
  • It has a built-in recording feature
  • Dial in conference option
  • Participants can take notes

Cons

  • Compared to TechSmith’s Morae (pretty much the industry benchmark), the note taking tools aren’t quite as refined.
  • The recordings don’t show mouse clicks, drags or other activities. You see just what appears on screen.
  • Users pay to dial in, by default.
    During our setup process, we had a scary moment when we realized our users were going to have to pay to call our Canada- or US-based conference line number. A quick call to WebEx and a little pleading and within a day or two we upgraded our account to include a toll-free number. Simple.

The Setup

Remote test configuration

WebEx provided full session recording capabilities, conference line and easy participant access.

As a backup, we used a laptop with Morae, as an additional participant on the call, to record the sessions. (Recording audio from the conference line box for that extra lo-fi feel!)

A further laptop was used by the Note Taker to record details of the sessions, with time codes and notes grouped by type (e.g., Usability Issue, Comment, New Task).

Results

  • The system worked perfectly, and even better than expected.
  • All users logged in on time with zero technical issues.
  • Users commented on the toll-free option as a positive thing. (Phew!)
  • We were able to forget the technology, and concentrate on running a smooth and focussed usability test to help our client improve their website.

Since WebEx is a hosted service, the recordings and conference line continued to work, despite a short Wi-Fi hiccup here in the office after an unscheduled security update by our IT team. We simply reconnected and continued taking notes with just a few seconds of downtime on our backup machine.

We’ve used the same system again, and I would recommend it to anyone wishing to run a remote test for their clients, or themselves really, especially if you are looking for an excellent screen sharing application.

Got a better idea? Had a similar experience? Let us know!

What do you think?

Did we hit the nail on the head? We want to know.

Add your comment below, or tweet us @FCV.

Post new Blog Comment

Name: *
Email: *
Your email will not be shown
FCV: *
To help cut spam so we can focus on nice real people, please type FCV here
Comment: *