Tips for testing your designs with UserTesting.com

I love quick and dirty user research. I try to eliminate barriers that prevent teams from talking to and observing people using their products. (And if you’re not watching people use your product, you don’t know what you’re missing.) For our Google Ventures portfolio companies, I teach an intensive “User Research, Quick ’n’ Dirty” workshop.

So it’s about time I finally tried UserTesting.com. How’d it go? Frankly, I expected to be a lot more critical of it. UserTesting.com is like a usability vending machine — that dispenses bite-sized research crack. It’s strangely addicting.

In less than three days I ran three tests with 15 testers for about $525, and documented a long list of important site usability issues. Pretty good ROI!

UserTesting.com is great for:

• Quickly uncovering basic usability issues. For example: Can users complete important tasks? Can they find a feature when they need it? Do they understand what’s available?
• Testing short, well-defined tasks with specific goals. For example: finding specific info, registering, completing a form, etc.
• Running blind tests. Try testing competitors’ sites — you can learn a lot from their “prototypes.”
• Sharing videos with teams. The testers’ videos are bite-sized (10–20 mins) and easy to share with a unique URL.
• Very general web consumers. UserTesting.com offers only very blunt screening criteria, such as age, gender, and income.

I’m usually more picky about selecting participants that match a site’s target users. For example, if you’re building a site for a specific group like parents or small business owners, you’ll get better results by testing it with parents or small business owners.

UserTesting.com does allow you to bring your own participants (as well as intercept visitors to your site via Ethnio), but if I’m going to invest time and money into recruiting specific participants I’d rather engage them in more thorough, moderated studies.

It’s not yet suited for:

• Investigating users’ actual habits, attitudes, nagging problems, or workflows. For example: How do people research and shop for a specific product or service? What other features might people need? I’ll stick with long-form methods for those.
• Testing anything confidential or sensitive that requires a non-disclosure agreement.
• Complex tasks that require visiting multiple sites, such as comparing sites, or gathering data from one site to complete a task on another. I expect the likelihood of testers getting off track would explode as the complexity of tasks increases.

Tips for running a study with UserTesting.com:

• As with any research, first define the objectives and research questions you’re hoping to answer.
• Draft your scenarios, tasks, and follow-up questions somewhere before pasting them into the “Set Up Your Usability Test” page. I learned the hard way that the site didn’t auto-save my work or let me copy all of the fields to paste them into a separate doc.
• Write short, simple tasks. Review their tips for writing great tasks.
• Indicate how much time you want testers to spend on certain tasks. I didn’t expect a tester to spend 8 minutes describing his first impressions of a page.
• Run a pilot test before paying for several testers. It’s tricky to write tasks and questions that will be followed as intended without any misunderstandings. Unlike moderated studies, you can’t explain or rephrase questions, or shepherd testers through a test.
• If you’re planning to run multiple tests, don’t launch them all at once. It’s worth the time to learn from one test before you start the next one.
• Give more weight to what you observe in the videos than to testers’ overly positive written comments.
• Plan time to watch testers’ videos. In case it’s not obvious, you still have to review and analyze the videos. UserTesting.com delivers the testers’ videos and comments — not findings or recommendations. That’s fine with me since I wouldn’t trust anyone to analyze it for me. The quality of your findings will depend on your ability to analyze the videos objectively.
• Promptly check videos to make sure they’re usable. I received three useless videos: two were extremely short, and one recorded at wrong screen resolution. If I’d checked the videos immediately instead of waiting until the next morning, I could have gotten replacements by the time I was ready to watch them. (UserTesting.com should add basic quality controls to avoid releasing videos that are too short or have other obvious faults.)
• When you receive a bad video, request a replacement from customer support via live chat. They immediately credited my account and deleted the bad videos. (So if you want to keep those lousy videos for some reason, download them before contacting them.)

While UserTesting.com definitely won’t replace my longer moderated usability and discovery research interviews, it’s a great addition to my tool kit. And for those teams that haven’t yet developed good habits of watching their users (you know who you are!), UserTesting.com is a great gateway drug.

How do you use UserTesting.com? What other tools and techniques do you use for quick and dirty user research?