The Key Ingredient in User Research: The Recruits

Like a fine meal, the recipe for success for any user research is somewhat complex. It takes careful planning, appropriate timing, and a few key ingredients. Far and away the most important ingredient is, not coincidently, the users who will test the concept.

The most rigorous and well thought-out methodology, the perfect study protocol, the ideal prototype and critically, a huge chunk of time and effort will be wasted if the study participants turn out to be sour. And embarrassingly, the soufflé flops as the client (who is footing the bill) watches.

Thus creating the appropriate screening criteria is vital. Here are some issues to consider when recruiting:

Recruit professionally and use floaters

Using a professional recruiting service is the best way to find great people. Not only can they help craft the screener questions, but they can spot potentially poor candidates that would otherwise slip through. And the time it takes to manage finding people in-house can be significant.

Professional recruiters can also set up floaters--people who wait on-site so they can participate in the study if (not when) there's a no-show. This is more expensive, but worth it when key clients are observing.

Manage segmentation-creep

More often than not, clients will quickly expand the scope of a study drastically by asking for deep comparisons between different groups of customers,  called segments. To compare segments this way you must include a reasonable sample size for each segment.

If a client wants to know if there are different usability issues for people on the east coast vs. the west coast, and for first-time users vs. experienced users, the sample size and analysis time explodes. Costs then rise quickly, so often it takes a little encouragement to limit extensive comparison across groups unless differences are likely but unknown and it's believed that these differences would affect the design.

Focus marketing-defined criteria

Sometimes the marketing department requests that we use their screener. This is great but only if it includes criteria that specifically defines actual users of the system. Sometimes marketing segments are defined too loosely to be actionable in user research. They can, however, be a great start to a focused screener.

Ensure technology expertise is carefully defined

For web usability studies the most important criteria to get right is experience using the web. This can be difficult to ascertain because people can't rate themselves well without some sort of baseline to compare against. A combination of age and several frequency of computer use questions are good proxies.

Filter out uncommunicative participants

Add a question or two to filter out people who may fit all the criteria perfectly, but then have little to say during the study. Self-reported ratings regarding "expressing oneself" work well to ensure vocal participants.

This list is just a start of course, but it captures some of the most important pitfalls. For more background and details on recruiting, Mike Kuniavsky's book, Observing the User Experience, is a great place to start.