Pat’s Point of View

There are many user research methods one can use, and there are even more variations and names for them. But regardless of what name they’re given, methods should be chosen that are suitable for the situation at hand.

In this article I give a quick overview of the methods I commonly use, broken down in to main categories:

• Direct user contact—where the researcher does very much interact with users, or members of the audience as I prefer to call them
• Indirect user contact—where the researcher does not actually interact with members of the audience

The list is by no means exhaustive but it gives a good indication of the breadth and depth of methods that can be employed in user research.

Direct user contact

Interviews

Interviews done with actual users are the mainstay of user research, as far as I’m concerned. Such an interview (also sometimes known as an IDI or “individual depth interview”) should ideally be conducted in the participants’ normal context, or environment in which they normally undertake the activity you’re studying. You can have a set of questions to guide you—called “semi-structured” interviews—or you could keep it completely open—called “unstructured”.

Tip: another potentially useful variation is known as the proxy interview (also known as ‘key informant’ or ‘intermediary’ interviews).

Focus groups

Small group discussions around one or more topics, often used in (and given a bad name by) market research but usually with a different purpose in mind and using a different approach. They are good for idea generation, brainstorming, comparing alternative designs. The social interaction can elicit more information than if you interviewed each person individually. However, they’re not so good because you can’t see how people do what they do, only what they say they do. Managing the group can be difficult and peer pressure can affect the answers given (known as “group think”).

Tip: storytelling sessions (aka “anecdote circles”) are a good alternative to a traditional focus group. These techniques are gaining popularity in management and business collaboration fields.

Workshops

Workshops are a good framework for a variety of smaller research methods that can be strung together to make the most of your time with participants. Almost any “workshop activity” can be used, depending on what it is you’re trying to achieve; I’ve listed two below, but you can find more at design games.

Tip: most of my “focus groups” are actually workshops, but the former is a term better understood by stakeholders.

Card sorting

Whilst normally used as an IA design technique, card sorting is an excellent way to research an audience’s mental models and terminology, particularly as part of a workshop. For instance, what do they call XYZ? where is there disagreement on the classification of content/things? what is the organisation’s mental model? (when running a stakeholder workshop). In a group situation, pay attention to the discussion and debate, not just the obvious output (the “quantitative taxonomic data”).

Tip: regarding that taxonomic data, you can use Donna Spencer’s card sort analysis spreadsheet to help see interesting patterns.

Product Reaction Cards

One workshop activity I’m quite fond of is the Microsoft Product Reaction Cards. You get users (either individually or as a group) to choose the words they feel reflect a certain topic, from a set of pre-defined words. For example: “what words best describe the current website?” or “how would you like the intranet to be in the future?”.

Tip: initiate a “talk aloud” protocol to gain some insight into the participants’ thoughts and what they mean by each word. Asking them to pick their top 5 words can help them really focus in on what they think.

Contextual inquiry

A contextual inquiry (or contextual enquiry if you prefer) is essentially an unstructured interview in the context in which users use the system. The researcher acts as the student and learns everything they can from the user (eg how they perform their job, what tools they use). As with a lot of direct contact research methods, it’s not so much about what you find out about a particular product/system/website/application, but more about getting access to people you know are your audience. You want to learn about what makes them tick, who they are beyond a one-dimensional demographic breakdown—which is so often how the audience is seen by organisations.

Visual Anthropology

Taking photos in the field is an excellent way to document the environment and behaviour of your audience. Documentaries, many of which could be called “Ethnographic films”, are a natural extension of this concept. They are a rich and emotive form of communication; particularly useful when presenting research findings to project stakeholders. This would usually be combined with, or part of, a wider study.

Some great examples of visual anthropology in the form of film have been published on the web by Michael Wesch, such as A Vision of Students Today.

Shadowing

Like a contextual inquiry, only on the move. The researcher basically follows his or her participants around as they undertake tasks or behaviour that is being studied (and probably quite a bit that is not!). Obviously you need to get their permission, otherwise it’s just called stalking.

I’ve heard a number of other names for this method, depending on who it is that’s describing it; for example journalists often call this a “ride-along”, as in “I rode along with the emergency services crew to get a first hand view of the effects of binge drinking on our city’s streets”.

Indirect user contact

Content inventory

A key research method for understanding content, also commonly known as a “content audit”. You develop an inventory of all pages, making note of meta information such as content and functionality, content owners, the status of the content (out-of-date etc). The output is useful for inputting into card sorting and content migration activities.

Heuristic evaluation

A person assesses the system (eg website) using a list of heuristics, or guidelines, based on usability best practice and design conventions. This is possibly the most common (and first) method of research undertaken by many user experience practitioners.

A formal heuristic evaluation involves the use of a comprehensive checklist of heuristics used by multiple assessors who score the system individually, for methodological validity. However, this approach is rarely taken in my experience; most heuristic reviews are less formal and more akin to an “expert review” of the system, so much so that the two names for this method are used interchangeably.

Competitor review

The competitors, or peers, of the system are reviewed to compare against each other and the system being (re)designed. This would usually use a heuristic review as the basis. The method is useful for setting a benchmark but also for collecting design ideas and identifying opportunities.

Questionnaires and surveys

Technically, questionnaires are self-reported (and thus indirect) whereas surveys are conducted by the researcher (and thus direct contact), but the terms are often used interchangeably. Most user research tends to use the self-reported form.

Either way, this method is good for collecting bulk responses (it’s usually a quantitative method) and for comparison of results over time. Among the reasons why this method is not so good if the fact that it can be very difficult to create questions that respondents understand and answer in the way you intend.

Tip: I often use a “survey” to quantify the findings from qualitative research. After having done in-depth qualitative research using a combination of the methods outlined above, some of these findings can be extracted and a much larger sample size of people can be quizzed to get some numbers around the effect/opinion/behaviour observed in the qualitative research.

Panels

Like an online survey that is given to a specific set of people, who have been recruited for the purposes of research. This method, commonly used for market research, can be an excellent way of getting input from a large sample of your audience. Many researchers are now using panels in a manner similar to a virtual focus group, with more “live” interaction being added to the traditional panel method (which is usually not real-time).

Tip: while the cost of a panel can be relatively expensive, it can be a good replacement for a survey to help quantify qualitative research findings with a larger sample size, assuming you can get enough of the right kind of people on the panel.

Analytics

Also called “web analytics”, “web stats” or “traffic statistics”. This refers to measuring key metrics about what users of a website or application are actually doing; hopefully something more sophisticated than “hits”, top pages, and top referrers. At it’s most basic, analytics gives an idea of usage and most popular sections, and should be a given for all websites—I’m often surprised when it isn’t. And of course, analytics can tell you what is happening, but not why.

Tip: internal search engine reports are often overlooked but can be extremely useful, particularly the most popular search terms and failed searches (ie zero results). The former tells you what users think are the most important bits, and the latter tells you the content that is missing on your site.

Workplace observation

Involves going out into the field and watching people conduct their normal activities. This usually has no direct contact with participants; if it did it would be a contextual inquiry or shadowing. Even so, it’s usually a good idea to get explicit permission from your ‘client’ in case anybody gets suspicious of your lurking.

Tip: this method is very effective in a “front line” environment, such as call centres, bank branches, libraries etc where customers interact with the organisation (assuming it’s the organisations customers whom are the audience you’re researching).

Photo ethnography

A contextual inquiry is all well and good, but what if you can’t get out into the field, or can’t spend much time in the field? Putting the camera in the hands of those being studied so they can tell their own story. “Self reporting” like this has its issues, in that they may not do or report what you want, but this is also serendipitously good.

In the age of social media people often do this of their own accord, for example check out the “what’s in your bag?”, “my desk” or “corners of my home” groups on Flickr.

Cultural probes

Going beyond photo ethnography involves creating a “kit” that can be given to participants for them to record aspects of their lives. Also known as “diary studies”, probes are great for going where the researcher can’t, collecting data and sending it back. Typically a probe kit includes a diary, camera, voice recorder or a variety of other “devices”. Increasingly, though, online tools such as Tumblr, Posterous and Revelation are being used as digital probes or diaries.

Tip: a cultural probe should be combined with pre and post interviews to allow the researcher to properly explain what’s expected and then talk through what has been recorded. So from this point of view there may indeed be some direct contact, but the whole point of this method is for when there isn’t much contact with the participant.

Virtual ethnography

There are many ways in which you can have people participate in your research, virtually. A digital cultural probe, as described above, allows the researcher to track/watch explicit thoughts and activities that are deliberately recorded by the participant. But you can also monitor the implicit or unconscious activities which constitute their daily lives. This is a more passive, consumptive, observational approach that may collect more information than a cultural probe.

Specifically, this method involves monitoring blogs, discussion forums and social networking applications (what are people talking about? what are they asking each other?) as well as social media tools like Flickr, Twitter and Slideshare (what are they sharing?).

My colleague Chris Khalil spoke about his use of ‘virtual ethnography’ and online diaries in his UX Australia presentation The New Digital Ethnographer’s Toolkit: Capturing a Participant’s Lifestream.

Evaluative methods

Most of the methods presented above are exploratory or formative—they inform the design process—but there are also summative methods that are used to evaluate or validate designs (I also explained this in my recent presentation Prioritising User Experience). For example, you’ve probably heard of at least one of these: usability testing, eyetracking or Card Based Classification Evaluation (aka “tree testing”). I haven’t talked about them here because I wanted to focus on the exploratory methods, those that are classic user research.

Design research workshop

I cover all of the above methods, and more, in my design research workshop which was last run at this year’s UX Australia conference.

The workshop discusses user research methods in detail, but also covers:

• analysis of research data
• planning and executing research
• documentation of results
• plenty of practical examples
• group activities
• Q&A with me

Get in touch if you’re interested in adding this workshop to your conference programme or interested in running an in-house version for your team.

This entry was posted on Sunday, December 6th, 2009 at 3:40 pm and is filed under Design research, User experience. If you enjoyed this post, you might also like to leave a comment, find out more about me or follow @patrickkennedy on Twitter.

You can also trackback from your own site or follow any comments made on this post through the RSS 2.0 feed.