The workshop is a new version of one I’ve run a few times before, updated with new examples and activities. The presentation will give a short intro to a few research methods that are not so common in the IA field.

Hope to see you there!

No related posts.

I’m slowly and not-so-surely making my way through Good Thinking that I talked about in a recent post. It’s full of really great tips for researhers, not just with regards to methods and methodology but also in terms of mindset.

In a chapter discussing the psychology of small groups, the author talks about the effects of deception and economy with the truth, on the part of the researcher. In her opinion, and I agree, a good researcher must adopt a more respectful attitude with regard to the people they are researching:

It means a shift in perception from treating respondents as laboratory experiment ‘fodder’ or human guinea pigs to consenting adults who have points of view to contribute. It means letting go of the idea that the information the client is looking for can be extracted from respondents whether or not they give permission. It means challenging the view that both client and researcher have the right to be manipulative and controlling because they have paid respondents to attend the session, or have paid the researcher (or company) a large amount of money to retrieve information to solve a marketing problem.

This passage is followed by a ‘quote’ from David Ogilvy:

The consumer may or may not be your wife, husband, child or parent, but you can be sure that he or she is a real living, breathing, thinking and doing human being, who has as much of a right to their way of life as you have to yours.

I like this, and in fact I have used the sentiment embodied in the above quotes as the basis for a ‘principles of user research’ blurb in my research proposals.

It’s all to easy for us to forget that our work revolves around people, not answers. The very difficulty in getting the answers we want to inform our business decisions, should be telling us something: that people are complex and messy. And thus the approach we often need to take in research is equally complex and messy and (immediate) success is not always garaunteed.

I think it particularly important to set the scene and establish a certain mindset—and expectation—with the client. And it touches on many different areas of research, how participants are treated, how results are interpreted. It might sound a bit soft and ‘user biased’ but isn’t that what user research is all about, findings out what users think/do/feel/want/need/say? Of course it needs to be balanced with the realities of business (and in particular the limitations of technology) and I am always going on about balancing the raw user-centric view, but when it comes to user input, it should be conducted on their terms, with respect and consideration for them.

Even when they do things that are stupid, silly, strange or simply incomprehensible to us :)

[Image credit: "Geminoid F" by Yoshikazu Tsuno/AFP]

Related posts:

1. The fundamentals of human psychology applied to market research
2. Stop calling it usability testing
3. Choosing design research methods

Seeing as I firmly believe that for every unpublished blog post there is one less bit of momentum keeping the interwebs spinning, I’d better put this up. And it’s interesting to look back at what I wrote two and a half years ago…

Day 1 started with a very inspiring talk by Bill Buxton. I think this was just the thing the industry needs, a bit of a reality check and a wake-up call. Firstly usability evaluation is not design and for that reason most people here don’t actually practice User-Centred Design. It’s all about data, rules, strict methodologies, large companies. They’ve even turned agile into something overly defined and bogged down (I have no strong belief either way when it comes to agile methodologies by the way). Bill’s talk about sketching as an important tool for the design process flies in the face of the artefact centric practice many Usability Professionals follow. No there’s no template for it, no there’s no software tool to do it, you have to use your brain! I mean the theme of the conference (“patterns”) says it all really.

This sounds really negative, but I don’t want to be. There are some smart and talented people here, but overall the industry is weighed down by strictness and illusions. Strictness in the sense that many people want some methodology to tell them what to do. I can understand that, but as Bill said, if you find yourself thinking that all the time (being scared of wining it) then maybe this isn’t the job for you. Illusions in terms of the discrepancy between literature and practice. A lot of the things published are not followed in practice (eg rapid, flexible approaches by clever people are replaced by limited, templated projects) and good practice is not published (eg using multiple design alternatives in usability testing). Then there’s the illusions of grandeur, like the way many practitioners think of what they do as some kind of scientific crusade and admitting there is some I-don’t-know-ness to it is an act of heresy.

For me, the best thing I saw at the conference was this talk. It’s a pity someone from outside the field (perhaps technically but really as far as I am concerned he’s slap bang in the centre of what we should strive for) had to be the one to say it. You can’t truly be doing UCD if you’re just evaluating, testing and documenting. This shouldn’t be about statistical analysis techniques.

I remember thinking that my approach to my work seemed at odds with how other attendees appeared to be working, and from the above it seems this annoyed me! Too many practitioners being more worried about following the ‘proper’ process, rather than actually thinking. And the post I did publish at the time, contains similar thoughts.

Related posts:

1. Web Directions South 2008 day one
2. Intranet (re)design wrap-up
3. Strategically managing intranet developments

[William] Schlackman taught me the fundamentals of human psychology applied to market research: people will not share feelings and thoughts with a researcher if these are in any way embarrassing to self-esteem; that people always try to ‘look good’ (rational, intelligent, helpful); that some attitudes and emotions are difficult to put into words; that ‘rationalisations’, ‘ambivalence’, ‘justifications’, ‘projections’ are part of everyday human life and that integrity in one’s relationship with respondents is paramount.

This certainly reflects my experience with qualitative market/user/social research and I think it’s an excellent micro summary of basic tips for researchers.

Should have included this quote in my So you want to be a user researcher? post :)

Related posts:

1. Living, breathing, thinking and doing human beings
2. A summary of user research methods
3. Design research workshop at OZ-IA

Hi Patrick,

I’m Hans, a technical writer from Zurich and wanted to say a quick hello.

I happened to visit your website and liked the article The 7 Stages of User Frustration :)

I have a interest towards user research and have been reading books.

I’m looking out for projects (websites or applications) where I could help with user research and data analysis. I was wondering how to get started in terms of skills required. I know it is not easy for people to entrust such an important activity to me. Could you help me with your suggestions?

Thanks,
Hans

I thought about how I would respond, but since this isn’t the first time I’ve been asked this question, I figured it might be helpful to respond in the form of a blog post for everyone to see.

I’ll start by saying that this is my take on how I do my kind of user research and how I think someone could help get themselves where I am.

What is user research?

While I teach whole workshops on the subject of user research, in essence it really is very simple. User research is about understanding an audience; what they need, what they want, what they think, how they interact with other people, information and information systems such as websites. And perhaps most importantly, for all of the above, it’s about why.

It’s a mixture of market research, psychology, sociology, anthropology and business analysis. You see, people are like onions; lots of different layers. Which layers you look at will depend on the research context. What do you need to understand about your users in order to meet your objective?

At the end of the day, the purpose of user research is to understand the needs of the users and how those needs might be satisfied. It’s the second half of that last sentence that is quite pivotal; you can learn a lot about what makes the users tick, but for most practical applications we need to be able to then design something. Thus, the term “design research” is often used as a synonym for user research in fields such as interaction design and user experience (UX), which is the field I work in.

Skills

There are of course many user research methods for you to master (which is no small accomplishment by the way). However, there are other skills you need as a user researcher.

I think the most important “skill” is to have an interest in discovering, learning, probing deeper. I’m not a strict methodologist, I rarely follow a rigid process for my work, but the curiosity and desire to discover the what, how and why has been an indispensable feather in my cap. You have to want to dig beneath the surface and find out about the people you’re researching. They’re not a faceless, anonymous, uniform mass of “users”. They don’t think like you, they don’t think like your organisation (or client) wishes they would, they don’t act the way that would be convenient for the UI designer or system architect. (Unless of course they do, but you won’t know that unless you go and find out.)

Facilitation and public speaking skills and very helpful. You need to be confident and commanding when necessary, such as when moderating a focus group, but also quiet and a good listener, such as when interviewing or observing. Confidence can be acquired, but being quiet and observant, not so easily. That’s why I think introverts make good researchers, they can sit and watch and listen, taking it all in rather than needing to talk a lot. However this may be me post-rationalising!

I’m good at taking notice of things about people and remembering faces (and the people attached to them!). I think I can “read” people well. And I’m pretty good at seeing connections between various traits, and indeed between people, identifying patterns in their personalities and behaviour. This is not to say I’m the world’s best researcher, just that I find these have been useful traits.

Learning from other fields

As you can see from my definition of user research above, I see it as having very fuzzy boundaries, with lots of overlap with other fields. This means there’s much to be learnt from people in other disciplines. This includes anthropologists, psychologists, sociologists, journalists, market researchers, technical writers, business analysts (BA) and industrial designers.

Research at its most basic, is about gaining insight. You choose something you want to better understand, then you go off and find out what you can about it. As you’d imagine this concept is pretty widespread, so you’ll find research expertise in lots of different places.

You can borrow techniques from these other fields, but probably more important when you’re starting out is being able to watch and learn from them. Shadow them, or perhaps some kind of mentoring relationship might work.

You then need to translate what you learn from these other fields and apply it to user research. This cross pollination has proven very useful for me.

Book learning

The overlap with other disciplines means there are lots of sources of information and insight. As Hans mentions, you can start by reading (actually not just start but all the way through your career).

Two good books to get you started with user research are:

• Don’t Make Me Think! by Steve Krug
• Observing the User Experience by Mike Kuniavsky

The first gives you the right attitude and approach to take, the second gives practical details on many research techniques.

Each of the disciplines mentioned in the previous section will no doubt have their own community, whether it be online or off. These communities are usually a great source of recommendations on books to read, but also for things like education, training, consultants. Look for mailing lists (such as anthrodesign) or face-to-face groups (such as UX book club).

I find these communities can be incredibly useful, especially if you’re new to their field and want to get up to speed. When I was getting established in this field, I made sure I did as many of these things as possible. I met people, talked about stuff, became part of the community.

Formal education

Is this necessary? I’m really in two minds about this topic. I certainly think that you can be a great user researcher but not have any formal education. Yet there are clear benefits to such an education. Everyone interprets and makes sense of what they see in research, from their own perspective—which is why it’s a good idea to have multiple researchers. I believe it’s useful to have an academic standpoint underpinning this perspective. For example, anthropology has several schools of thought that provide frameworks for analysing observed behaviour. This is a definite benefit, even if they reject that standpoint.

So, whether you want to do a university course is up to you. If you’re like me you won’t be able to find a degree or masters program that is focused enough, or flexible enough, to give you just what you want without lots of what you don’t want. In that case, more targeted training courses are the way to go. For example, earlier this year I did a one day course on multivariate data analysis to brush up on quantitative techniques. The course was part of the Australian Market and Social Research Society‘s (AMSRS) winter school, and a good example of utilising a complimentary field.

Other short courses you might want to look into, could include: facilitation skills, active listening, “story listening”, mentoring/coaching as well as courses specific to particular research methods.

Out-sourcing

Consultants and freelancers can be hired to get projects done, but you can also learn from them. You might go so far as to get them in purely to act as mentors. Some organisations have mentoring programs, such as the IA Institute. It is becoming more popular for consultancies who operate in this space, to offer mentoring as a service as opposed to just doing the work for their clients.

Going guerrilla

I don’t mean you should start wearing a Che t-shirt and ambushing your boss with demands for social reform and more involvement in user research projects, although that may very well work for you. What I mean is that there are some less traditional ways of building expertise and experience, to gain a foot hold in the field, whilst keeping your day job.

• One could be the literally use so-called “guerrilla” techniques: cheaper and less formal research methods. For example, doing research in-house rather than recruiting external participants. This might be enough to allow you to add on research to current projects or fit it in during “down time”.
• If your organisation already has a user research or UX capability, you might want to shadow a team member who is a more experienced researcher, this will be a good way to learn and start to get involved.
• You could start moonlighting by doing work on the side for your own clients, friends or family. I did this when I was really breaking through, supplementing my web design services with UX and user research.
• You could take part in crowdsource projects such as Drupal 7 UX.

Gaining trust

It’s interesting that Hans says “it’s not easy for people to entrust such an important activity to me”. First of all, if you’re in a position where your colleagues (especially senior stakeholders) think research is that important, then that’s an excellent start! Possibly the biggest problem with web UX is a lack of understanding or value in research. More often than not we have to fight to get the time and resources to do it (or do enough of it).

To answer Hans’ question though, their trust in you personally is something you can work on. Using some of the techniques I’ve described below, you can build up expertise and visibility which should convince others you can be trusted with the task.

Mixed methods

In terms of building trust when you’re doing user research, make use of triangulation or a “mixed method” approach. This essentially means using a few different forms of research to help balance out the risks associated with each. This gives increased confidence in the results, and thus in you.

For example, you could back up your user interviews with some third party statistics that related to the topic being studied. Secondary research of this nature, particularly when it is sourced from big players (eg academic or big consultancies like Forrester, Nielsen, Roy Morgan, NN/g) can really bolster confidence in your work and make important stakeholders into believers rather than ardent sceptics. Doing this might also help you build bridges with other departments such as marketing.

Starting from scratch

If your organisation has no UX or user research capability, you’ll need to engage in some self promotion and reputation building. Here are my tips:

• Show initiative by willing to learn (eg try things yourself, do training).
• Make yourself the resident expert on user research.
• Write book reports or do “brown bag” presentations to your colleagues on the books you’ve read.
• Create an internal newsletter or blog that allows you to send around links to pertinent articles. Compile a persuasive argument for why user research should be done, and for you to be involved.
• Represent your organisation at conferences, getting to know the experts in the field. Bring that knowledge back to your organisation, but also let everyone else know your organisation is serious about user research and that you’re the go-to guy (or gal).
• Make yourself the liaison for any external parties doing user research work (consultants, contractors etc).
• Invite guest speakers to come in and give a talk about UX and user research to your organisation. Make sure you’re the person they deal with.

All of this will, over time, mean your colleagues start to think of you when they think of user research. You can convert this association into permission; permission to get out there and do some research!

Then, do whatever small pieces of research you can, but make suggestions about further research that could be done and the benefits that this would bring—such as increased confidence in the results, increased clarity, increased depth or richness of information. This lets those who make the decisions know what the possibilities are. Back this up with a fairly concrete plan for the work: how much will it cost? how long will it take? how will it improve the work that is already being done?

If you’re a UX practitioner

If you’re a UX practitioner who doesn’t have much experience in doing research, I would recommend:

• Don’t get too hooked up on methodology.
• Let them talk! Don’t hound your research participants for feedback on a specific usability issue.
• Think about useful as well as usable, what does your audience need and what will they use?
• Use activities and design games to get info from users without grilling them.
• It’s not (just) usability testing, though that might form part of your research.
• Think critically, why has that person said that or reacted that way?

How I got started in user research

My first piece of user research involved card sorting. They were graphic cards, rather than text, showing different types of cars. This allowed me to get an idea of how people who didn’t know a lot about cars would classify the different styles (or “body shapes”) and how they might go about choosing the car that was right for them. This worked very well and allowed the design team to challenge the client’s view of how the vehicles in their range should be organised on the website. Their view was very much according to the automotive industry view of things which whilst technically correct, was at odds with how your average punter sees things. Using our research, we could think about better ways of communicating with website users.

It was far from a perfect project, I was operating with very little resources and looking back on it I can see there were many things I would differently now to improve the method. Yet the results were very useful and this led me to continue my shift from general web design into the field of UX and user research.

I took it upon myself to do this work—which I would now call design research—to help create the IA and UI. Nobody else in the team was doing it and I knew enough about cars to know that not everyone else knows that much about cars, so we needed to explore how regular people would group them and what they would call each group. Hence the card sorting.

I recruited participants within my company, I ran the sessions largely in my own time and learnt a lot of things as I went. I also read Krug.

I remember writing out scripts for exactly what I was going to say and setting up a makeshift observation room by using a video camera connected to a TV set by a very long cable (to no avail since nobody turned up to watch!).

Despite its shortcomings, all of this was noticed by those running the project, and by senior management. Soon “usability” made a regular appearance on client proposals, capability statements and project plans. I was encouraged to learn more and to improve the quality of what the team was producing.

Later in may career I was working as a consultant, which almost by necessity means I had to investigate and analyse before compiling recommendations—which is, in essence, research. I was spending a lot of time undertaking various kinds of research (user research, stakeholder research, usability testing etc) and less time designing and implementing, which meant a lot of practice with techniques such as interviewing, groups workshops, card sorting.

From then on, it’s been a matter of doing more projects; refining my approach, practising techniques, trying new methods and tools.

Above all, my ultimate piece of advice would be to just jump in and have a go; it’s easier than you probably think!

Related posts:

1. Talk to your staff
2. Guerrilla user recruitment
3. Research methods workshop at UX Australia

In this article I give a quick overview of the methods I commonly use, broken down in to main categories:

• Direct user contact—where the researcher does very much interact with users, or members of the audience as I prefer to call them
• Indirect user contact—where the researcher does not actually interact with members of the audience

The list is by no means exhaustive but it gives a good indication of the breadth and depth of methods that can be employed in user research.

Direct user contact

Interviews

Interviews done with actual users are the mainstay of user research, as far as I’m concerned. Such an interview (also sometimes known as an IDI or “individual depth interview”) should ideally be conducted in the participants’ normal context, or environment in which they normally undertake the activity you’re studying. You can have a set of questions to guide you—called “semi-structured” interviews—or you could keep it completely open—called “unstructured”.

Tip: another potentially useful variation is known as the proxy interview (also known as ‘key informant’ or ‘intermediary’ interviews).

Focus groups

Small group discussions around one or more topics, often used in (and given a bad name by) market research but usually with a different purpose in mind and using a different approach. They are good for idea generation, brainstorming, comparing alternative designs. The social interaction can elicit more information than if you interviewed each person individually. However, they’re not so good because you can’t see how people do what they do, only what they say they do. Managing the group can be difficult and peer pressure can affect the answers given (known as “group think”).

Tip: storytelling sessions (aka “anecdote circles”) are a good alternative to a traditional focus group. These techniques are gaining popularity in management and business collaboration fields.

Workshops

Workshops are a good framework for a variety of smaller research methods that can be strung together to make the most of your time with participants. Almost any “workshop activity” can be used, depending on what it is you’re trying to achieve; I’ve listed two below, but you can find more at design games.

Tip: most of my “focus groups” are actually workshops, but the former is a term better understood by stakeholders.

Card sorting

Whilst normally used as an IA design technique, card sorting is an excellent way to research an audience’s mental models and terminology, particularly as part of a workshop. For instance, what do they call XYZ? where is there disagreement on the classification of content/things? what is the organisation’s mental model? (when running a stakeholder workshop). In a group situation, pay attention to the discussion and debate, not just the obvious output (the “quantitative taxonomic data”).

Tip: regarding that taxonomic data, you can use Donna Spencer’s card sort analysis spreadsheet to help see interesting patterns.

Product Reaction Cards

One workshop activity I’m quite fond of is the Microsoft Product Reaction Cards. You get users (either individually or as a group) to choose the words they feel reflect a certain topic, from a set of pre-defined words. For example: “what words best describe the current website?” or “how would you like the intranet to be in the future?”.

Tip: initiate a “talk aloud” protocol to gain some insight into the participants’ thoughts and what they mean by each word. Asking them to pick their top 5 words can help them really focus in on what they think.

Contextual inquiry

A contextual inquiry (or contextual enquiry if you prefer) is essentially an unstructured interview in the context in which users use the system. The researcher acts as the student and learns everything they can from the user (eg how they perform their job, what tools they use). As with a lot of direct contact research methods, it’s not so much about what you find out about a particular product/system/website/application, but more about getting access to people you know are your audience. You want to learn about what makes them tick, who they are beyond a one-dimensional demographic breakdown—which is so often how the audience is seen by organisations.

Visual Anthropology

Taking photos in the field is an excellent way to document the environment and behaviour of your audience. Documentaries, many of which could be called “Ethnographic films”, are a natural extension of this concept. They are a rich and emotive form of communication; particularly useful when presenting research findings to project stakeholders. This would usually be combined with, or part of, a wider study.

Some great examples of visual anthropology in the form of film have been published on the web by Michael Wesch, such as A Vision of Students Today.

Shadowing

Like a contextual inquiry, only on the move. The researcher basically follows his or her participants around as they undertake tasks or behaviour that is being studied (and probably quite a bit that is not!). Obviously you need to get their permission, otherwise it’s just called stalking.

I’ve heard a number of other names for this method, depending on who it is that’s describing it; for example journalists often call this a “ride-along”, as in “I rode along with the emergency services crew to get a first hand view of the effects of binge drinking on our city’s streets”.

Indirect user contact

Content inventory

A key research method for understanding content, also commonly known as a “content audit”. You develop an inventory of all pages, making note of meta information such as content and functionality, content owners, the status of the content (out-of-date etc). The output is useful for inputting into card sorting and content migration activities.

Heuristic evaluation

A person assesses the system (eg website) using a list of heuristics, or guidelines, based on usability best practice and design conventions. This is possibly the most common (and first) method of research undertaken by many user experience practitioners.

A formal heuristic evaluation involves the use of a comprehensive checklist of heuristics used by multiple assessors who score the system individually, for methodological validity. However, this approach is rarely taken in my experience; most heuristic reviews are less formal and more akin to an “expert review” of the system, so much so that the two names for this method are used interchangeably.

Competitor review

The competitors, or peers, of the system are reviewed to compare against each other and the system being (re)designed. This would usually use a heuristic review as the basis. The method is useful for setting a benchmark but also for collecting design ideas and identifying opportunities.

Questionnaires and surveys

Technically, questionnaires are self-reported (and thus indirect) whereas surveys are conducted by the researcher (and thus direct contact), but the terms are often used interchangeably. Most user research tends to use the self-reported form.

Either way, this method is good for collecting bulk responses (it’s usually a quantitative method) and for comparison of results over time. Among the reasons why this method is not so good if the fact that it can be very difficult to create questions that respondents understand and answer in the way you intend.

Tip: I often use a “survey” to quantify the findings from qualitative research. After having done in-depth qualitative research using a combination of the methods outlined above, some of these findings can be extracted and a much larger sample size of people can be quizzed to get some numbers around the effect/opinion/behaviour observed in the qualitative research.

Panels

Like an online survey that is given to a specific set of people, who have been recruited for the purposes of research. This method, commonly used for market research, can be an excellent way of getting input from a large sample of your audience. Many researchers are now using panels in a manner similar to a virtual focus group, with more “live” interaction being added to the traditional panel method (which is usually not real-time).

Tip: while the cost of a panel can be relatively expensive, it can be a good replacement for a survey to help quantify qualitative research findings with a larger sample size, assuming you can get enough of the right kind of people on the panel.

Analytics

Also called “web analytics”, “web stats” or “traffic statistics”. This refers to measuring key metrics about what users of a website or application are actually doing; hopefully something more sophisticated than “hits”, top pages, and top referrers. At it’s most basic, analytics gives an idea of usage and most popular sections, and should be a given for all websites—I’m often surprised when it isn’t. And of course, analytics can tell you what is happening, but not why.

Tip: internal search engine reports are often overlooked but can be extremely useful, particularly the most popular search terms and failed searches (ie zero results). The former tells you what users think are the most important bits, and the latter tells you the content that is missing on your site.

Workplace observation

Involves going out into the field and watching people conduct their normal activities. This usually has no direct contact with participants; if it did it would be a contextual inquiry or shadowing. Even so, it’s usually a good idea to get explicit permission from your ‘client’ in case anybody gets suspicious of your lurking.

Tip: this method is very effective in a “front line” environment, such as call centres, bank branches, libraries etc where customers interact with the organisation (assuming it’s the organisations customers whom are the audience you’re researching).

Photo ethnography

A contextual inquiry is all well and good, but what if you can’t get out into the field, or can’t spend much time in the field? Putting the camera in the hands of those being studied so they can tell their own story. “Self reporting” like this has its issues, in that they may not do or report what you want, but this is also serendipitously good.

In the age of social media people often do this of their own accord, for example check out the “what’s in your bag?”, “my desk” or “corners of my home” groups on Flickr.

Cultural probes

Going beyond photo ethnography involves creating a “kit” that can be given to participants for them to record aspects of their lives. Also known as “diary studies”, probes are great for going where the researcher can’t, collecting data and sending it back. Typically a probe kit includes a diary, camera, voice recorder or a variety of other “devices”. Increasingly, though, online tools such as Tumblr, Posterous and Revelation are being used as digital probes or diaries.

Tip: a cultural probe should be combined with pre and post interviews to allow the researcher to properly explain what’s expected and then talk through what has been recorded. So from this point of view there may indeed be some direct contact, but the whole point of this method is for when there isn’t much contact with the participant.

Virtual ethnography

There are many ways in which you can have people participate in your research, virtually. A digital cultural probe, as described above, allows the researcher to track/watch explicit thoughts and activities that are deliberately recorded by the participant. But you can also monitor the implicit or unconscious activities which constitute their daily lives. This is a more passive, consumptive, observational approach that may collect more information than a cultural probe.

Specifically, this method involves monitoring blogs, discussion forums and social networking applications (what are people talking about? what are they asking each other?) as well as social media tools like Flickr, Twitter and Slideshare (what are they sharing?).

My colleague Chris Khalil spoke about his use of ‘virtual ethnography’ and online diaries in his UX Australia presentation The New Digital Ethnographer’s Toolkit: Capturing a Participant’s Lifestream.

Evaluative methods

Most of the methods presented above are exploratory or formative—they inform the design process—but there are also summative methods that are used to evaluate or validate designs (I also explained this in my recent presentation Prioritising User Experience). For example, you’ve probably heard of at least one of these: usability testing, eyetracking or Card Based Classification Evaluation (aka “tree testing”). I haven’t talked about them here because I wanted to focus on the exploratory methods, those that are classic user research.

Design research workshop

I cover all of the above methods, and more, in my design research workshop which was last run at this year’s UX Australia conference.

The workshop discusses user research methods in detail, but also covers:

• analysis of research data
• planning and executing research
• documentation of results
• plenty of practical examples
• group activities
• Q&A with me

Get in touch if you’re interested in adding this workshop to your conference programme or interested in running an in-house version for your team.

Related posts:

1. Choosing design research methods
2. Design research workshop at OZ-IA
3. Research methods workshop at UX Australia

Unfortunately I couldn’t make it for the second day of the conference due to family commitments, which is a shame since the programme looked even better for the Saturday!

Our presentation on our new SuperRacing site (not yet live at the time of writing thissite is now live) went down well considering we had to cram everything we wanted to talk about into 25 minutes. The slides are shown below.

After two conference presentations in one week, it’ll be a while before I make another appearance, not to mention that I’ll be pretty brain dead for a while after the new baby arrives.

Related posts:

1. Design research presentation at Oz-IA 2009

I tried to make the topic a bit more interesting (read controversial) and I think it went quite well.

I covered two main topics, firstly I outlined why I think user experience (UX) should be prioritised over information architecture (IA), and then I discussed prioritising UX within the organisation.

IA vs UX

They’re the same thing aren’t they? I have certainly said as much in the past. I’ve also made distinctions between them. And depending on the level you’re looking at, both of these views hold true.

At a very high level, both IA and UX are forms of user-centred design (UCD) and would be more similar to each other than either would be to, say, cooking. And of course many people consider IA and UX to be synonymous (as well as a whole host of other terms such as IxD, ID etc). But for the purposes of this discussion, let’s define them as being quite different from each other.

What is IA?

A fairly narrow definition of IA might be something like the design of structure, navigation and labelling of a website or other information system. In other words, it’s very product-centric; it’s about the user interface, the system, the thing we’re designing and making usable. No doubt many people would argue with this definition or want to offer their own definition, and that’s ok. The point I’m trying to make is that if one was to take quite a low level definition of IA—one that is focussed on the interface—then you’re leaving out quite a bit!

What is UX?

The holistic user experience is made up of many different factors, and many different components. It’s more than just the IA, it’s more than just the UI, it’s more than even the product itself. It’s all the things that might make up the experience someone has with, or around, a product, service or organisation. A good UX will satisfy your audience, and keep them coming back.

If we expand this definition just slightly, you can see why UX is starting to merge into the area of Service Design and Customer Experience. We’re talking about multiple touch points and multiple media, not just the technical elements of UX in the Jesse James Garrett sense.

The contrast with the above definition of IA should be pretty clear.

Why UX over IA?

The purpose of this discussion is not to criticise IA (as defined above) or those that do not distinguish it from UX, but rather to highlight what we should be focussing our attention on. We should be focussing on the overall experience, not the individual aspects of the design of the a product. This includes the IA, but also the visual design, technical design and usability. Yes that’s right, usability is not the most important factor. I believe usefulness, appropriateness and the overall experience are much more important than usability.

So when we talk about prioritising UX, we’re talking about making sure what we’re building is useful for the people we are building it for. Does it make some aspect of their life easier/faster/better? Does it fit in with the rest of their lives in an appropriate and beneficial way?

When I say appropriateness I mean fitting a user’s “ecosystem”. You ecosystem is the (typically large) variety of sources from which you gather information that allows you to perform some function or role (Sonnenwald uses the term information horizons to describe a similar concept). These sources are often offline and used with little loyalty, killing the myth of the “one stop shop” that users will come back to like lemmings. This is a very useful perspective to take as a sense check for usefulness and appropriateness of proposed solutions. For instance, we might be tasked with designing a website, yet does that make sense? Not just ‘can we build a website’ but how realistic is it to propose the audience goes to a website to perform the task or action? Sometimes it just doesn’t add up and you need to re-think how you can sensibly be part of their ecosystem.

Of course there are times when a new product or service—even something as intangible as a website—can have a marked impact on people’s lives; change their ecosystem, change their habits, change the way they see the world. Good examples include Google, the iPod, Tivo and mobile phones; people do do things differently now because of the impact of those things. But I tend to think of these successes as serendipitous rather than calculated. Over time we have changed in response to products/services/technologies that could probably not have been predicted, let alone planned.

So to expect to design and launch a life changing product/service/technology is probably being a tad optimistic. Yet that is so often the reason given for ignoring the audience’s ecosystem: “this will be a market killer!”. It’s not that this doesn’t happen by design, just that only a small percentage of us are in this league. And it takes a hell of a lot of hard work to do this, it’s not as easy as saying your pride and joy will be a “one stop shop”. If you build it, they won’t necessarily come.

Ok back to the point, which is that in almost all cases you need to fit into your audience’s ecosystem and provide something useful, otherwise it doesn’t matter how pretty, clever or usable your product/service/technology is.

Hence I believe that we (those that might be willing to undertake this IA thing) can easily miss the boat if we’re not careful, and forget to meet our true, if perhaps implicit, objective which is to meet the needs of real people in the real world. If you’re just making it usable you’re not going far enough. If you’re just doing IA then you’re not doing enough. Similarly, focussing just on usability will ultimately fail.

Crucially you must understand who your audience is and what they need, want and do, before you can have any hope of creating something that is useful and that fits into their ecosystem in an appropriate way. You need to undertake research to find out these things.

Testing vs research

There is “usability testing” and then there is “user research”. These two activities are often confused and before we continue, let’s clarify.

User research is generative, exploratory and formative. It’s aim is to gain insight and inform the design process. That said, it doesn’t have to happen prior to or disconnected from the design process, just that it’s uncovering information that goes into design. In fact, this kind of research can continue well into the design process, especially if an iterative approach is being taken.

Usability testing on the other hand is evaluative, validitory and summative. It’s aim is to assess the outputs of the design process. That said, it doesn’t have to happen after the design process is finished, just that you’re taking stuff you’ve created and seeing how well it works, on many different levels. In fact, testing can take place from any point when there is something produced by the design process that you want to check or validate. Again, this is especially the case if an iterative process is being taken. (Oh and usability testing is not UAT)

The main similarity between the two is that they involve, nay require, user involvement.

On a practical level, there are many methods or techniques that are the same or similar between research and testing. The way you use them, or the value you extract from them, might be different, however. For example, interviews and focus groups are commonplace research methods, but they can also be used to “test” concepts, ideas and an understanding of the audience’s needs. Likewise, the “talk aloud” methodology typical of usability testing can be used as a research tool to learn about the audience and gain insight into how they use information systems, almost disregarding feedback on the actual user interface users are shown.

But what we’re talking about here is the research side of things; generating insight that informs design, and strategy.

Understanding users

A large proportion of the insight you’re gaining when you undertake user research is an understanding of users, or the audience as I prefer to call them. For example the kinds of question we might aim to answer include:

• Who is our audience?
• What are their goals, attitudes and behaviours?
• What are their information needs?

How do you discover all of this? Well there are a variety of methods you can use, ranging from face-to-face interviews and focus groups, through to “virtual ethnography” where you monitor the digital footprints of your audience to see what they’re doing online. And everything in between. There’s plenty of information on all these methods, just Google it, or you could attend one of my user research methods workshops.

Informing IA

Once we’ve got this understanding of our users, how do we use it to inform our UX strategy? And how does this trickle down to structuring the IA? Both excellent questions.

Let’s start with the strategy. Putting it simply, strategy is deciding “what to do” at a big picture level. User research can help with this in two ways: ensuring what you’re going to do is what people want, need and will use, but also ensuring that you going to do it to the right people. Once you have that sorted you would be able to go on to the IA.

Whilst at one level we want to ensure we focus on the UX, the useful, if we’re designing a website, intranet or other kind of information system, we will need to create an IA at some point. This is where you can get more specific and look at those three basics of IA I mentioned before: structure, navigation and labelling. Your user research can inform each one of these.

The structure of the information is something you can easily gain insight into through research. You might use card sorting to uncover how your audience see the content fitting together or you might get feedback on an existing categorisation scheme.

Navigation is closely related to both structure and labelling, however, a key research finding for navigation might be the outcomes of a task analysis that tells you the process users follow and on which you may want to model your navigation.

Labelling involves naming parts of your IA in a way which will allow your users to correctly understand what content or functionality is identified by that label. To make the IA as usable and meaningful as possible, the labelling used should reflect the words/terminology/jargon that the audience uses. A simple example can be found in most corporate organisations where the official vernacular might include the term “personal leave” but staff refer to it simply as “taking holidays”.

As such, going hand in hand with learning the language of users, you may need to unlearn the official language of the organisation for whom you are producing the IA. Or at least bridge the gap by somehow using both terms and establishing a process for migrating (“educating”) the audience, if that is appropriate.

I find that this quickly falls out of the research, because in order to speak with people effectively you have to arrive at a point where there is a common language between the researcher and the researchee. Ideally it is the researcher who should be actively bridging the gap and adapting their terminology to match the user. By doing this you need to recognise what words and phrases users use to refer to the problem space. If the subject you are researching has specific jargon associated with it, then this is something you need to pick up, and in most situations it would make sense to use that jargon when labelling the IA (assuming that the audience is familiar with that subject). The bottom line is: speak your audience’s language.

While research can inform the IA design process, it won’t necessarily give you the answers. The hard part is still to come up with a solution, which through no fault of your own might not be perfect. A great deal of compromise is often involved, whether it be regarding what can be implemented within the budget available, or a lack of support for taking on board the results of the research.

Encouraging user involvement

Ok user research is what we need. How do we get them involved? Encouraging users to get involved is relatively easy. Costs can be high if you outsource recruitment to a market research firm, but there are ways to cut this almost to zero if you’re willing to do some of the heavy lifting yourself. Such as, recruiting participants yourself through internal connections or perhaps using a survey or poll on your website (eg Ethnio).

You can even just grab people in the hallway, run concepts or sketches past them, rather than go for full on usability testing in a lab. The approach really depends on your objectives and available resources.

To help encourage this user involvement you should offer a form of incentive, either cash or some other compensation. But often the audience are keen to be involved, either because they want to help improve the product or service, or because they want to let you know how good/bad it is. Of course if you’re recruiting non-users or non-customers then this won’t be a factor, hence it’s usually a good idea to include both users and non-users in your research to see both sides of the fence.

Even before recruiting for a specific piece of research or testing, you should be open to feedback and suggestions from your audience. This might mean asking for feedback on your website, through an email address or a tool such as UserVoice. Or you could sign up to Twitter and listen to you users/customers. Or maybe you can just stop hiding your customer service phone number in the bowels of your website (in 3pt text).

No matter which approach to recruitment and research you take, finding or making opportunities to get in front of users is relatively easy. Encouraging your designers, developers and managers to allow user involvement is somewhat harder.

Many people (including Dan Szuc, Joel Flom, Jared Spool and Bruce Temkin) have talked about the idea of an organisation requiring a certain level of maturity before things like user centred design (or “service design” or “design thinking”) can take place. And user involvement is definitely one area in which this is quite evident. Sometimes an organisation or a team just aren’t ready to have user involvement.

It could be that they’re too focussed on establishing the business and getting products out the door; refining the user experience just isn’t a priority.

It could be because they are scared of what users might say (this is also a very common reason for the resistance to social media that many companies have).

It could be that the team(s) involved do not want to relinquish authority over the product or its design. They may feel threatened by users having input, because their colleagues might start to question their worth.

It could be that reward and recognition in the organisation comes not through creating the best product or service, but from cost-cutting or just doing what the boss says.

It could simply be because of the time and cost. It’s a popular view that everything slows down if we have to go and do user research. Or my favourite teeth-clencher: “this is an Agile project, so we don’t have time for research”.

The solution? IF you can sell the idea of user experience as a crucial part of the business, then the user involvement becomes a no brainer. You have to research and you have to test if you want to nail the user experience, and IF the organisation sees that bad user experience means loss of revenue, lower loyalty and higher costs then they’re going to want to make time and budget for those activities. Even in an Agile world.

Those are, of course, rather large IFs. Selling the idea of UX being crucial to business success is much easier said than done. Even in companies whose sole business is products and services that rely fundamentally on a digital user interface (websites, software, mobile etc), where you would think it’s be an easier sell. Alas, it isn’t.

This has become a hot topic in recent years, with excellent advice coming from such wise people as Dan Szuc and John S. Rhodes, to name but two. I’m not going to repeat all of their great suggestions for prioritising UX, but rather I’ll discuss a few tactics that I have seen work.

• Gain their belief – don’t expect respect and understanding, rather “gain their belief” (paraphrased from Mark Schenk’s great post on successful leadership). You’ll do this by showing you’re effective; make some quick wins, get a few runs on the board, prove why you should be given the time/budget/resources you need. This requires choosing the right projects to demonstrate value and impact.
• Communicate clearly – as I said before with reference to labelling, ditch the pseudo-scientific jargon and geeky terminology and speak the language of your audience, which in the case of selling UX is likely to be business and management types. This will probably require you to drop the ego (something which will work wonders with all the suggestion I’m putting forward here).
• Listen to the business – they might just tell you what they’re looking to get out of your relationship, thus giving you something to aim for and hopefully exceed. You may need to study up a bit in order to understand them and what they do; get your business groove on!
• Make deliverables visible – many products of UX work are very useful for (some might even say their sole purpose is for) attracting attention and generating discussion around the work you’re doing. Personas and concept models are two types of deliverables that immediately some to mind. Stick them on the wall in a high traffic part of the office. Make sure it’s clear who created them and that they welcome feedback. You don’t need to go so far as the life-size cardboard cut-out personas some organisations have made, but a nicely presented A3 poster that clearly communicates an idea or concept will do wonders for the visibility of the UX team and the work they do.
• Tell stories – they are a great way of breathing life into what can be a rather dry subject, tell stories of your success but also of UX challenges. The latter works in the same way as that old usability cliche: show management a video of usability testing where the user becomes frustrated by the product and they will probably get on board the usability train pretty quickly. Better yet invite key stakeholders to research and testing sessions. It can be tough letting them watch but you’ll need to get past that if you really want them to take you and your work seriously.
• Collaborate – break down the silos, drop the ego (again) and look for other parts of the business who have overlapping or complimentary skills and capabilities. A UX “Community of Practice” can be a good, low cost way to kick off this co-operation between different parties with an interest in UX. In many ways such a community approach can be much more effective than a dedicated UX team going it alone. And the cross-pollination effect can be a real bonus, not only in terms of gaining knowledge and skills from other teams but also because their reputation, legitimacy and respect can rub off on you, hopefully in a positive way.
• Be passionate – if you’re boring and act like the situation is dire, then why on earth would anyone want to encourage or support you?
• Find executive champions – I put this one last because ultimately what all these ideas are pointing towards is getting someone with authority on board and help you change the way the organisation functions (to a certain extent). Look for like-minded or sympathetic people who “get it” throughout the organisation, use them to help spread the word and build a case for doing things the right way. Usually these allies will be the result of demonstrating value through a project (see the first point above) but they can also be the result of networking performed “off the clock” so to speak. A born networker is thus a valuable asset for the UX team. If nothing else they can give the UX “elevator pitch” when an executive is more susceptible down at the pub.

Related posts:

1. Putting the cart before the horse
2. Talk to your staff
3. Stop calling it usability testing

It’s all very hush hush at the moment because we’re operating under an embargo, but the general gist of it is how user research can inform the design process for a product (in this case a website) that pertains to an activity that is normally an offline experience.

We’ll talk about how we identified the audience’s “information ecosystem” and found an opportunity for our website to fit in with that in a useful way.

We’ll also talk about the very different groups within that audience and how we catered for each.

Alun will then talk about the transition into design and go through the process we followed.

And if all goes to plan we’ll be able to unveil the new website resulting from all this work!

All this in 25 minutes, including questions :)

Update: if you promise to come along and watch our preso you can use this discount code: PK0265. Use it to get $100 off when you register for OZ-IA.

Related posts:

1. Oz-IA 2009

All research methods have their pros and cons, the problem comes when you rely on just one method. I’m often disappointed when UX and IxD practitioners describe the research they do, and it’s obviously very one dimensional. They only do surveys, for example. Or they only do usability testing at the end of the project (it’s quite alarming but this practice does continue).

…

This is where the concept of “triangulation” comes into its own. Also known as “mixed method” research, triangulation is the act of combining several research methods to study one thing. They overlap each other somewhat, being complimentary at times, contrary at others. This has the effect of balancing each method out and giving a richer and hopefully truer account.

This will of course be one of the many topics I will be covering in my full day workshop on design research methods for UX practitioners at UX Australia 2009 – a 3-day user experience design conference to be held next week (26-28 August 2009) in Canberra (Australia).

Related posts:

1. Research methods workshop at UX Australia
2. Giving eyeballs the chop
3. Design research workshop at OZ-IA