Stop calling it usability testing
Now that I’ve got your attention, let me clarify what I mean. When we refer to this activity called “usability testing” there are often a lot of misunderstandings. It’s really not very applicable for the thing we should be doing. Here are some reasons why…
- It gets mistaken for UAT.
- It gets mistaken for technical testing.
- It makes it sound more ‘scientific’ than it (usually) is.
Usability testing is not UAT
UAT, or User Acceptance Testing, is a term used in software engineering to describe when the client would give final approval for the built system to be delivered. This becomes really confusing when the term is taken out of that context and used in web design, for example. When you don’t understand the “user” it refers to and what is being “accepted”, you might very well think UAT is the same thing as “usability testing”.
Conceptually, the intention of UAT might be to make sure those who will ultimately be using the system are happy with it. But in practice, this is rarely the case. I could go on, but this is not a discussion of why I don’t like UAT (which I don’t) but rather my point is that we don’t want stakeholders thinking that usability testing is UAT, and thus something that can be dispensed with because “we always go through UAT”.
Usability testing is not done by Mr Test Manager
The term “usability testing” often gets misconstrued by technical types, project managers and business analysts. It gets turned into a stale, rigid, bureaucratic affair. The old “unit, integration, system” mantra. It’s done as a matter of course, at the end of the gantt chart, to tick a box. That’s pointless.
Again, in theory, test driven design is not a bad thing. Software, websites and anything technically complex should be checked to make sure it has been built as was required. That assumes a lot though, for example are the requirements valid? do users actually want or need what is being built? But let’s leave that one alone.
What I’m trying to say, is that usability testing shouldn’t be mistaken for the technical testing done by a Test Manager according to a test plan, using a test script. (At least not the kind of usability testing I want to be doing, which is possibly a qualification I should have stated up-front). People do run highly structured usability tests—typically summative in nature—which are very similar to technical testing. In my experience this is the minority of cases and the least valuable. On to my next point.
Usability testing sounds really scientific
Following on from the tail end of that last point, the term “usability testing” makes the activity sound more definitive, more scientific. Let’s be honest, even when we try, it usually isn’t. But it doesn’t have to be!
There is a place for large scale, highly structured, task-timing testing, but often what is most useful in terms of formative (but also summative) user input is something more simple. Many use the term “guerilla usability testing”, but that’s really just cowering in the face of academics and purists who scoff at our “lax methods” and “dismal sample size”.
If we don’t put that connotation of science on it, we won’t have to battle questions over statistical significance, or waste time defending something that gave useful results and improved the design process.
Closing the loop with user feedback
So let’s stop calling it usability testing. Let’s call it what it is: feedback, confirmation, validation. Showing people who will be using the thing we’re designing, and getting their feedback. It should be a natural part of the design process, closing the loop to ensure that what we’re designing it usable and useful for the intended audience.
Don’t show your designs to your boss, project manager or stakeholders for “approval” (ok not just to them). Show them to the only people who can truly sign off on them, your users.
I’m talking informal sessions. Collaborative or participatory design, if you will, but not testing.
UX practitioners often call it that to make it sound more than it is, give it more persuasive weight and the importance—or should that be respect?—that it deserves. I did it today in fact (well the day I started writing this post).
But no more. Let’s call it what it is and act like it’s part of the process. We can start to educate our colleagues and get them to the point they assume it’s part of the project too. “No, Miss Project Manager, we don’t need to wait until UAT to see if all the money we spent has paid off”.
What do you think? All feedback appreciated.
[Photo credit: Closing the loop by jspad]
Comments
Agree, agree, agree, agree.
The word “testing” has a quantitative, scientific echo. Words like evaluation, insight, feedback, etc really do describe both the process, and the output more accurately.
@Katie glad to see someone of your credentials agrees with me! :)
LOL and likewise!
My twitter bio reads “Hauling qualitative research’s butt out of its fluffy black box”: something I think you’ve done very eloquently in your post.
: )
Interestingly, I often do this kind of (qualitative) evaluative research, but haven’t ever used the term ‘usability testing’ (I’d never even heard about it until recently!). The discussions I have with my clients are typically about ‘getting insight/feedback on how people use/interact with your website’. No jargon. No ‘testing’. And very useful output that can either stand alone, or can feed into/frame quantitative research.
Well, @Katie, there are times when usability is indeed what the name suggests, and can be quantitative (eg time to perform tasks are measured for comparison to benchmark). So in that sense it’s not really research.
So it’s quite different to what you do, and what I’m talking about here. I prefer some research with, or feedback from, my audience during the design process to help shape and inform my decisions. As opposed to testing it afterwards. Both have their place, trouble is that the former is usually given the same name as the latter.
Patrick, agreed on all counts.
Reading your post took me back to those moments of horror when I used to ask people how they were involving users (a term I’ve never been that fond of either, but that’s for another conversation) in the design of whatever product of system they were thinking about unleashing.
The stock reply I used to get was “Oh yes, we’re doing user-centred design. We’re doing UAT just before we go live.”
At which point I would have to take a very, very deep breath.
Another insightful post Pat. One of the reasons I use the term “usability walkthrough” instead of “usability testing” for some of my work is that it sounds a bit more approachable, especially when we’re dealing with internal users (and having to describe the processes to them). While formal usability testing has its place, I love running more frequent “usability walkthroughs” with users through out the project life cycle to get the ongoing feedback. It also helps with some project managers who might be afraid of the “test” word!
I went through a stage of calling it user experience evaluations but people just got it mixed up with expert usability reviews. I actually like the term usability testing now. But I agree with your point about UAT.
I think that the term testing might help keep people’s perceptions of it away from a very opinion led type of study. After all it is still behaviour that is the most important think to observe.
When I was client side I had a big problem with internal folks thinking it was UAT. Now I have a problem with clients thinking of it as market research. They then transport all of their thinking from market reseach studies over to usability testing. And so the study gets bogged down in the minutae of recruitment and demographics.
LOVE THIS! I struggle with terms because I’m trained in formal QA but most of the work I’m hired to do doesn’t require test cases and test plans (as much as I think clients should be doing this during a build.) I’m always the after-thought. The person who reviews and as you say, offers validation and feedback. It’s not testing. There’s no metrics, although in more elaborate cases, there could be.
In any case, this post was refreshing :)
Patrick
Interesting post. I agree there is often misunderstanding about the definition of “usability testing.”
Well, as you may know, usability testing has been around for nearly 30 years now. It became commercially fashionable about 1990 when Jakob Neilsen made a big splash on the scene. I believe it will take another 10 years until companies realise the direct relationship between website ease-of-use, customer satisfaction and sales; and that usability testing is the only proven method of increasing these metrics.
Posts like yours raise awareness, generate discussion and “get the word out there.” Keep up the good work!
Regards
Scott
Thanks for feedback @Scott @Kim @David @Ruth @Nic !
@Ruth I like your use of “walkthrough”. A few times in the past I’ve used similar terms for the same reason, but I am liking walkthrough.
@David good point about coming off sounding like opinion rather than fact, this is a constant danger (swinging to the complete opposite end of the spectrum of sounding too “scientific” that I referred to above).
Either that or people think we’re talking about market research, as you point out. I think this is even more reason why we need to start using a new term; not testing, not opinion, not market research….
Something that incorporates “evaluation”, “observation” or “walkthrough” are my current favourites.
Great post Patrick. Very interesting.
I would call it user feedback. Can’t agree more with you on “closing the loop”. It’s all about the users, their needs and listening to them.
This is what drives us. We focus on user feedback for any web product (site, app or service). UserFix enables users to share their input by requesting new features and by reporting bugs – allowing product team members and users to collaboratively fix, improve and shape products on the web.
The difference is that you’re referring to it as part of the design process…
We actually call it social feedback, check it out – http://userfix.com :)
Usability testing is something very specific. It is a test (or evaluation) of the actual usability of a running system with real users using real scenarios. It uses both subjective and objective measures, and focuses heavily on the ability of users to successfully complete relevant tasks. It has been utilized by human factors engineers well before it became ‘popular’ in the ’90′s with the work of Nielsen. Have you ever conducted a system level usability test? Call it what it is. If it really is a usability test. What about a task analysis, or a requirements identification session, or a concept engineering session, or a focus group, or a design walkthrough (high and low levels) or a prototype test,or…..? No one outside of UCD or Hf or Ux or..? Really cares. You are doing your job when your exec wants you to present ‘your stuff’ every week at the portfolio management or executive team meeting.
@Greg: Fair point, all this talk about what we call something is moot if we’re not producing useful results. Which could be said of any field of practice.
But of course some people do pay attention to the terminology we use, even if they have only the most superficial understanding of what it means. For example, I often find that it pays to be careful when talking to producers and product managers about the methodology I’ll be using, because one little word can completely skew their perception early on in the process. Of course, once we get underway and they see the goodness being delivered, it matters much less, but early on when trying to buy-in to a proposed methodology, using the word “walkthrough” instead of “testing” might make all the difference.
Not everyone will look beyond the labels and examine the inherent value (or lack) of the method being proposed.
Ok. I’m totally confused here (not to mention quite probably way out of my depth).
@david Hamill; beyond observing behaviours to answer specific (closed ended) questions, how do you interpret/make sense of the observed behaviours without asking the users? Wouldn’t the output from observation alone be relatively biased by the tester’s particular framework; what they notice, how they interpret it, etc?
@Patrick; not sure I agree that this isn’t ‘market research’. Isn’t “Something that incorporates “evaluation”, “observation” or “walkthrough”” research? And if the output is ultimately going to be used for commercial purposes, then isn’t it “market research”?
Definitely a particular sort of market research, but nevertheless, surely…
: S
Hi Katie That’s a great question. Like others, I use the think aloud technique. I ask participants to talk through their actions while they do it. This gives remarkable insights into user behaviour. Asking them why they did things afterward is totally unreliable.
That is why we concentrate on what people do over what they say.
You can read a good article on that subject here:
http://www.userfocus.co.uk/articles/askingwhy.html
I am still not sure I understand your point. Why would we stop calling usability testing what it is: usability testing? It is a very specific type of user activity involving an actual running product focusing on objective task measures as well as subjective measures, using representative users, representative tasks, performed at the end of the development effort, prior to product release. So what is the problem with call ing it usability testing?
The problem I have is when people (especially HF people, this really burns me up) call ALL user activities ‘usability testing’. Usability testing is not performed early in the design process, like a requirements brainstorming session, or a high-level paper design walkthrough. These are diferent user activities, deserving of different names.
I am sorry, but I can’t let it slide when the project manager refers to a recent set of telephone interviews with IT admins on pain points with managing IT systems as a usability test. If I do, I think it’s a disservice to everyone involved. I know it can lead to painful discussions, but these points are not esoteric, mattering only to the HF person in the group. The project manager would not call a high-level code review a unit test, and if they did, they would be educated pretty quickly, and probably publicly abused for being so ignorant of the process.
@Katie I wouldn’t label user experience research (or design research) as market research, as there are some important differences. During design research we’re focusing on target demographics at large, whereas design research focuses on observing people and their behaviour with products. Through these insights, design research aims at providing user-friendly products that will help create a better user experience.
The people we talk to in design research might be part of ‘the market’ but to thus call it market research is like saying you’re a psychologist just because you have a chat to someone who might happen to see a psychologist (ok that’s the best analogy I could come up with!).
And this was my whole point, the name we give a technique markedly affects its perception and acceptance by other disciplines.
@Greg what I’m saying is that quite often the activity that people are doing and calling “usability testing” isn’t what you’ve just described. And that’s the problem.
If you do…
…then fair enough, continue to call it usability testing.
I’m talking about lower fidelity, more casual forms of user engagement. Testing is not the right word for that. And it seems you agree with me:
And I also agree with you last point. Possibly why you’re not getting me is that we are talking about quite different activities that are unfortunately both called usability testing.
Out of interest, what’s your background?
Patrick Kennedy: You write: “Possibly why you’re not getting me is that we are talking about quite different activities that are unfortunately both called usability testing.”
Which two different activities are unfortunately both being called usability testing and by whom? And if they are, whomever they are, they are wrong. There is only one type of activity that should be called usability testing.
I think we may both be in violent agreement.
I guess my background is pretty classical, old school human factors in software design. I started work before I heard of interaction designers, information architects, user experience designers, Ux, IDx, etc. etc. I have 25 years in applied human factors in industry software development and design with classical academic training (Ph.D.s) in human factors psychology and systems engineering. I am an old dog that cut its teeth in a dark usability test lab conducting system level usability tests. Bow wow. ;-) Thanks for the dialog. I guess we can consider the loop closed. But by all means, please continue if you prefer.
@Greg you are right, there is only one thing that should be called usability testing, and I think I even agree with your definition of what that thing is. Thus I suppose I (and many of the people you seem to dislike) have used to term in the past to describe other activities during the design process, such as summative sense-checking with users.
Hence I’m suggesting we stop calling anything usability testing unless it is indeed usability testing.
So here’s to violent agreement :)
Here! Here! :)
[...] an interesting debate brewing over the definition and use of the term “Usability Testing” over at Patrick Kennedy’s [...]
Hmmm, UAT the stomping ground of demonistic CIO’s and even worse ‘business people’. UAT is a far from a science and is a necessary evil, if for no other reason than to get some degree of ‘business’ buy in and appreciation for what the widget does, especially given most people narrow sighted approach to UAT and testing only the ‘good’ scenarios and forgetting end users and clients have an uncanny ability to find limitless examples of never before considered ‘bad’ or ‘not good’ scenarios. So anyway how’s your day?
[...] an interesting debate brewing over the definition and use of the term “Usability Testing” over at Patrick Kennedy’s [...]
Post a comment