Pat’s Point of View

Almost three years ago I wrote stop calling it usability testing, essentially making the argument that the term “usability testing” has a lot of baggage and gets mistaken for other things.

I still don’t like using the term in most cases, and I’ll explain why. But in the intervening years I have come up with an alternative, which I’d like to share with you. Within the UX team here at NDM, I’ve been referring to user sessions as CIDeR (Collaborative Iterative Design Refinement) sessions. I’ve had some success in convincing my team-mates and the term is starting to permeate out into the business.

Why CIDeR?

Typically our users are involved in our design process by way of a series of one-on-one sessions where users are shown stimuli of some kind, to elicit feedback. The purpose is to guide the design process and allow decisions to be made (usually) regarding the user interface. Successive rounds are used to allow the design to evolve based on user feedback, in effect making users collaborators in the design process.

Hence the name:

• Collaborative – The user is an integral part of the process, as are our colleagues from other disciplines. This word also helps break down the ‘UX guy is expert’ and ‘participant is lab rat’ dynamic that can accumulate.
• Iterative – The approach works best if it’s a process of constantly evolving the design or the idea. This word helps convey to the business that this isn’t a one shot deal, there will be several rounds of user involvement, with some thinking and designing in between.
• Design – Typically these sessions are for the purpose of producing something tangible, whether it’s designing a website or a concept. This word grounds the name/description.
• Refinement – We are working towards producing something. In conjunction with ‘iterative’ this word impresses upon people the fact this is a process, and in conjunction with ‘design’ it gives a sense of progress.

Oh, and of course there’s the added benefit of being able to say “let’s have some CIDeR and think it through” when the team reaches an impasse or isn’t sure how to proceed.

We involve users in our process in many other ways, from up-front ethnographic research through to large quantitative market research, and lots of things in between, but the bread and butter would be the CIDeR sessions. Hence it’s important for us to be clear what this work is and what it delivers—to our team but also to our business.

Why not “usability testing”?

There are four problems with the term usability testing as a label for the type of work done in a CIDeR session, some of which are refinements of the point I made last time:

• Promises conclusive, definitive results – The term sounds too absolute. As you’d expect from “testing”, after all other types of testing deliver conclusiveness or they’re considered a failure.
• Implies a focus on just the UI and usability – Much of what we do is more than usability of the user interface. We’re digging deeper, talking through preferences, perceptions. Part of this is due to the fact that for news products, the content is as much a part of the interface as the buttons, links, labels and code.
• Suggests summative application – To many people, when you say “usability testing” they think that’s something to be done at the end, a validation exercise to make sure we can go live. This isn’t at all the case for most of the work our team does, which is more about exploration over time; a fluid process rather than check-list item.
• Coloured by past experience – Any term that has been around for a while, and widely misunderstood or misused, will be horribly tainted by the experience stakeholders have had with things labelled with that term. This is certainly the case with “usability testing”. I often see this as a tendency towards quant; people expect task failure rates, ‘time on task’ and other rigid measurements and won’t give up on those kinds of outputs from our work. Again, these are rarely the things we are looking to obtain.

Don’t get me wrong, if you practice a method that does live up to all of these things, and you call it usability testing, good on you. Our team rarely does, so I don’t want to set an expectation in the minds of my stakeholders that that is what they’re going to get. We needed a new name.

How does CIDeR work with other techniques?

The CIDeR approach is qualitative and indicative, rather than conclusive. Which means that some findings (ie opinions, perceptions, propensity to buy/use) may not be representative of the larger population, and as such it is necessary to:

1. exercise care in taking these findings on board, using them in the right way, and
2. make use of quantitative methods, either before or after CIDeR, to determine the implications for the broader audience.

Sometimes a more formal method for involving users in the design process is used, which we do call “usability testing”. A more rigorous approach is taken to assessing how easily users are able use a given design, typically later in the design process. Because this technique is dealing strictly with usability, it is acknowledged that relatively small sample sizes (~5) can be used to draw conclusions about the usability of the design for the entire audience.

Questioning regarding opinions or propensity to buy/use, however, do require larger sample sizes. So, alongside both the CIDeR and “usability testing” methods, quantitative research may also be employed, typically to gauge reactions to a product proposition or design. This focuses more on supporting decision making at a product level as opposed to the design or user interface level.

“Pour me a glass!” or “Ewww that’s left a bitter taste”?

What do you think of the name CIDeR? Would you use it in place of the term usability testing? Why or why not? All feedback greatly appreciated.

(Originally posted to the USiT blog, reproduced here with some minor alterations)

My colleague Angus Fraser recently gave The Claw a good workout on some mobile site testing in Brisbane. The screenshot below shows the output of the two webcams side-by-side, viewed in a neat piece of software called AMCap. The two AMCap windows, and the audio from one of the webcams, was recorded and broadcast to observers using GotoMeeting.

There’s nothing like some real world feedback to improve a product, and Angus was most helpful in this regard. Besides the introduction of AMCap, Angus pointed out that the webcam mounting solution wasn’t the best. The original screws from the webcam base were too short to make it all the way through the perspex and securely hold the webcam, even with the countersinking. So I’ve enhanced the design by using meatier screws (wood screws actually) that are longer than the original and with wider thread to really bite into the plastic of the webcam. Note this is a destructive enhancement, the new screws will wreck the hole for the original screws and you’ll no longer be able to attach the webcam’s circular desk stand.

The new screws also have bigger heads that a normal screwdriver will drive (not requiring a jeweller’s screwdriver like the original screws). So not only are the webcams held nice and securely, but it’s easier to undo and move them.

The other enhancement Angus suggested was to use ‘velcro’ on the handset, so it can be taken off the Claw for setup changes, but then securely re-attached. The ‘velcro’ strips are Command Picture Hanging Strips (annoyingly, they don’t refer to them as ‘velcro’).

I also tied the two USB cables for the webcams together with cable ties, making them less messy when using the Claw.

The Claw is evolving!

(Originally posted to the USiT blog, reproduced here with some minor alterations)

The ominous black shape featured in last week’s “Guess that object” post is in fact my take on a mobile device usability testing jig, inspired by the work of Kirk Henry of Lokion Interactive (via Harry Brignull). I’ve been working on this device to help with testing site and app designs on mobile phones and tablets. Quite often these contraptions are called a sled but I’ve been calling this one “The Claw”, for hopefully obvious reasons.

Its purpose is to allow you to get a good view of the screen of a mobile device—handset or tablet—as well as the user’s face, during usability testing (or any other activity that you’d like to see what’s happening while someone uses a mobile device. Using software such as TechSmith Morae 3.0, you can easily record from both cameras.
(more…)

What is this a photograph of?

The answer will be revealed shortly, along with a full explanation.

(Note: you NDM folks are disqualified from entering!)

I originally started writing this post when I was at UPA 2007, but for one reason or another I never published it. On several occasions, I played with the idea of combining the conference notes with some later half-written posts on generally the same topic. But alas it never made it live.

Seeing as I firmly believe that for every unpublished blog post there is one less bit of momentum keeping the interwebs spinning, I’d better put this up. And it’s interesting to look back at what I wrote two and a half years ago…

Day 1 started with a very inspiring talk by Bill Buxton. I think this was just the thing the industry needs, a bit of a reality check and a wake-up call. Firstly usability evaluation is not design and for that reason most people here don’t actually practice User-Centred Design. It’s all about data, rules, strict methodologies, large companies. They’ve even turned agile into something overly defined and bogged down (I have no strong belief either way when it comes to agile methodologies by the way). Bill’s talk about sketching as an important tool for the design process flies in the face of the artefact centric practice many Usability Professionals follow. No there’s no template for it, no there’s no software tool to do it, you have to use your brain! I mean the theme of the conference (“patterns”) says it all really.

This sounds really negative, but I don’t want to be. There are some smart and talented people here, but overall the industry is weighed down by strictness and illusions. Strictness in the sense that many people want some methodology to tell them what to do. I can understand that, but as Bill said, if you find yourself thinking that all the time (being scared of wining it) then maybe this isn’t the job for you. Illusions in terms of the discrepancy between literature and practice. A lot of the things published are not followed in practice (eg rapid, flexible approaches by clever people are replaced by limited, templated projects) and good practice is not published (eg using multiple design alternatives in usability testing). Then there’s the illusions of grandeur, like the way many practitioners think of what they do as some kind of scientific crusade and admitting there is some I-don’t-know-ness to it is an act of heresy.

For me, the best thing I saw at the conference was this talk. It’s a pity someone from outside the field (perhaps technically but really as far as I am concerned he’s slap bang in the centre of what we should strive for) had to be the one to say it. You can’t truly be doing UCD if you’re just evaluating, testing and documenting. This shouldn’t be about statistical analysis techniques.

I remember thinking that my approach to my work seemed at odds with how other attendees appeared to be working, and from the above it seems this annoyed me! Too many practitioners being more worried about following the ‘proper’ process, rather than actually thinking. And the post I did publish at the time, contains similar thoughts.

I spent today at Ark Group’s Strategically Managing Intranet Developments conference, which I blogged about before.

There were some good things being said, and by real people who have done the hard yards. They’re not “industry luminaries”, but people out there in the trenches working out how to create effective intranets. Grounded and real are two words I would apply to the conference.

Then there was my presentation, a tad more abstract, but I felt it went well. Slides below.

I felt compelled to steer my presentation towards audience participation, if only because of the collective knowledge in the room; about half the room were presenting at the conference so I was learning as much, if not more, than I was dishing out. That’s the downside of being a consultant, you rarely get that rich experience that in-house staff have. Some great examples were offered by the audience, complementing my own examples.

There were one or two people twittering, you can follow the conversation on #smid.

Happy to hear your comments on my slides, either here or on slidehsare.

More than likely your content management system (CMS) will have many usability problems if you just use it “out of the box”. Having been involved in a number of projects tasked with implementing a these types of systems—including content management systems for websites, intranets and wikis for knowledge management—I’ve noticed that there are a number of key areas of the user interface that frequently need fixing from a usability point of view.

All the usability tips you see here link back to general usability principles, and they apply to any software package or web application, it just seems that they are an issue in most CMS implementations.

Use these tips to improve your current CMS or to help you when implementing a new one.

1. If in doubt, leave it out

The user interface should be devoid of everything that is not necessary in terms of users completing their tasks. Most CMS products will have capabilities in excess of what is being used, but don’t show it if they don’t use it. And many products will have optional extras and upgrade possibilities, so your version might not have all the bells and whistles. For better or worse, some vendors will leave a stub to these missing features (possibly to help encourage up-sell). Don’t show it if they can’t use it.

Use CSS to hide stuff if you have to, but clean up that interface. We’re talking about main navigation, links, and irrelevant details spat out by the system. This also applies to words; as Steve Krug said “Krug’s third law of usability: get rid of half the words on each page, then get rid of half what’s left”. Each page title, sub heading, button label, navigation label, form field label, icon and graphic should be useful and meaningful, clearly communicating what it should.

(more…)

In my recent IA tutorial for OZCHI08, I told my students about performing low-fi usability testing on a draft information architecture. I introduced them to a technique which Donna Spencer called Card Based Classification Evaluation (CBCE) and is known to other people as tree testing or task-based information architecture testing (which really doesn’t have the same ring to it).

(Incidentally, CBCE is an evolution of closed card sorting but crucially different. I quite like Donna’s explanation for the need for such a technique from about the time she coined the term: Categorising information and finding it are two entirely different tasks, with entirely different cognitive processes. The only way to test whether a classification will allow people to find information, is to ask them to find information…You don’t learn it by asking them to place information in the classification. Hence just using closed card sorting won’t do.)

As usual, some people quickly tired of my affection for fiddly index cards and asked if you can do this electronically or even online. It reminds me of that line in Star Wars where the dude says to Vader “your sad devotion to that ancient religion has not helped you conjure up the stolen data tapes”…and you know what happened to that guy. Rather than crush their throats with my mastery of the Force, I explain that there are some advantages to good old cards:

• Cards are quick and easy to create and modify
• Cards remove the distractions associated with ‘using a computer’
• Cards allow a greater affinity between participant and facilitator
• Cards function consistently for all participants (whereas software may not)
• Cards can be made by anyone (eg don’t require special skills or software)
• Cards are cheap

However, I must admit there are situations when something more computery might be more useful, such as:

• You need to include remote participants (eg widely dispersed target audience)
• You intend to involve a large number of users in the testing (eg more than 20)
• You want to spend less time recording results into an electronic format (ok I’m reaching now)

You can use one of the many card sorting tools—either desktop software or online—but there are also a few dedicated tools. For example, Treejack which Sam Ng from Optimal Workshop showed at OZ-IA this year. It’s quite a nice tool and would do nicely for getting that all important user feedback on your design decisions early (and possibly later) in the IA design process.

Of course, one can use something like PowerPoint or Dreamweaver if one likes, it all depends on the resources at your disposal and the capabilities of your test participants—you might be a FileMaker whizz but there’s no point sending out a test app built in it if your users can’t open it.

Feel free to leave a comment below if you have any other suggestions.

Now that I’ve got your attention, let me clarify what I mean. When we refer to this activity called “usability testing” there are often a lot of misunderstandings. It’s really not very applicable for the thing we should be doing. Here are some reasons why…

• It gets mistaken for UAT.
• It gets mistaken for technical testing.
• It makes it sound more ‘scientific’ than it (usually) is.

Usability testing is not UAT

UAT, or User Acceptance Testing, is a term used in software engineering to describe when the client would give final approval for the built system to be delivered. This becomes really confusing when the term is taken out of that context and used in web design, for example. When you don’t understand the “user” it refers to and what is being “accepted”, you might very well think UAT is the same thing as “usability testing”.

Conceptually, the intention of UAT might be to make sure those who will ultimately be using the system are happy with it. But in practice, this is rarely the case. I could go on, but this is not a discussion of why I don’t like UAT (which I don’t) but rather my point is that we don’t want stakeholders thinking that usability testing is UAT, and thus something that can be dispensed with because “we always go through UAT”.

Usability testing is not done by Mr Test Manager

The term “usability testing” often gets misconstrued by technical types, project managers and business analysts. It gets turned into a stale, rigid, bureaucratic affair. The old “unit, integration, system” mantra. It’s done as a matter of course, at the end of the gantt chart, to tick a box. That’s pointless.

Again, in theory, test driven design is not a bad thing. Software, websites and anything technically complex should be checked to make sure it has been built as was required. That assumes a lot though, for example are the requirements valid? do users actually want or need what is being built? But let’s leave that one alone.

What I’m trying to say, is that usability testing shouldn’t be mistaken for the technical testing done by a Test Manager according to a test plan, using a test script. (At least not the kind of usability testing I want to be doing, which is possibly a qualification I should have stated up-front). People do run highly structured usability tests—typically summative in nature—which are very similar to technical testing. In my experience this is the minority of cases and the least valuable. On to my next point.

Usability testing sounds really scientific

Following on from the tail end of that last point, the term “usability testing” makes the activity sound more definitive, more scientific. Let’s be honest, even when we try, it usually isn’t. But it doesn’t have to be!

There is a place for large scale, highly structured, task-timing testing, but often what is most useful in terms of formative (but also summative) user input is something more simple. Many use the term “guerilla usability testing”, but that’s really just cowering in the face of academics and purists who scoff at our “lax methods” and “dismal sample size”.

If we don’t put that connotation of science on it, we won’t have to battle questions over statistical significance, or waste time defending something that gave useful results and improved the design process.

Closing the loop with user feedback

So let’s stop calling it usability testing. Let’s call it what it is: feedback, confirmation, validation. Showing people who will be using the thing we’re designing, and getting their feedback. It should be a natural part of the design process, closing the loop to ensure that what we’re designing it usable and useful for the intended audience.

Don’t show your designs to your boss, project manager or stakeholders for “approval” (ok not just to them). Show them to the only people who can truly sign off on them, your users.

I’m talking informal sessions. Collaborative or participatory design, if you will, but not testing.

UX practitioners often call it that to make it sound more than it is, give it more persuasive weight and the importance—or should that be respect?—that it deserves. I did it today in fact (well the day I started writing this post).

But no more. Let’s call it what it is and act like it’s part of the process. We can start to educate our colleagues and get them to the point they assume it’s part of the project too. “No, Miss Project Manager, we don’t need to wait until UAT to see if all the money we spent has paid off”.

What do you think? All feedback appreciated.

[Photo credit: Closing the loop by jspad]

I was reading the Qantas in-flight magazine yesterday, and came across a review for the Olympus Tough Smart 1050SW which was claimed to be “astonishingly resilient to holiday rough and tumble”. And as a keen snowsports fan, it would seem to be a perfect camera for skiing. Not only is it sealed, waterproof and able to withstand freezing temperatures (which I can attest most electronic devices are not able to do) but the “entire camera body is touch-sensitive, making it easier to use [whilst wearing] gloves”.

All you need to do is touch the top of the camera to take a photo or you can select “play” mode by tapping the LCD screen. To browse to the next photo you just tilt the camera to the left or right. If only all products were this straightforward to use, and contextually respectful. I’ve not used one, so I don’t know how sophisticated it is at detecting these gestures, but it sounds perfect for snowsports. Apart from not wanting to take off your gloves because it’s cold, doing so takes precious time that might mean you miss that spectacular stack that you really want to capture forever.

This is excellent design for the context of use. There may be other cameras like this, but this is a great example of seeing the problems people have using technology, and then solving those problems in a suitable way. Well done.