Pat’s Point of View

In my recent IA tutorial for OZCHI08, I told my students about performing low-fi usability testing on a draft information architecture. I introduced them to a technique which Donna Spencer called Card Based Classification Evaluation (CBCE) and is known to other people as tree testing or task-based information architecture testing (which really doesn’t have the same ring to it).

(Incidentally, CBCE is an evolution of closed card sorting but crucially different. I quite like Donna’s explanation for the need for such a technique from about the time she coined the term: Categorising information and finding it are two entirely different tasks, with entirely different cognitive processes. The only way to test whether a classification will allow people to find information, is to ask them to find information…You don’t learn it by asking them to place information in the classification. Hence just using closed card sorting won’t do.)

As usual, some people quickly tired of my affection for fiddly index cards and asked if you can do this electronically or even online. It reminds me of that line in Star Wars where the dude says to Vader “your sad devotion to that ancient religion has not helped you conjure up the stolen data tapes”…and you know what happened to that guy. Rather than crush their throats with my mastery of the Force, I explain that there are some advantages to good old cards:

• Cards are quick and easy to create and modify
• Cards remove the distractions associated with ‘using a computer’
• Cards allow a greater affinity between participant and facilitator
• Cards function consistently for all participants (whereas software may not)
• Cards can be made by anyone (eg don’t require special skills or software)
• Cards are cheap

However, I must admit there are situations when something more computery might be more useful, such as:

• You need to include remote participants (eg widely dispersed target audience)
• You intend to involve a large number of users in the testing (eg more than 20)
• You want to spend less time recording results into an electronic format (ok I’m reaching now)

You can use one of the many card sorting tools—either desktop software or online—but there are also a few dedicated tools. For example, Treejack which Sam Ng from Optimal Workshop showed at OZ-IA this year. It’s quite a nice tool and would do nicely for getting that all important user feedback on your design decisions early (and possibly later) in the IA design process.

Of course, one can use something like PowerPoint or Dreamweaver if one likes, it all depends on the resources at your disposal and the capabilities of your test participants—you might be a FileMaker whizz but there’s no point sending out a test app built in it if your users can’t open it.

Feel free to leave a comment below if you have any other suggestions.

I’ve just returned from a week in Cairns, a trip which was meant to be partly for the OZCHI 2008 conference and partly for holiday. Not the best trip.

Last Sunday we flew up to Cairns. When we arrived the Qantas check-in staff didn’t provide an infant boarding pass for my daughter, and I didn’t such a thing even existed, so when we went to board there was a bit of dilemma. Thankfully the staff at the gate were very helpful and gave us an upgrade to business class! (Although I think this result had more to do with another family of four who were giving them a hard time about wanting to sit together so our two ‘spare’ economy seats were a welcome relief). It was a great start to the holiday, and not a bad flight to get upgraded for: it’s 3 hours to Cairns.

Then we arrived in Cairns. Hot. Humid. Ouch. I’m not built for hot weather, but I soldiered on and tried to find the cab we had booked online the night before, with a special request for baby seat. A lot of cabs in Cairns are now based on the eco-friendly Toyota Prius, so I was thinking hey they are pretty high tech up here. Sadly no. After asking some people and having no luck I called the cab company. They told me two things: a) they don’t accept bookings from the airport and b) “in Queensland public transport vehicles are exempt from having to carry baby capsules” (a phrase we were to hear over and over again) so they can’t do that either. FAIL! Why does your website allow you to request both those things if you can’t deliver?

So now I know why you see so many baby seats/capsules on the luggage conveyors at the airport, it’s because you can’t rely on anyone you just have to bring your own. Seriously, I don’t give a crap if the Queensland government have deemed “it’s perfectly legal to just hold the baby on your lap”, it’s not safe! How can they feel it’s unnecessary for essential safety equipment for babies to be available? I would happily have waited and/or paid more to secure a safe mode of transport for Grace. We had few choices, the best of which seemed to be catching a cab with Grace on Jenn’s lap in the back seat. We discussed the lack of baby seats in Queensland cabs with the driver, as well as how using a hybrid vehicle for a taxi is really just a PR stunt and not yet economically viable (nor environmentally friendly at the end of the day).

Monday. A day of relaxing and acclimatisation in preparation for my OZCHI tutorial on Tuesday. Possibly even more hot and humid. Yuck. It’s not glamorous, but we spent most of the day in Smithfield shopping centre because we had to get some groceries for Grace and it was air conditioned! (not something there is a lot of in Palm Cove). In the spirit of the tropical theme I bought a lovely knock-off ‘Aloha shirt’. Cool in that absurd kind of way.

Tuesday. Went off to James Cook University for my tutorial. Still hot. Got into the room the tutorial would be in…urrgh not much cooler. Lovely. The tutorial went well, although almost everyone there was not exactly at the right level for an introductory course, but I got good feedback and I think we all got something out of the half day.

Wednesday. Can’t remember what we did in the morning, but I remember the heat hadn’t relented. We went for a late lunch at Pepper’s Beach Club on the esplanade at Palm Cove, we’d been there many times when we last stayed there and it was always nice. I had a ham focaccia and a light beer. Big mistake. I remember thinking our order arrived quite quickly, Jenn only got a salad so I was wondering if my sandwich was fresh. It turns out it probably wasn’t and given the hot and humid conditions something was off. To cut a long and painful story short I spent the next 8 hours witnessing my body performing a complete evacuation of its digestive system by any and all means possible. I’ve not been struck by any major illness in my life, and I’m sure this is nothing in comparison, but this bout of food poisoning was the worst such thing I’ve ever experienced.

By midnight I was completely wrecked, severely dehydrated and in need of medical assistance. I was worried, about me but also about my family, what if this was viral? At this point I was also pondering, in my barely lucid state, how vulnerable we are when on holiday. Without your normal support network, local knowledge and even transport, you kinda feel on your own. But we managed to get an ambulance to the resort which was harder than expected. The paramedics agreed it was food poisoning, possibly with a bit of heat stroke to boot. I opted not to take them up on the offer of hospitalisation because I had a wife and baby to consider. Besides, the mass evacuation seemed to have slowed by the time the paramedics arrived, so I thought I would ride it out and see how I was in the morning. It was a rough night.

After the ‘purge’ had finished the ‘reboot’ began, as my digestive system was rebuilt, a process that ultimately went on for about 60 hours. I was trying to rehydrate myself but Mr Stomach wasn’t playing nicely. I couldn’t digest anything for a while, so eating and drinking were very difficult.

Thursday. Why does it always seem to be overcast in Palm Cove? It makes is more humid and it’s depressing rather than a tropical paradise. But on this day I think the sun was out all day. It might have been hotter but at least not as humid, but I wouldn’t know since I spent the entire day in bed. Made a break for it at dinner time and chanced a trip to the restaurant. Saw OZCHI folks going off to the conference dinner, but I was in a daze. Didn’t last long, had to return to the room.

Friday. Started to get better, I even ventured out for breakfast and lunch. Arranged to meet James Breeze and some others for dinner, but as the sun started to set the humidity rose and as we couldn’t find any restaurants with air-conditioning I quickly wilted and had to make a very early exit. The heat, sweat and thought of food just made me ill. Why no restaurants in Palm Cove have A/C is beyond me.

Saturday. Feeling much better. Packing, breakfast then transfer to airport (and guess what? no baby seats available, because “in Queensland…” grrrr!). At the airport things were looking up, we managed to all get on the same flight (long story) and even got moved to a bulkhead row with a bassinet so we could put the baby down for a sleep during the flight. Back home in Sydney. Dry heat. Hmmmm.

Man, I need a holiday.

Now that I’ve got your attention, let me clarify what I mean. When we refer to this activity called “usability testing” there are often a lot of misunderstandings. It’s really not very applicable for the thing we should be doing. Here are some reasons why…

• It gets mistaken for UAT.
• It gets mistaken for technical testing.
• It makes it sound more ‘scientific’ than it (usually) is.

Usability testing is not UAT

UAT, or User Acceptance Testing, is a term used in software engineering to describe when the client would give final approval for the built system to be delivered. This becomes really confusing when the term is taken out of that context and used in web design, for example. When you don’t understand the “user” it refers to and what is being “accepted”, you might very well think UAT is the same thing as “usability testing”.

Conceptually, the intention of UAT might be to make sure those who will ultimately be using the system are happy with it. But in practice, this is rarely the case. I could go on, but this is not a discussion of why I don’t like UAT (which I don’t) but rather my point is that we don’t want stakeholders thinking that usability testing is UAT, and thus something that can be dispensed with because “we always go through UAT”.

Usability testing is not done by Mr Test Manager

The term “usability testing” often gets misconstrued by technical types, project managers and business analysts. It gets turned into a stale, rigid, bureaucratic affair. The old “unit, integration, system” mantra. It’s done as a matter of course, at the end of the gantt chart, to tick a box. That’s pointless.

Again, in theory, test driven design is not a bad thing. Software, websites and anything technically complex should be checked to make sure it has been built as was required. That assumes a lot though, for example are the requirements valid? do users actually want or need what is being built? But let’s leave that one alone.

What I’m trying to say, is that usability testing shouldn’t be mistaken for the technical testing done by a Test Manager according to a test plan, using a test script. (At least not the kind of usability testing I want to be doing, which is possibly a qualification I should have stated up-front). People do run highly structured usability tests—typically summative in nature—which are very similar to technical testing. In my experience this is the minority of cases and the least valuable. On to my next point.

Usability testing sounds really scientific

Following on from the tail end of that last point, the term “usability testing” makes the activity sound more definitive, more scientific. Let’s be honest, even when we try, it usually isn’t. But it doesn’t have to be!

There is a place for large scale, highly structured, task-timing testing, but often what is most useful in terms of formative (but also summative) user input is something more simple. Many use the term “guerilla usability testing”, but that’s really just cowering in the face of academics and purists who scoff at our “lax methods” and “dismal sample size”.

If we don’t put that connotation of science on it, we won’t have to battle questions over statistical significance, or waste time defending something that gave useful results and improved the design process.

Closing the loop with user feedback

So let’s stop calling it usability testing. Let’s call it what it is: feedback, confirmation, validation. Showing people who will be using the thing we’re designing, and getting their feedback. It should be a natural part of the design process, closing the loop to ensure that what we’re designing it usable and useful for the intended audience.

Don’t show your designs to your boss, project manager or stakeholders for “approval” (ok not just to them). Show them to the only people who can truly sign off on them, your users.

I’m talking informal sessions. Collaborative or participatory design, if you will, but not testing.

UX practitioners often call it that to make it sound more than it is, give it more persuasive weight and the importance—or should that be respect?—that it deserves. I did it today in fact (well the day I started writing this post).

But no more. Let’s call it what it is and act like it’s part of the process. We can start to educate our colleagues and get them to the point they assume it’s part of the project too. “No, Miss Project Manager, we don’t need to wait until UAT to see if all the money we spent has paid off”.

What do you think? All feedback appreciated.

[Photo credit: Closing the loop by jspad]