Pat’s Point of View

Now that I’ve got your attention, let me clarify what I mean. When we refer to this activity called “usability testing” there are often a lot of misunderstandings. It’s really not very applicable for the thing we should be doing. Here are some reasons why…

• It gets mistaken for UAT.
• It gets mistaken for technical testing.
• It makes it sound more ‘scientific’ than it (usually) is.

Usability testing is not UAT

UAT, or User Acceptance Testing, is a term used in software engineering to describe when the client would give final approval for the built system to be delivered. This becomes really confusing when the term is taken out of that context and used in web design, for example. When you don’t understand the “user” it refers to and what is being “accepted”, you might very well think UAT is the same thing as “usability testing”.

Conceptually, the intention of UAT might be to make sure those who will ultimately be using the system are happy with it. But in practice, this is rarely the case. I could go on, but this is not a discussion of why I don’t like UAT (which I don’t) but rather my point is that we don’t want stakeholders thinking that usability testing is UAT, and thus something that can be dispensed with because “we always go through UAT”.

Usability testing is not done by Mr Test Manager

The term “usability testing” often gets misconstrued by technical types, project managers and business analysts. It gets turned into a stale, rigid, bureaucratic affair. The old “unit, integration, system” mantra. It’s done as a matter of course, at the end of the gantt chart, to tick a box. That’s pointless.

Again, in theory, test driven design is not a bad thing. Software, websites and anything technically complex should be checked to make sure it has been built as was required. That assumes a lot though, for example are the requirements valid? do users actually want or need what is being built? But let’s leave that one alone.

What I’m trying to say, is that usability testing shouldn’t be mistaken for the technical testing done by a Test Manager according to a test plan, using a test script. (At least not the kind of usability testing I want to be doing, which is possibly a qualification I should have stated up-front). People do run highly structured usability tests—typically summative in nature—which are very similar to technical testing. In my experience this is the minority of cases and the least valuable. On to my next point.

Usability testing sounds really scientific

Following on from the tail end of that last point, the term “usability testing” makes the activity sound more definitive, more scientific. Let’s be honest, even when we try, it usually isn’t. But it doesn’t have to be!

There is a place for large scale, highly structured, task-timing testing, but often what is most useful in terms of formative (but also summative) user input is something more simple. Many use the term “guerilla usability testing”, but that’s really just cowering in the face of academics and purists who scoff at our “lax methods” and “dismal sample size”.

If we don’t put that connotation of science on it, we won’t have to battle questions over statistical significance, or waste time defending something that gave useful results and improved the design process.

Closing the loop with user feedback

So let’s stop calling it usability testing. Let’s call it what it is: feedback, confirmation, validation. Showing people who will be using the thing we’re designing, and getting their feedback. It should be a natural part of the design process, closing the loop to ensure that what we’re designing it usable and useful for the intended audience.

Don’t show your designs to your boss, project manager or stakeholders for “approval” (ok not just to them). Show them to the only people who can truly sign off on them, your users.

I’m talking informal sessions. Collaborative or participatory design, if you will, but not testing.

UX practitioners often call it that to make it sound more than it is, give it more persuasive weight and the importance—or should that be respect?—that it deserves. I did it today in fact (well the day I started writing this post).

But no more. Let’s call it what it is and act like it’s part of the process. We can start to educate our colleagues and get them to the point they assume it’s part of the project too. “No, Miss Project Manager, we don’t need to wait until UAT to see if all the money we spent has paid off”.

What do you think? All feedback appreciated.

[Photo credit: Closing the loop by jspad]

In November I’ll be appearing at the Ark Group conference Enhancing Online User Experience in Melbourne. My contribution will be an interactive discussion on “Documenting for interactive websites”.

Since it’s an interactive discussion, the most crucial part of the session will be input from the audience and the discussion that emerges. To that end I’ve launched a special forum to allow conference attendees (or anyone for that matter) to suggest the topics they would like discussed:

(If you can’t see the widget above, you can go to patrickkennedy.uservoice.com)

You can suggest a question, or vote for an existing question. I’ll try to cover as many as I can in the time available during the session.

I see two broad aspects to ‘documenting for interactive wesbites’, the first being design documentation and the second being project documentation. By design documentation I mean the sketches, wireframes, storyboards, flow charts or whatever other form of documentation that is used to document what is it that being developed. And by project documentation I mean those documents such as SOWs and project plans that assist with project management and logistics.

I’ll attempt to cover both aspects but ultimately the session will be run by audience participation, so get voting on the topics you want discussed!

The conference details are:

Enhancing Online User Experience

Developing and executing user experience strategies that will provide business benefits

• Date: 12 – 14 Nov 2008
• Location: Melbourne, Australia

More details on their website

My interactive discussion is on the last day, just after lunch.

I was watching Jamie’s Ministry of Food last night and thought that his “pass it on” model would be great for propagating user experience skills. We teach our colleagues—who work in other disciplines—some basic UX skills, then they teach others, and so on and so forth. Start with the basics, then work up to more advanced topics.

Before we know it we’ll have a whole bunch of people who not only appreciate the importance of usability (and related concepts) and give it the attention it deserves, but whom can also get on with some of the work themselves. Thus the exponential “trickle down” or “snowball” effect could alleviate the reliance on consultants, contractors and specialist practitioners, as well as putting the necessary skills right at the heart of the problem throughout the process (because essentially everyone would be the “UX guy”). It’d be a lot less expensive than formal training courses too.

This approach could be supported by mentoring and providing the equivalent of the “Food Centres” that Jamie uses. I imagine this could be in the form of a central, online resource with more information for both ‘teachers’ and ‘guests’ to refer to.

What do you think? Is this any different to teaching cooking skills?

[Diagram courtesy Misterteacher]

Adobe. You may have heard of them. They’re one of the biggest software companies on earth, with some pretty great products—particualry since they bought Macromedia—and what seems to be almost a ubiquitous brand. I’ve not had a great deal to do with them besides use their products, and I have to admit I assumed they would have a respectable customer service ethic (similar to Apple whose reputation for product usability tends to imply they would do customer service equally well).

However, I’ve recently had a glimpse of some pretty poor customer experience. “Adobe ID” is the customer account you use to interact with Adobe on their website, be it for downloading product trials/updates, online purchasing or contacting customer service. When I first signed up for the Adobe ID which I currently use, I lived in the UK. As such my address and country and set to the UK. Since then I have moved back to Australia, and I think I’m well within the bounds of reason to expect to be able to update my details to reflect this change.

But no. I can update my phone number and address but not my country, which means the phone number and address aren’t valid. When I first moved back to Australia I tried to do this and had no luck. I contacted customer service and was told it wasn’t possible, I would have to close my account and open a new one in Australia. That’s pretty stupid for a “global” company, but I accepted the situation and tried to close my UK account. This proved far too difficult (I asked them how I do this and never received a reply) so I gave up. Not a big deal since I rarely use my Adobe ID and having the incorrect country and address didn’t bother me too much.

Just this week I logged into the Adobe website and noticed the UK address, so I thought I would investigate changing it again; maybe they had changed the ‘rules’? I tried updating my account details. I can change some details but not the country, because it’s just text not a form field. Then I noticed the “United Kingdom (change)” link in the very top navigation. Ah-huh! This looked promising! I clicked that and it gave me a choice of countries, I selected Australia and hit submit. The link now said “Australia (change)” and so I went back to the account details form to see if it had any effect. None. It still lists my country as United Kingdom and won’t let me change it.

Frustrated I thought I would contact customer service again. I filled in the form and complained about the situation, asking “How do I change the country for my account?”. Then I went back to whatever I was doing.

Eleven days later I receive an email from Adobe (at Thursday, 16 October 2008 20:07:03 EST):

Subject: Thank you for contacting Adobe Customer Care; your case has been closed.

Dear Patrick Kennedy

Thank you for contacting Adobe Customer Care. Our records indicate your service request 201300597 has been closed. If this request has not been completed to your satisfaction, or should you need to contact us on the particular issue again, please reference this transaction number. You can contact Adobe Customer Care at http://www.adobe.com/go/uk_supportportal/ or by calling 01 71 23 01 96 from 9:00 AM – 5:00 PM (CET).

In an effort to constantly improve service to our customers, we would be very interested in hearing from you regarding our performance. Would you be so kind to take a few minutes to complete our survey? If so, please click here:
[URL removed]

Thank you for contacting Adobe Customer Care.

What do you mean it’s closed? Nothing has been done about it yet! I logged into the customer support site to see what I could find out. I found this note attached to the case:

Thursday, 16 October 2008 11:07:13 o’clock BST

Dear Patrick Kennedy ,

What are the changes , your’re looking into changing ??

Yours Sincerely,

Adobe Customer Service
Tel: (UK) 02073650733
Tel: (Eire) 012421552
Fax: 0031 20 5820800

http://www.adobe.co.uk/support/main.html

Firstly, what a crap reply from customer service?! I said specifically what details I wanted to change, so why ask me? Then there is the poor spelling and grammar, which really makes me think Adobe have outsourced their customer service to a third-world nation.

Secondly, they don’t even email you when they update the case, so I knew nothing about this reply from their customer service staff. Thirdly, the ‘closed’ email was sent was only a few hours after the note was added to the case. How is that long enough for me to reply to their reply?

It seems to me that the deadline for their response to a complaint was looming, so they put in a quick and useless response and then closed the case. Someone’s “customer care” quota was one notch closer to being met. But “customer care” is not what resulted.

Naturally I complained about this, pointing out the flaws in their customer service. I doubt anything valuable will come out of it, but it’s worth a try. Ironically, this is the message I was presented with after my latest contribution:

A customer service representative will review your issue and get back to you within 24 business hours. Read more about case turnaround times. Once you receive a response from a customer service representative, you will have 5 days to respond before the case closes. After a case has been closed, if you still feel the matter has not been resolved, you may reopen the case.

Hmmmm. Last time they didn’t get back to me within 24 hours, and even if they did I wouldn’t have known unless I logged into their customer service website. And I sure wasn’t given 5 days to respond before they closed the case.

What a joke, get your act together Adobe. That’s a poor excuse for customer and user experience. I’m not asking for much (not asking them to fix a bug in a product) just to update the country attached to my Adobe ID account. If they want to encourage customers to use Adobe ID for greater interaction with customers, particularly as the main channel for complaints, you think they would ensure it works faultlessly.

Short stories are an excellent way of communicating user needs; not just use cases but needs, desires, joys and frustrations. While it was written as a creative short story, “To do” is a very realistic depiction of how someone might use personal organisation tools, and more importantly how they might feel about their experiences. Here’s a taste:

Every time I cross one item off the list, another seems to appear. Making a dental appointment should be a cause for celebration, a satisfying tick; instead, it leads to more appointments, white waiting rooms, condescending child-receptionists and x-rays. And while I now know exactly what I need to do, it’s getting harder to keep track of the when.

Read more

This story has feeling, it is human. Compare this to the way in which user experience requirements that have been captured for one of your projects. I think there’s a huge benefit in presenting the needs of our audiences in this way. It goes beyond a mere scenario, in that it uses the power of creative writing to put you inside the head of the person using—or should that be surviving?—the things we design.

I know the author of this story probably wasn’t describing a real series of events, but the story would have been inspired by real situations, pieced together to create a whole story. This story represents his experiences using to-do lists and other productivity tools. You can easily do the same thing based on the findings of design research.

My IA for Agencies survey is now closed. Thanks to everyone who took the time to fill in the survey, all 206 of you from across the globe.

I’ve also selected the winner of the prize, a copy of Don’t Make Me Think: A Common Sense Approach to Web Usability, 2nd Edition by Steve Krug. Congratulations!

Now comes the task of analysing the submissions and compiling the results into a useful format. I’m also compiling a list of lessons learned from the process, as there have been a few. I’ll make all of this available in due course (read as when I get some time to spend on it!).

Thanks again to everyone who filled in the survey, those who helped me with it, those who offered encouraging words and constructive feedback, and thank you even to those who felt the need to leave negative comments.

While not everyone agreed with the manner in which I conducted the survey, it’s all been very educational.

I recently conducted a quick survey among my colleagues and peers in the UX field, regarding how to determine which research methods to use for a particular project. My objective was to try and work out the four qualities of any research method that need to be considered when putting together a plan for design research.

For example, interviews don’t allow as many people as focus groups but they can dig deeper into the thoughts of the participant, thus ‘breadth’ and ‘depth’ might be two such differentiating qualities.

I’ll be using this information for an ‘edutainment’ game I’m including in my upcoming OZ-IA workshop, whereby participants will select a set of research methods for a given scenario and limited by a ‘budget’ for each quality.

I got lots of useful replies from a small sample size. Fortunately—I got more than what I had literally asked for (my bad…a poorly worded email). Almost everyone gave me great insight into how they go about planning research, as well as some gentle rhetoric regarding the subtle complexities of the subject and how perhaps I was oversimplifying :) So, I stand duly chastised, planning a research programme is definitely more complex than choosing four qualities, but that’s what I needed for my workshop game. Lesson learned :)

The survey question was framed in terms of user research for web design, but I expected that I would get responses from a broader perspective than this, and my recipients did not disappoint (eg BA’s gave their views from the point of view of business requirements gathering).

My findings were numerous, starting with the fact that it’s hard to analyse responses when they are all wildly different; some people ranked, some gave low/med/high values and some simply discussed the relative importance of the qualities I had listed in my question. So I had to interpret some of the responses in order to collate them. I should have used a proper survey form, but I was under the influence of spontaneity. Another lesson learned :)

Without further delay, here are the results of the survey. These are the top ranked qualities, taking into account all the information I received from respondents:

I’m not so concerned with the order, but it was clear the top factors were the breadth, depth and amount of analysis required.

The cost directly associated with a method were not seen as a useful differentiator since they are all much the same in this regard. Similarly the facilitation time, or face-to-face contact time, was not seen as a crucial factor since it can be varied as needed on a case by case basis, regardless of the method employed. Difficultly, or the effort required, was seen as a crucial factor only by a few people, the rest didn’t see it as a sufficient differentiator (perhaps reflecting their own level of experience in the field).

Total cost (ie time and materials) was seen by many as a crucial factor in that a research undertaking might “live or die” on the basis of this alone, but it’s an aggregate of facilitation time, analysis time and material costs, which are already in the list. So I removed it from the final list.

Two other factors are worth a mention. Firstly, rigour was nominated by one respondent as a reasonably important factor. I took this to mean how easily the research method can be repeated with consistency across sessions. However, I grouped this with difficulty, or effort required, because I see it as a quality of the practitioner rather than the techniques themselves.

Secondly, the difference between qualitative and quantitative analysis was suggested as a key consideration. That is, some techniques will require more discourse analysis as opposed to data analysis. Excellent point, but probably too fine-grained for my purposes.

My conclusion is that I should use reach, coverage, analysis time and effort required as the differential qualities for my workshop game. Feel free to comment on this below.

Thanks to everyone who responded to my email, and for taking the time to make sense of it. Thanks also to James Breeze for reminding me to blog about this…I have been neglecting my blogging lately!

I’m running a survey and I’d like your help.

Best practice design of websites, and other digital media, involves a set of skills known broadly as Information Architecture (IA) which generally means making designs user friendly. IA is also known to people doing this work, by such terms as User Experience (UX), User Centred Design (UCD), Interaction Design (IxD) or simply “usability”.

A significant amount of this sort of work is performed by agencies—whether they be advertising agencies, digital agencies or communication agencies. As a practitioner and educator in the field of IA, I am interested in learning how people go about practicing it, in particular how agencies “do IA”. This is to both confirm and challenge my own understanding of the way agencies work and how IA fits into their processes, who it gets done by and how it might be possible to give agencies the skills they need to perform better in this regard.

To this end, I’ve launched an online survey to get some answers straight from the people who work in agencies (or used to). The survey will take approximately 5-10 minutes to complete and I’ll give away, to one lucky person who completes the survey, a copy of the acclaimed best-selling book by Steve Krug Don’t Make Me Think: A Common Sense Approach to Web Usability, 2nd Edition. I recommend this book for anyone considering doing anything to do with IA or usability, so it’s only fitting to offer it as an incentive.

You’ll find the survey at: http://www.gurtle.com/survey/index.php?sid=61824

If you don’t work in an agency, you can still help me out by forwarding this to your clients, peers and friends who do work in agencies. I may run a more general survey in the future, but for now I’m focused on agency folks.

The survey will run until the end of September, so there should be plenty of time for word to get around.

I’ll share the results of the survey (aggregated not raw data), either here on this blog or through conferences in and around the IA community. Stay tuned.

What do you do when your client seems to be missing the target?

Take, for instance, the situation I recently experienced regarding a fashion-related website whose owners have—how shall I say—an aspirational view of their audience. That is, they want to target an older, more sophisticated, style conscious women. I’ll call them the ‘fashionistas’. This happens to be the audience for their parent brand, an offline publication, and also happens to be the people with whom the brand is most associated.

However, the reality is that those who use the website are younger women who primarily want to chat about gossip, fashion bargains, celebrities and the like. I’ll call them fashion ‘fans’.

I suppose there is nothing wrong with having an aspirational target audience; those people whom you would like to be your users. But research shows that the fashionistas are simply not interested in using a website; it’s too low rent, not part of their world. These are people who sip lattes in exclusive fashion boutiques. They live high fashion, they’re not chatting about it with hotThang267. Lol. Wtf?

Research was conducted into a broad cross-section of fashion-minded people and it showed this segmentation; the fans, who are less capable of actually partaking in designer fashion, were at one end and the fashionistas who read the magazine, were at the other. The latter were unlikely to use the website. Unfortunately, the client said we recruited the wrong people, because the participants didn’t match their audience. Rudimentary web analytics showed that the most used part of the website is the discussion forum, and the demographic of users of the forum matches the ‘fans’ segment. More weight to the argument that fashionistas don’t use the website.

I think there are two main reasons for this. The first is something that many designers and experience architects have to deal with: clients who design for themselves instead of who will actually be using their product. My clients are, or want to be, the fashionista. They are style conscious and do participate in the world of high fashion. Through the website design process, they’re trying to cater for themselves, either consciously or subconsciously.

The second reason, closely related to the first, is that the client won’t accept who the real audience is. It reminds me of the “serious” rock band who refuses to accept that their fan base is predominantly 14 year old girls from middle class suburbs—that just ain’t cool. I believe this issue is part and parcel with the field I work in; as people creating websites, magazines and advertising get older, wiser and wealthier, it is increasingly difficult to relate to audiences that are not in the same life-stage. Is there a point at which you can’t, or don’t want to?

Aiming for a certain target audience is one thing—stretching yourself and your product—but it would appear in this case that it’s unrealistic since the people they’re aiming for won’t use their product. Gone are the days when online properties were simple, “me too” attachments to offline brands, with typically little more to offer than a placeholder for the domain name. Now days, these websites need to stand on their own and be successful, even if that means catering for a markedly different audience.

In the future, the ‘fans’ might very well grow up to become the ‘fashionistas’ that my client wants to target, but how long is that going to take? And would they continue to use the web if this did happen? Perhaps I’m missing the point; perhaps the plan is to leverage the fact that the ‘fans’ aspire to be ‘fashionista’, and that’s part of the attraction of the brand. If we targeted the ‘fans’ in a way that reflects their actual age and socio-economic status, maybe they would no longer be attracted because that wouldn’t be “on brand”. If this is the case, it should be openly stated, which hasn’t been.

Having said all this, the website is being developed with functionality for the ‘fans’, so the issue I have is really the perception my client has of who their audience is. Yet this is still troubling, since if you don’t have a clear picture of who you’re designing for, or if the team is not in total agreement regarding this, it’s difficult to design and deliver successful solutions.

I’m not sure how this will play out, my clients are good people and excel in their respective fields, but perhaps are “too close to it”. I’m also open to suggestions, have you found an effective way of addressing a situation such as this?

Photo credit: Missed the target… by malavoda.

I recently saw a post to a mailing list where someone was asking for a consultant with expertise in Typo3, an open source CMS. Now, I know nothing about this CMS, and I’m sure it has its pros and cons, but isn’t that a strange name for something whose primary purpose is to allow people to type in content?

Talk about tempting fate, you’re almost willing your users to make a typo! I’m sure there’s some psychological pattern of subliminal influence that will result.

It’s like calling a telephone “Stutter” or a brand of high heels named “Tumble”.