A little history

Usability testing showed me that UX was legit.

At the beginning of my career I was a “Junior Human Factors Engineer” and my job involved reviewing specification documents, painstakingly red-lining individual words and sentences across pages of text to ensure they properly described how our products would meet human factors requirements. I didn't feel particularly impactful doing this and it didn't feel like I was using anything I learned in school in my job. But then I ran my first usability test and wow, that was a revelation.

For context, this was the mid-90’s and “usability testing” was not common to see in practice. Where I worked, no one had ever conducted a usability test, much less knew what it was. I was tasked to assess a prototype product and I remember thinking, this sounds like the exact scenario for this method I learned in school, usability testing. To justify this approach to other engineers I told them, “we’ll get more valid results if we watch the people we’re building this product for (aka, target users) actually use it.” It was a little bit of a hard sell. Assessing a product in this manner was very unique at that time. Normally, we would sit in a large conference room with all the stakeholders and look at engineering drawings projected on a wall, one page at a time, point out flaws and argue about them.

I used low fidelity prototypes (paper) to conduct the usability test so test participants completed tasks by interacting with hand-drawn “screens”. I still remember two eye opening moments. First, I was genuinely surprised at how participants generally lost themselves in the interaction with the paper screens. This exposed actual interaction usability challenges and the participants generated insightful comments around usage and how things worked as opposed to just look and feel. Second, and even more importantly, the engineers who watched the sessions believed the problems we saw were problems that they needed to fix. I had been working with this group for many months and convincing them to change anything was always extremely challenging.

This pattern played out repeatedly over the course of my early career. We’d run a usability test and a skeptical product team would watch real people struggle and suddenly be convinced that yes, we have issues, let’s fix them. It's been a key factor in solidifying usability testing into a foundational tool in our UX toolbox. Over the years, usability testing methodology has become more streamlined and efficient. Remote testing is the norm and unmoderated testing (with no facilitator) is commonplace. At the same time, perhaps as an unintended consequence of its ever increasing efficiency, usability testing has become viewed as a tool with clear limits on its scope of impact. Anecdotally, usability testing doesn't inspire the same excitement with a lot of UX researchers since it doesn't drive innovation like discovery research techniques. I think it's a good time to revisit just how much impact a simple usability test can have, and a recent project gave us the opportunity to do just that.

Revisiting usability testing

One of my favorite client scenarios is introducing UX methodology (aka user centered design) to UX immature organizations. As an established UX research and design firm, we have the advantage that UX-immature organizations that reach out to us already believe that UX is important. They just don't know what it looks like to properly apply UX methodology. They've never felt that "aha" moment when you connect the dots between really understanding your users and using that information to impact how you design your product.

Recently, a product manager reached out to us with an initial ask (paraphrased): "We need to base our product decisions on understanding our users, and we don't. Can you help?" The product manager felt that ideally, we'd do an extensive discovery activity to build detailed personas and journey maps. However, with a limited budget and a management chain that did not fully appreciate the value of UX work, we decided that a usability test would give the best bang for the buck. Not only would it identify immediate and actionable things they could improve (addressing their management's desire to see tangible payback on their investment), but we hoped it would lay down seeds for teaching the product team the value of understanding their users at a deeper level.

This usability test was a throw-back because we ran it fully in-person. The test participants came to a specific testing location, the facilitator facilitated the test face to face, and the product team stakeholders attended and observed the test in person (from a separate observation room). We also dedicated a larger than normal portion of the pre-test interviewing to understanding the participants' backgrounds and motivations for using the product.

The usability test was, as typical, very effective in identifying problems. But what stuck out to me this time was how the product team’s in-person attendance enhanced the impact of the usability test. After each session, I would immediately walk to the observation area and engage with the product team. What stuck out to you? Do you have any follow up questions you'd like me to ask? This is where the in-person interaction really showed its value. I could see the expressions on team members' faces and read their body language which helped me to guide our conversations - pushing topics that resonated with them and building their enthusiasm towards wanting to fix things. I could also guide the conversations to focus on root causes of issues and away from reflexively trying to create solutions for the most recent problems we saw.

The impact

For this project, we had an important underlying goal: to educate the product team on a user-centered approach to product development. The usability test was our vehicle to do that and the in-person nature of the activity really enabled us to tailor how the team received the information from the testing. We drove these in-person discussions to force the team to constantly talk about the characteristics we were learning about their target users, especially, their users' goals and motivations, the paths their users took in trying to achieve their goals, and using that information as the basis for product design decisions. The basic structure of the usability test - real users doing real tasks and the opportunity to talk to them more deeply about their goals and motivations - gave us a solid foundation to learn more deeply about their users. But the additional aspect of the in-person stakeholder attendance greatly enhanced our ability to embed and teach basic principles of user centered design.

I'll leave you with one anecdote that made me feel that we really did accomplish our larger goal of educating the team. At the end of the project, we did a findings workshop to immerse the team in the results and kick-start their design solution process. When every group's proposed solutions all started with commentary around what they saw users trying to do, I knew we had accomplished our higher goal of getting them to think more user centric.