How the Accessible Usability Scale Makes User Research More Inclusive

by Alwar Pillai, CEO and Co-Founder, Fable

I started Fable, an accessibility platform powered by people with disabilities because I wanted to change the status quo around accessibility. While creating a platform to research and test products with people with disabilities is a transformative shift, it doesn’t solve the whole problem. We also need to make the way we research and measure usability more inclusive.

It wasn’t until I worked directly with people with disabilities that I learned how current research methods limit our ability to design good products. Despite having formal education and years of experience in user experience (UX), I had such a limited view of what a good product was. A good product isn’t just one that I can use – it’s one that anyone can use. This sounds obvious, but a significant amount of the digital world doesn’t work for people with disabilities.

If our UX research methods don’t consider the needs of diverse participants, we will consistently miss the mark when it comes to building inclusive products. We need to make a change, and we need to make it now.

This article explores the limits of some popular research methods and how they can hinder a researcher’s ability to include people with disabilities in a meaningful way.

Then it’ll introduce you to the Accessible Usability Scale (AUS) developed by Fable, an adapted version of the System Usability Scale (SUS) for assistive technology users. It is already enabling a shift in making user research more inclusive, by scoring the experiences of people who use assistive technology – which is over 1 billion people around the world.

What Current Methods Can — and Can’t — Do

One of the best ways to think about Inclusive Design is “one size fits one,” instead of “one size fits all.” There are many effective research methods, but their effectiveness is not equal with all participants.

Consider a think-aloud study, where you ask participants to use a product while verbalizing their thoughts as they move through its user interface. When you have a participant who catches on to this method, it can be incredibly rewarding. It also has the advantage of being cheap, robust, and flexible.

However, this test inherently poses issues for some people with disabilities. For example, consider a participant who is blind and relies on a screen reader. While listening to their screen reader, they’re also being asked to talk.

Either they’ll talk over the screen reader, which can be particularly challenging for a researcher, or they’ll navigate in an unusual way by taking breaks to explain what they’re doing and pausing. For another person who uses a form of alternative navigation, voice navigation, for example, the assistive technology may hear commands and overlapping speech from both the user and researcher.

There are certainly ways to make think-aloud studies work for both screen reader and voice navigation users, but it’s a notably different experience compared to those who don’t rely on these technologies. We need to keep this in mind and consider how the methodology impacts the user.

Fable researcher Elana Chapman worked closely with a member of Fable’s community to develop a method to conduct a think-aloud type of study. In this case, however, the individual is non-verbal. With some video editing and captioning tools, they adapted the think-aloud study to be more inclusive. This is exemplary.

Researchers who prioritize the convenience of the participant above their own are going to have more inclusive data. It’s important for us to adapt our engagement methods to be more inclusive of our wide range of testers. With that in mind, the creation of this new method prioritized finding a comfortable communication method for both tester and researcher.

Another common method for user research is to host a usability lab. While a usability lab can have the benefits of facilitating observation through a one-sided window, environmental controls, and increase the efficiency of a study, it has significant drawbacks.

For people with disabilities, it’s often asking for participants to use unfamiliar technology in an unfamiliar environment. Given that assistive technology is almost always customized, observing participants with disabilities in a usability lab with unfamiliar equipment can result in observing unnatural behaviors and drawing the wrong conclusions. Further, travel often requires allocating more time for transit than for the research itself.

For example, it’s not uncommon for a motorized wheelchair user to spend two hours getting ready to leave the house, two hours waiting for transportation, and over an hour in transportation – just to participate in a 60-minute research session.

Add to that, the environment may not have an accessible washroom, or the participant may require a disability support worker to travel, and it becomes a cumbersome experience for the participant. For these reasons, we often emphasize the importance of remote testing. This involves users testing solutions from their own comfortable environment with the assistive technology and tools that are familiar to them.

The Measurement Problem

In addition to methods, inclusivity issues also extend to measurement. One of the most common tools used to measure usability is the System Usability Scale (SUS). The System Usability Scale provides a “quick and dirty” score of how usable a product is. It consists of a 10-item questionnaire with five response options for respondents ranging from strongly disagree to strongly agree.

Its uniformity – and simply how long ago it was developed – poses challenges. This is especially true for people with disabilities who rely on assistive technology. SUS does an exceptional job at scoring, handling acceptance bias, and being broadly applicable. Although being broadly applicable is also a downside. At Fable, we felt like the SUS model needed adapting to accurately capture a set of underrepresented users.

An Experiment in Inclusion

Fable released a tool called the Accessible Usability Scale, which we have made freely available. It is specifically meant for conducting research with assistive technology users. The AUS is a 10-item questionnaire that has the same scoring structure as the SUS. The key difference is that AUS is inclusive of the abilities and experiences of assistive technology users. It is not meant as a replacement. Rather, the tool is a complement to ensure that usability measurement includes everyone.

Here are some examples of how we modified the SUS to include the needs of assistive technology users.

There is a difference between requiring technical support and requiring someone who has sight. There is a difference between most people and most people like me. When individuals choose to participate in research, they are making a commitment. As researchers, we must ensure that we meet all participants halfway and that our methods are inclusive and can capture insights accurately.

Average AUS Score Based on Assistive Technology Type

Here is some preliminary data based on those using our free online AUS survey tool. The average Accessible Usability Scale scores from screen reader users are lower than those of magnification and alternative navigation users, suggesting there is still more work to do on major platforms, particularly for screen readers.

An Invitation

Ultimately, I am confident that we can make user research more inclusive. No researcher is intentionally excluding users. Research methods we use can dictate who we choose to include and exclude.

By being flexible and prioritizing the needs of our users, especially the ones who have been excluded, we can build products that are inclusive. The AUS is a step in the right direction, and I can encourage you to take that step and try it out.

Equal Entry

Accessibility technology company that offers services including accessibility audits, training, and expert witness on cases related to digital accessibility.