This article is based on a talk Jesse Beach who supports a team of engineers at Meta and a founding contributor to the jsx-a11y-eslint-plugin, gave at A11yNYC.
During the pandemic, Jesse Beach got into podcasts especially those related to history and Dungeons and Dragons. This is the story of her journey to build automated accessibility testing tools. It turns out that that story has deep roots in time.
The Story Begins with Herman Hollerith
Herman Hollerith, engineer of mines, joined the staff of the Massachusetts Institute of Technology in 1882 where he taught mechanical engineering. Four years later, in 1886, he proposed the idea of using punch cards to transport data for the 1890 census.
He later founded the Tabulating Machine Company which in 1924 would be renamed the International Business Machines Corporation, or IBM. In the late 1880s, the punch card was used for the Census. This is one of the first times they see digitalization of information that will eventually launch the industry.
In 1986, 100 years after Herman Hollerith proposed the punch cards that led to the founding of IBM, Jim Thatcher built the first screen reader, the IBM Screen Reader (for DOS). The field of assistive technology will be driven by users, enthusiasts, and professionals for decades to come.
This is the tech that will eventually inform the guidelines that will shape our thinking around automated accessibility testing.
Jumping ahead, the graphical user interface had been in the works since the ’70s, but the personal computer boomed in the ’90s. In 1994, the price of computers had finally fallen enough that a middle-class family like Jesse’s could finally afford a computer. Jesse wanted one for one reason: To play the LucasArts X-Wing and TIE Fighter games.
The computer her family bought did not have sufficient RAM to play these games. But somehow, she learned that you could update the computer’s BIOS to distribute some memory into HI MEM, and that got the games to run. Her life as a software engineer had begun.
Jesse’s 13-year-old self was unaware that four years prior in 1990, the Americans with Disabilities Act (ADA) was signed into law by George H.W. Bush. Thirty years of disability advocacy from Berkeley to DC had culminated in landmark legislation.
The ’90s were capped by another big moment. Version 1. 0 of the Web Content Accessibility Guidelines was published in 1999. The web as most people knew it was barely six years old.
Then came the catastrophic bursting of the web bubble in the 2000s. It felt like the end of the web, but it was just the beginning. The aughts were all about blogging. The Drupal content management system came out in 2001 and WordPress came in 2003.
The aughts were about browsers and browser development tools. Firefox came out in 2004 at the same time Jesse graduated from college. In 2006, Firebug developer tools were released that allowed developers to inspect CSS in the DOM. Firefox added the developer toolbar in 2007. Then the game changer occurred in 2008 when Google Chrome dropped. The field of web development was finally a real field.
Developers suddenly had these incredible tools and the web was alive with page-building frameworks. Just as the business of the Web was growing up, so was digital accessibility. Apple introduced VoiceOver in 2005. This was a screen reader that was built into the Macintosh operating system.
WCAG 2.0 finally came out in 2008. This was the guideline that would shape Jesse’s work through the 2010s. It introduces the grouping concepts of perceivable, operable, understandable, and robust (POUR). It also introduced 61 guidelines, and most crucially, that all of these guidelines be testable with success criteria.
Javascript and Open Source
Right as WCAG 2.0 was coming out, Jesse was reeling from her first start-up failure. Nearly penniless, she moved to Cambridge, Massachusetts, looking for a job in tech. What she found was a blossoming web scene, full of energy and ideas.
One cold night, she joined a small group in a classroom on the campus of the Massachusetts Institute of Technology, Herman Hollerith’s old haunt. There were a dozen people in the room. The energy was buzzing.
John Resig was there to talk about his new idea. He had just released jQuery. It was very new. People didn’t know the concept of a framework to build JavaScript with. Coming out of that, Jesse wanted to work in user interface (UI) development.
JQuery unleashed a kind of Cambrian explosion of open-source tools for front-end development and frameworks over the next decade. Each week, a new package was released. Developers barely had time to get one installed in a project before the new one was released and improved upon the last one.
In 2009, Selenium WebDriver (Selenium 2. 0) entered the ecosystem and with it you could instrument end-to-end tests in a browser. In 2010, the node package manager (NPM) was released. The NPM allowed developers to share code easily. The year 2011 came with testing platforms like Mocha JavaScript test runner. Writing unit tests suddenly became easier.
In 2012, the Grunt JavaScript task runner arrived and helped to automate all of the steps in building a package or a web page. These projects and those like them provided an ecosystem for building projects in JavaScript. Everyone was building more and more complex web pages. And web accessibility was getting worse and worse because of it.
It was about this time that Jesse joined the office of the CTO at a company called Acquia working for Dries Buytaert, who founded the Drupal Project way back in 2001, right after the dot com bubble burst.
Drupal 7 had just launched in 2012. During the development of Drupal 8, someone needed to lead the accessibility gate to ensure Drupal 8 would be accessible. Jesse volunteered though she had no idea what she was doing. She had never worked in the field of accessibility and needed to accelerate the work.
QuailJS
Jesse found a project called QuailJS. Everyone was building continuous integration (CI) tools. Kevin Miller at UC Monterey, had ported a testing library over from PHP to JavaScript. Right into this world of JavaScript tooling and frameworks and continuous integration tools, it was a beautiful idea.
She made her first commit to QuailJS on November 17, 2018, and this was arguably her first commit to an accessibility project. The name of that commit or the title of that commit was: Added grunt convert to the build. Converted a guideline… Converted the guideline YML to a JSON file in the dist directory. This is a perfect 2013 title for a commit.
The group of volunteers took the premise of WCAG 2.0 that the guidelines are testable and tried to automate that testing. WCAG had success criteria that defined how a guideline should be tested. The team wrote tests for all of the guidelines.
They took the guidelines and turned them into JavaScript. They referred to the pieces of the guideline in the testing code. That way, when they evaluated and came up with violations, they could reference the parts of the code that those were for. Wilco Fiers spun out the AutoWCAG W3C group, which attempted to formalize the definition of an issue that you would find in an automated testing system. So, that they could have interoperability between the found violations.
Fiers went on to work for a company called Deque that built up an automated testing service that they run today. By 2013, you could automate about 33% of the guidelines. You can flag another 33% of the guidelines for a human to look into it further. And the rest of it needed a human to come in and make a call.
The team also discovered that the work of building an automated testing platform against WCAG 2.0 is about 10% coding up the tests, and about 90% dealing with web scraping, collecting violations, categorizing them, collating them, turning them into tasks directing them towards an engineer, getting them to fix the task, and verifying that the task is fixed. In essence, they were trying to build a company inside of an open-source project, and it wasn’t working.
QuailJS was ultimately a failure. It ended like so many open-source projects. While Jesse’s journey with Drupal was coming to an end, she wasn’t about to give up on automated accessibility testing.
What’s going on in the environment at this point? Around 2014, it became clear that QuailJS would not work, the frontend ecosystem was flipped on its head, and React had arrived. React turned into open source in May of 2013. A month later, the eslint project was released. React introduced the idea of representing HTML in JavaScript with JSX or Java syntax extension. JSX produces React elements that then render out to the DOM. S
Linting and Reacting
Instead of having your HTML file, CSS file, and JavaScript file, that get pulled together by your grunt task manager and packaged up, you’re running your JavaScript code in the file itself. And this is a big deal.
Eslint and other linting tools that were being developed around this time provided the means of transposing JavaScript, React, JSX into what’s called an abstract syntax tree or representation of that code, that they can then go and traverse. Run Bools against, and evaluate arbitrarily.
ARIA
These two technologies changed the game in the frontend world. ARIA 1.0 came out in 2014. It stands for accessible rich internet applications. The W3C Web Accessibility Initiative’s accessible rich internet application or WAI-ARIA guidelines provided the semantics or UI patterns. Crucially, it offered crucial patterns for things that existed on the web like buttons and input forms and forms.
It also gave patterns for things that didn’t exist, such as sliders, switches, tooltips, and things built into web applications. But there was no way of expressing to an assistive technology, to say: This is what the UI is. React made it easy to declare the UI in its code. To represent the semantics, the states, and the properties that were defined in the ARIA spec.
In a couple of years, the industry had everything they needed to assert the correct shape of the UIs under the guidelines of WAI-ARIA. In January 2015, Jesse attended the second React Conf. Five months later, Jesse went to work for Facebook as an accessibility engineer.
A month later, Matthew King joined Jesse’s team as a subject matter specialist. He had just come over from IBM where Jim Thatcher was his mentor. Matthew King was also working on ARIA 1.1. These two specialists come together to work on this idea.
Jesse tried to update the WebDriver end-to-end, continuous integration system. This was Selenium WebDriver. It drove the end-to-end tests on the web. They also had other drivers that were driving their mobile platforms as well. They were trying to get in there and run accessibility checks inside of preexisting end-to-end tests. As Selenium was clicking through, they would check to see if the button that was just clicked on had, for instance, a label or keyboard accessibility.
But what is the really difficult part of trying to work with end-to-end tests? They had to convince engineers to write them. If end-to-end tests are a crucial point in your strategy for what you’re doing, Jesse advises thinking of a different strategy. End-to-end tests are probably the least written tests. They have their time and their place. But if you’re looking for wide coverage, you’re just not going to get it with this method.
Eventually, Jesse came to the same conclusion that she came to with QuailJS. She wasn’t going to write tests, get the output, get them into tasks, and get them to engineers at any sort of scale that was going to change the problem. During these months, Matt and Jesse spent late nights in the office talking about UI patterns, how to represent them and test for them.
Jesse’s struggles and insights informed Matt’s thinking on ARIA. His struggles and insights on ARIA informed the way Jesse thought about UI testing. Over the holiday break of 2016, Jesse wondered if she could write a lint rule that could do something very simple. Something like raising an error when it found a button that didn’t have a label.
Eslint and JSX
Jesse spent the holiday break writing this lint rule, learning how to write rules for eslint. And it turns out you can do this! You can find buttons in your codebase for the web and then look to see if that button declares that it has text content. And then she asked herself if she could do this for every button in the codebase. Yes. You could run this and figure out how many buttons don’t have labels in your codebase.
Could she write rules to check for other things? Like if an element has a click handler, can you check to see if it has a key event handler? Yes, you can do that. You can raise a flag. If it has an ARIA role, can you check it against a list of existing ARIA roles? Yes, you can do that. If it is an interactive element, does it support focus? This one is tricky.
Jesse searched GitHub to see if anyone else had this idea. And of course, someone had. It turned out that Ethan Cohen had started a project called the eslint-plugin-jsx-a11y project. He had started this just a few months before Jesse had this insight.
A couple of days later, on December 21, 2016, Jesse landed her first commit to this project with the title: Added a paragraph to onclick-has-role.md, to explain the use of role presentation. It was a very simple commit to the documentation. That was bug number 129.
With this linting plugin, eslint-plugin-jsx-a11y, they targeted the ARIA spec. It defines a few dozen patterns for web elements. Using eslint and JSX elements in React allows them to validate those details against the spec. For example, if an element has the role of a button, does it also have a label? If an element has a click handler, does it also have an interactive role?
They asserted that the code in JavaScript, in JSX, matches the patterns that are defined in the spec. And the added benefit of lint rules is that the violations are raised in context. As you’re working, the lint rule pops up.
Or you run a script, as you’ve been working, and you get a list of violations that you’ve created in the course of your work. There’s an immediacy to the violation and the developer and that can get resolved quickly. Essentially, they eliminated the whole requirement to build an ecosystem around this project.
In 2017, Jesse started contributing to this plugin in earnest. She met with the creator and what she planned to do, which was to refactor this plugin from the ground up. Specifically, she wanted to take all of the content and knowledge of the spec of ARIA out of the plugin and put it in its own library.
They also had this added problem that they had to represent the semantics that are inherent in the web browser. When you put a button on a web page or an input on your web page, that element has a file behind it. A class that represents the header of that element.
AXObject query library
If you go and look in the guts of Chrome, you’ll find a directory that lists all the elements, the classes for the elements, to describe the behavior of those. It also lists the states and properties that are supported, and they needed to understand what the mapping of those to the ARIA spec would be.
They had a second library, the AX object query library. Jesse proposed the ARIA query library and the AX object query library. These were going to contain the information about the spec and the browser that the lint plugin would then refer to. The lint plugin would contain the rules. This separated these concerns to be developed in parallel. Jesse set about this task and it took about four months to do this refactor.
On January 23, 2017, the ARIA query version 0.1.0 was released. Three months later, Jesse released the AX object query version 0.1.0. They were able to write our rules faster. They went from half a dozen rules in the plugin to many dozens, and they were able to test things like are you trying to convert an interactive element to a non-interactive element? Are you trying to make a button into a div? Are you trying to make an input or a form into a list item?
They wanted to disallow these kinds of transformations in HTML because it made the semantics difficult to express to an assistive technology. And because they could reference these two libraries, they had what they needed to essentially list out all of the interactive elements or the non-interactive elements, like the main element or the main role or the list elements, which are non-interactive, like links and buttons.
These rules were running in real-time. Hundreds of developers at Facebook used them. Jesse was getting real-time feedback on whether or not they were working. She could count how many violations they would find at various points. And to iterate on them fast. When you have thousands of developers and tens of thousands of JavaScript and React files to look at, they developed this plugin at high speed.
This was the point where Jesse felt like she finally had what she had set out to do about right years prior. And that was to develop an automated testing something for accessibility. And it was working well. How well?
In 2017, she wondered if she could write a script that’s going to check out the Facebook codebase a month before the current date. She’d run them, collect all the violations, put them in a table, go back another month, check out the codebase, and run the rules.
What Jesse found was that over time violations of the rules increased in a vaguely exponential fashion. They saw that this trend was going to continue if they didn’t intervene. They intervened. They took the rules that they had written and flipped them around. They turned them into code that went and found the violations and updated the code. And they were able to eliminate hundreds of violations with scripts that wrote code.
For example, the code would find icons. They mapped icons to specific labels like an X button to the close label. And they were able to add labels to all of their X buttons with one pull request. They did hundreds of those at a time. Those mappings had to be tweaked. Sometimes you want “exit” for that label. Sometimes you want “close.” But having some label was more than having none. They were on their way to fixing these problems.
They took this code and did a little bit more with it. They pulled it into the browser and had it running in real-time. This let us raise these violations for engineers and expose more of the spec. The difference is that when you run a lint rule over static files, you’re running them over components that might be broken apart in specific ways. You might not be able to say that a component doesn’t have a label because its content is rendered by a function. That function might return a label, or it might not.
At that point, you may have to bail. But if you can pull those components together and render them into a browser, you can make some strong declarations about whether or not a component matches the WAI-ARIA spec or not. That has been running inside of their codebase for quite a while, and catching some esoteric bugs.
Video Highlights
- Exploration of the past beginning with Herman Hollerith
- QuailJS
- Joining Facebook
- AXObject Query
- Q&A with Jesse
Watch the Presentation
Bio
Jesse Beach‘s technology career started in linguistic research, shifted to UX design, and then landed solidly in front-end development. At Meta, she channels her energies into building tools and components that support accessible interface development. She believes that all humans should have access to information and services, whatever their abilities or circumstances.
Jesse is a founding contributor to the jsx-a11y-eslint-plugin, a contributor to the QuailJS automated accessibility testing library, and creator of the aria-query and axobject-query libraries.
