The Accessibility Tales: Stories from the Before Times

JESSE: Thank you all for having me to town to give this talk. When you proposed this, a kind of history of my experience building automated accessibility testing tools, I thought it would be a fun time to reflect. So... Hello, everyone. My name is Jesse Beach. I support a team of engineers at Meta. We're part of a group that's responsible for the Facebook design system.

In fact, some of those engineers are here with us in the audience today. Hello, engineers! And that also includes accessibility support. Big part of the design system is accessibility support.

So during the pandemic, I got into podcasts, and specifically two subjects. Dungeons and Dragons, and history. I see some people nodding. I learned that who we are today is the function of everything that came before us. And I also learned that Silver Barbs is a really good first level spell. I want to tell you the story of my journey to build automated accessibility testing tools.It turns out that that story has deep roots in time. Some of the details are part of the plot, some are rhymes in time and some are just fun. So here we go, way, way back. But before, I want to make sure we describe some of the images on the screen. We have... A picture here of some cats around a campfire, telling stories. Evocative of what we're about to do. Herman Hollerith, Engineer

of Mines, joined the staff of the Massachusetts Institute of Technology in 1882 where he taught mechanical engineering. Four years later, in 1886, he proposed the idea of using punch cards to transport data for the 1890 census. He later founded the Tabulating Machine Company that in 1924 would be renamed the International Business Machines Corporation, or IBM. This little company will show up again and again in our story. On the screen, we have an image of a punch card from the late 1880s, that was used for the Census. This is one of the first times we see digitation or digitalization of information that will eventually launch the industry that most of us work in today.

The '80s, the 1980s, specifically 1981. I was born. Five years later, In 1986 - 100 years after Herman Hollerith proposed the punch cards that led to the founding of IBM – Jim Thatcher, then employed at IBM, built the first screen reader, the IBM Screen Reader (for DOS). The field of assistive technology would be driven by users, enthusiasts and professionals for decades to come.

This is the tech that will eventually inform the guidelines that will shape our thinking around automated accessibility testing. I, however, had yet to learn to use a spoon. Jumping ahead, the graphical user interface had been in the works since the '70s, but in the '90s, that's when we saw the personal computer boom, and the price of computers had finally fallen enough in 1994 that a middle class family like mine could finally avoid a computer. And I wanted it for one reason: To play the LuasArts X-Wing and TIE Fighter games. This was my introduction to computing, out of sheer desperation. The computer that we bought did not have sufficient RAM to play these games. But somehow I learned that you could update the computer's BIOS to distribute some memory into

HI MEM and that got the games to run. My life as a software engineer had begun. I see some nods here. Maybe some other hackers of the BIOS back in the '90s! At this point, I think I became a software developer. I was hooked on this power with the computer. My 13 year old self was unaware that 4 years prior in 1990 The Americans with Disabilities Act (ADA) was signed into law by George H. W. Bush. 30 years of disability advocacy from Berkeley to DC had culminated in landmark legislation. The 90s were capped by another big moment. Version 1.0 of the Web Content Accessibility Guidelines were published in 1999. The web as most people knew it was barely 6 years old. On the screen we have a bunch of unicorns and a giant bubble.

This was the 2000s, which started with the catastrophic bursting of the web bubble, and for those of us who were aware, it felt like the end of the web, but it was just the beginning. The oughts were all about blogging. In 2001, Drupal content management system was released, in 2003, WordPress was released. The oughts were about browsers and browser development tools. In 2004, the Firefox browser was released. I graduated from college. This was the first real browser. In 2006, Firebug developer tools were released and with it, we could finally inspect CSS in the DOM, in 2007, the developer tool bar was released, I graduated from grad school, and we all lost our collective minds in 2008 when Google Chrome dropped. It was a game changer. The field of web development was finally a real field. We suddenly had these incredible tools and the web

was alive with page building frameworks. And just as the business of the Web was growing up, so was digital accessibility. In 2005 Apple introduced VoiceOver, a screen reader that is built into the Macintosh operating system. And then in 2008 the second version of WCAG was released, WCAG 2.0.

This was the guideline that would shape my work through the 2010s. It introduces the grouping concepts of perceivable, operable, understandable, and robust. The shaping concepts of our field. It also introduced 61 guidelines, and most crucially, that all of these guidelines be testable. With success criteria. This concept of a guideline that we could test comes into the world around 2008. On the screen we have a picture, sort of cartoony, of a classroom, someone standing up at the board. A picture of someone standing in front. Much like the scene we're part of today. Right as WCAG 2.0 was coming out, I was reeling from my first start-up failure. Nearly penniless, I moved to Cambridge Massachusetts, looking for a job in tech. What I found was a blossoming web scene,

full of energy and ideas. One cold night, not unlike a night like this, I joined a small group in a classroom on the campus of the Massachusetts Institute of Technology, Herman Hollerith's old haunt. There were a dozen people in the room. The energy was buzzing. John Resig was there to talk about his new idea. He had just released jQuery. It was very new. People didn't know the concept of a framework to build JavaScript with. Finished up his talk, neat and tidy. Opened up the room for questions. Coming out of that, I knew that I wanted to work in UI development. On the screen is a picture of a cafe. People working at their laptops. JQuery unleashed a kind of Cambrian explosion of Open Source tools for frontend development and frameworks over the next decade.

Each week, a new package was released. We barely had time to get one installed in your project before the new one was released and improved upon the last one. And everyone wanted to work out of a cafe. In the early 2010s. In 2009, Selenium WebDriver (Selenium 2.0) entered the ecosystem and with it you could instrument end-to-end tests in a browser. In 2010, NPM was released. This is the Node package manager that allowed us to share code easily. I don't know if anyone tried to share code before that. But you ended up on some pretty dicey websites to share packages around. In 2011, we got our testing platforms. Mocha JavaScript test runner, one of the many -- writing unit tests suddenly became easier. And in 2012, the Grunt JavaScript task runner arrived and helped

us automate all of the steps in building a package or a web page. With these projects and those like them, we finally had an ecosystem for building projects in javascript. Everyone was building more and more complex web pages. Design was what we talked about in the web. And web accessibility was getting worse and worse because of it. It was about this time and in this maelstrom that I joined the office of the CTO at a company called Acquia, it was in this maelstrom that I joined the Office of CTO team at Acquia under Dries Buytaert, who founded the Drupal Project way back in 2001, right after the dot com bubble burst. In 2012, Drupal 7 had just launched. We were working on Drupal 8. And someone was needed to lead the accessibility gate, essentially to make sure that

Drupal 8 was gonna be accessible. I volunteered. I had no idea what I was doing. I had never worked in the field of accessibility before, and I had to find something to accelerate our work. And what I found was a project called QuailJS. So up on the screen we have a picture of a stylized quail bird, which was the logo for this project. QuailJS was definitely something of its time in 2012.

Everyone was building continuous integration tools or what we called CI tools. Kevin Miller, at UC Monterey, had ported a testing library over from PHP to JavaScript. Right into this world of JavaScript tooling and frameworks and continuous integration tools, and it was a beautiful idea. I made my first commit to QuailJS on November 17th, 2018, and this was arguably my first commit to an accessibility project. The name of that commit or the title of that commit was: Added grunt convert to the build. Converted a guideline... Converted the guideline YML to a JSON file in the dist directory. This is a perfect 2013 title for a commit. On the screen we have a picture of the GitHub repo for equal JS. Running down the left hand rail are a bunch of JavaScript files that

list out test names that exist in this project. It's illustrative of the amount of testing that went into this. I spent my nights and weekends on this project. I had a full-time job, where I had to get things done, and I was also working on this Open Source project. I fell in love with it. We took the premise of WCAG 2.0, that the guidelines are testable, we took this seriously, and we tried to automate that testing. We had success criteria that defined how a guideline should be tested. We should be able to turn them into code and test that on a web page. So we wrote tests for all of the guidelines. For 1.4.3 contrast minimum and 1.4.6 contrast enhanced. And everything in between. There was a group of us volunteers. Working on this project. Working around the

world. Mostly based out of the Netherlands. We modeled our WCAG in the code itself. So we took the guidelines and turned them into JavaScript. So we referred to the pieces of the guideline in our testing code. So that when we evaluated and came up with our violations, we could reference the parts of the code that those were for. Wilco Fiers spun out the AutoWCAG W3C group, which attempted to formalize the definition of an issue that you would find in an automated testing system.

So that we could have interoperability between the violations that we found. And he went on to work for a company called Deque that built up an automated testing service that they run today. So I'm sure many of you have worked with Deque in this room. What we found in 2013 is that you can automate about 33% of the guidelines. You can flag a further 33% of the guidelines for a human to go look at further. And the rest of it -- you know, 34%-ish -- really needs a human to come in and just make a call. We also discovered that the work of building an automated testing platform against WCAG 2.0 is about 10% coding up the tests, and about 90% dealing with web scraping, collecting your violations, categorizing them, collating them, turning them into tasks. Directing them

towards an engineer. Getting them to fix the task. And verifying that your task is fixed. In essence, we were trying to build a company inside of an Open Source project, and it just wasn't working.

QuailJS was ultimately a failure. It ended like so many Open Source projects. Slowly and then all at once. My journey with Drupal was coming to an end as well around this time. But I wasn't about to give up on automated accessibility testing. What's going on in the environment at this point? Around 2014, on screen, we've got a picture of a race car, next to a horse and buggy, evocative of the change that is about to happen to the frontend web development world. Just as it was becoming clear that QuailJS would not work, the frontend ecosystem was flipped on its head. React had arrived. In May of 2013, React was Open Sourced. In June of 2013, the eslint project was released. React introduced the idea of representing HTML in JavaScript with

JSX. Java syntax extension. JSX produces React elements that then render out to the DOM. So now instead of having your HTML file, CSS file, JavaScript file, that get pulled together by your grunt task manager and packaged up, you're running your friend JavaScript code in the file itself.

And this is a big deal. Eslint and other linting tools that were being developed around this time gave us the means of transposing JavaScript, React, JSX into what's called an abstract syntax tree or representation of that code, that we can then go and traverse. Run Bools against, and evaluate arbitrarily. These two technologies changed the game in the frontend world. Then, in 2014, ARIA 1.0 was released. It stands for accessible rich internet applications. This was a different kind of spec. The web accessibility initiative's accessible rich internet application or WAI-ARIA guidelines gave us the semantics or UI patterns. We didn't have this before, formalized in the spec. And crucially, it gave us the patterns for things that existed on the web. Like buttons and input forms and forms... And things that didn't exist. Like sliders and

switches. And tool tips and things that we were building in our web applications, but we had no way of expressing to an assistive technology, to say: This is what the UI is. I need you to express it in a very particular way. And React gave us ways to easily declare our UI in its code. To represent the semantics, the states, and the properties that were defined in the ARIA spec.

In the span of a couple years, we had everything we needed to assert the correct shape of our UIs, according to the guidelines of WAI-ARIA. On screen is a picture of me in 2015, standing in front of the Facebook thumbs up logo. It was May of 2015. I'll pull back a little bit and say that in January of 2015, I got to attend the React Conf, the second one that they had put together, on the campus of Facebook. And it was this conference that convinced me that I really wanted to work in this company. Fast-forward ahead, five months later. Standing in front of Facebook, where I'm now working, I was a very young, still had brown in my hair... Accessibility engineer. I had sort of invented this role for myself. I really wanted this to be the job that I worked in. And

I had a mandate to figure out how we were going to automate or scale in 2015 terms accessibility testing for the web and crucially for mobile apps. A month later, Matthew King joined our team as a subject matter specialist. Where had Matthew King just spent almost three decades of his career before joining Facebook? A little company called IBM. Who is Matthew King's mentor, working over at IBM? None other than Jim Thatcher. And what was Matthew King working on, when he joined our little company, Facebook? He was working on the next version of ARIA. ARIA 1.1. And he was sitting right next to me at a desk. So here we have this confluence of time. These two specialists come together to work on this idea. I spent my first 1.5 years at Facebook pursuing several failed

projects. In fact, I tried to rebuild the concept of QuailJS inside of our testing platforms at Facebook. All the connections that I'm making here right now were not obvious to me in 2015. They become obvious later. In the retelling, it feels linear. But in the living, it was definitely chaos. I tried to update our WebDriver end to end, continuous integration system. This was Selenium WebDriver. It was driving our end to end tests on web. We also had other drivers that were driving our mobile platforms as well. IOS and Android. We were trying to get in there. And run our accessibility checks inside of preexisting end to end tests. So that as this Selenium was clicking through, we would check to see if the button that was just clicked on had, for instance, a label.

Or had... Keyboard accessibility inside. And this was technically possible. We were able to do it.

But what is the really difficult part of trying to work with end to end tests? You have to convince engineers to write them. If end to end tests are a crucial point in your strategy for what you're doing, I would advise you to think of a different strategy. End to end tests are probably the least written tests. They have their time and their place. But if you're looking for wide coverage, you're just not gonna get it with this method. And I didn't realize that. But we pushed forward and I eventually came to the same conclusion that I came to with QuailJS. I wasn't going to write tests, get the output, get them into tasks, get them to engineers at any sort of scale that was gonna change the problem. During these months, Matt and I were spending late nights in the office. We were

talking about UI patterns. How to represent them. Test for them. My struggles and insights informed his thinking on ARIA. His struggles and insights on ARIA were informing the way I was thinking about UI testing on the other side. It was about the winter of 2016. I was on a break. We have a picture of a young woman sitting at her desk. It's snowing both inside and outside somehow.

But it evokes a winter theme. And over the holiday break of 2016, I wondered if I could write a lint rule that could do something very simple. Just raise an error when it found a button that didn't have a label. So I spent my holiday break writing this lint rule, learning how to write rules for eslint. And it turns out... Yes, you can do this! You can find buttons in your codebase for the web and then look to see if that button declares that it has text content. And then I asked myself: Well... Could I do this for every button in the codebase? Yeah, definitely. You could run this and figure out how many buttons don't have labels in your codebase. Could I write rules to

check for other things? Like if an element has a click handler, can I check to see if it has a key event handler? Yes, you can definitely do that. You can raise a flag. If it has an ARIA role, can I check it against a list of existing ARIA roles? Yes, you can do that. Add another rule. If it is an interactive element, does it support focus? This one is tricky. It's very hard to get a very positive signal on this. But I felt like I had something. A way forward. I searched GitHub to see if anyone else had had this idea before me. And of course, someone had. It turns out that Ethan Cohen had started a project called the eslint plugin jsx a11y project. He had started this

just a few months before I had had this insight, so the beginning of 2016. And a couple days later, on December 21st, 2016, I landed my first commit to this project, with the title: Added a paragraph to onclick-has-role.md, to explain use of role presentation. Very simple commit to the documentation. Just getting my feet wet on this project. That was bug number 129. So up on the screen, we have on the left hand side of the screen... A bunch of oranges. On the right hand, a bunch of apples. We're about to compare apples to oranges. With Quail, we tried to automate WCAG 2.0. WCAG 2.0 covers perceivability, operability, understandability and robustness. These are very different fields of concern: And even if you can write the rules, we established that getting

those violations sent out to engineers is hard to overcome. You have to build a whole ecosystem in your testing system to do it. With this linting plugin, eslint-plugin-jsx-a11y, we targeted just the ARIA spec. It's a very small spec. It defines a few dozen patterns for web elements, and using eslint and jsx elements in React, we can validate those details against the spec.

In code. So, for example... If an element has a role of button, does it also have a label? If an element has a click handler, does it also have an interactive role? We're simply asserting that the code we're finding in JavaScript, in JSX, matches the patterns that are defined in the spec. And the added benefit of lint rules is that the violations are raised in context. As you're working, the lint rule pops up. Or you run a script, as you've been working, and you get a list of violations that you've created in the course of your work. There's an immediacy to the violation and the developer and that can get resolved really quickly. So we essentially eliminated whole requirement to build an ecosystem around this project. We had a nice little testing system that we could run. In

place. All right. So let's put it all together. On the screen we have a picture of a beautiful cake with all the ingredients that go into that cake. In 2017, I started contributing to this plugin in earnest. I met with the creator of the plugin on Market Street, down in downtown San Francisco. We had a nice chat about what I was planning to do. And what I was planning to do was a little unheard of in the web world. I wanted to refactor this plugin from the ground up. Specifically wanted to take all of the content and knowledge of the spec of ARIA out of the plugin and put it in its own library. We also had this added problem that we had to represent the semantics that are inherent

in the web browser. When you put a button on a web page or an input on your web page, that element has a file behind it. A class that represents the header of that element. If you go and look in the guts of Chrome, you'll find a directory that lists all the elements, the classes for the elements, to describe the behavior of those. It also lists the states and properties that are supported, and we needed to understand what the mapping of those to the ARIA spec would be. So we had a second library, the AX object query library. So I proposed the ARIA query library and the AX object query library. And these were going to contain the information about the spec and the browser that our lint plugin would then refer to. The lint plugin would just contain the rules. So

we could separate these concerns, and develop them in parallel. I set about this task, it took about 4 months to do this refactor. On January 23rd, 2017, the ARIA query version 0.1.0 was released, and then three months later, in April, I released the AX object query version 0.1.0. We were then able to write our rules faster. We went from a half a dozen rules in the plugin to many dozens, and we were able to test things like... Are you trying to convert an interactive element to a non-interactive element? Are you trying to make a button into a div? Are you trying to make an input or a form into a list item? We wanted to disallow these kinds of transformations in HTML, because it made the semantics really difficult to express to an assistive technology. And because

we were able to reference these two libraries, we had what we needed to essentially list out all of the interactive elements or the non-interactive elements, like the main element or the main role or the list elements, which are non-interactive, like links and buttons. These rules were running in realtime. Hundreds of developers at Facebook were using them. I was getting realtime feedback on whether or not they were working. I was able to count how many violations we would find at various points. And to iterate on them fast, which was amazing. When you have thousands of developers and tens of thousands of JavaScript files, React files, to look at, this was the company that released React to the world -- we were able to develop this plugin at high speed. All right. It's

at this point where I felt like I finally had what I had set out to do about 8 years prior, which was to develop an automated testing something for accessibility. And it was working quite well. How well, in fact? Well, let's look at some data. So... On the screen, we have a graph. The text is unreadable, but that's not the important part. The important part is the vibe of the graph. We have lines going up and to the right, in a vaguely exponential curve. What is this? Well, curiosity took hold of me in 2017. I had this plugin. I had these rules. I thought... Can I write a script that's going to check out the Facebook codebase? A month prior to the current date? Run them, collect all the violations, put them in a table, go back another month, check out the codebase,

run the rules... It took three days for the script to run. Because the checkouts took quite a long time. And I let it loose on a Friday and came back on a Monday, and I had a table of data. That ran a few dozen rules that we had written. And what we found is that over time violations of our rules increased in a vaguely exponential fashion. I have no data to support this. But my gut tells me that for every engineer you add to your team in any company, you're probably slightly exponentially increasing your accessibility violations, as they work on your platform. Maybe someday I'll substantiate that. But that is what my gut is telling me. And we saw that this trend was going to continue, if we didn't intervene. What we did was intervene. So

the drops in the graph here are where we took our rules that we had written and flipped them around. We turned them into code that went and found the violations and updated the code. And we were able to eliminate hundreds of violations with scripts that wrote code. That went and, for instance, found... Icons. We mapped icons to specific labels. Like an X button to the close label.

And we were able to go and add labels to all of our X buttons, with one pull request. Hundreds of those at a time. Those mappings had to be tweaked. Sometimes you want exit for that label. Sometimes you want close. But having some label was better at that point than having none. We were on our way to fixing these problems. That is pretty much the end of this journey. We took this code, and we did a little bit more with it. We pulled it into the browser. We had it running in realtime. So we could raise these violations for engineers and expose more of the spec. The difference being...

When you run a lint rule over static files, you're running them over components that might be broken apart in specific ways. So you might not be able to say that a component doesn't have a label, because its content is rendered by a function, and that function might return a label, or it might not. So you just have to kind of bail at that point. But if you can pull those components together and render them into a browser, at that point, you can make some pretty strong declarations about whether or not a component matches the WAI-ARIA spec or not. And that has been running inside of our codebase for quite a while, and catching some pretty esoteric bugs, when it comes down to the use of, like, treegrid or other of these bigger WAI-ARIA components that

can only be seen essentially when you've pulled them together and rendered them. So I would like to conclude there. On this chat, and open the room to questions. Shall we do questions?

We sure do. JESSE: Fantastic.

Thank you, Jesse! All right. Let's get right into it. Ben? Your exponential chart...

Is the... Is it the case that you had growing size of engineering teams, many of whom were not running linters in the IDEs? So you see that increase because they just were introducing bugs that they didn't have awareness of, and you would be able to go find them retroactively and address them? Is that... Or is there a different variable that kind of contributed to that trajectory?

JESSE: So when I wrote that script that produced that graph in 2017, I think I got back to 2011. When some of the first JavaScript commits were made to the codebase. And that's why it starts at zero. Because we had, from the first file onward... And I truly think that that graph represents just a common trend in any large app. As you're adding engineers, as you're building features, the issues in accessibility are kind of multiplying and feeding off of each other. The patterns get copied and pasted by more engineers you hire, and eventually it accelerates. So to intervene with something at developer time is just one layer in your system that's gonna help prevent some of the issues from getting out. Yeah.

Adrian? Cool. Hey, there. So the chart shows me some nice graphs, and I appreciate that. I'm curious... How much that represents a count of errors, versus density of errors. Because I know those two things can diverge in exciting and weird ways. JESSE: Can you tell me more what you mean by "density"?

So you might identify that there are ten errors on a page. But if that page stays at ten errors, but then gets another 10,000 lines of code, your density goes way down. You have fewer errors per page. But if you're only looking at the count, maybe you don't see progress. Whereas if you reduce the page code dramatically and you still have ten errors, now you've got a much more error-ridden page by weight.

JESSE: Do you mean web page? Or do you mean like JavaScript file?

I'm gonna say web page. Because I'm thinking the whole context... And specifically ARIA errors, for example. Or HTML errors, et cetera. JESSE: Yeah. So this is a crucial difference.

What we found in the early 2010s is that trying to write rules for a web page becomes very difficult. If you're doing it against a spec, like WCAG 2.0, the ecosystem of running that code in the browser and then collecting the errors and pulling them out... I found to be too overwhelming. A problem we couldn't overcome. So we're not testing the page in this case. We're testing a JavaScript file. So you have a file that renders your button, or your webapp. But in the case of a very, very large webapp, you probably have 50 or 60 different button files that are getting rendered. And it's not just the definition of that file. It's where it's used. So if you have 50 or 60 different buttons, and they're getting implemented, each in a hundred different places,

well, now you have 5,000 or 6,000 instances, where you have to check to see: Did the implementer provide a label in that button? And that honestly is the simplest case. We pretty much always start with buttons when we're testing something new. Can we get buttons to work? Because I think we did a histogram of, like, UI across our app, and found that it is a perfect Pareto curve that starts at button and goes to link and input. Right? And if you do that, essentially what we're looking at is across a field of tens of thousands of files. How many times we find this particular error. So I've never looked at it in terms of density. But I wouldn't know what the categorization would be to determine, like, the volume.

I appreciate you qualifying that. Because I agree. And I didn't even think of that, when I asked the question. That you're measuring components, which makes sense. Density isn't as much of a concern there. So in that case, thank you for humoring me. JESSE: Yeah, yeah, definitely. Yeah.

First, fantastic storytelling. That was really nice. That was a really nice way to explain the history of it all. I was an Angular dev. So we didn't have those tools.

JESSE: 2012! Angular!

Yes. But the question I have is... A lot of the work that you have done worked for static analysis. In parallel to static analysis? Or... As a secondary add-on to static analysis. That's question one. Question two: Did the introduction of all these linting tools reduce the amount of testing that needed to be run by quality assurance and by developers with tools and other browser extensions and what have you?

JESSE: Yeah, yeah. Let's start with the second question and then I'll ask you to repeat the first one. Because it's already popped out of my mind. These tools, I think, make... Well, first of all... We still have humans testing our products. I don't think you could ever eliminate that step. The work that testers do to work through a flow to try out different methods of interaction, different assistive technologies, is invaluable. It has to happen. I think what these tools do is they eliminate the really easy stuff from the testing, so that you don't get, like, 500 tasks back from your human testers. That's just like... Add a label to this element. So if you can eliminate classes of problems like that, what you're gonna get back from your human testers is higher quality. It's gonna be things that look at the ease of use and

usability and not the mechanics. What we're trying to solve here, by looking at WAI-ARIA, is... Guaranteeing the contract between the web page or the iOS app or Android app, the physical artifact, and the assistive technology that's going to be interpreting it. The contract between those two is WAI-ARIA. And if we can shape our code, our output, to conform to the contract, then those clients can use it. So that's the first part. What was the...

That's the second part! What's the first part?

The first part was between the... Because I used to work in Angular, and we didn't have these tools -- how much is the eslint tools in parallel or an extended companion to static analysis tools for accessibility?

JESSE: I think the eslint tools, if you add them to your workflow, are a very good way to eliminate the simple, quick problems. Like... I'm building a component fast. And there are two dozen things I need to remember in building this component. I need to think about focus. And keyboard interoperability, and making sure that the semantics are right. What are the three dozen roles that I can use from ARIA? Oh, I typed too many. It's not correct. It makes that contract easy to reference in the moment. But what it doesn't do is get at the... If we go back to perceivability, understandability, operability, and robustness, we're operating at the perceivability and the operability stage of this. Not at the understandability. And so much of the WCAG specification deals with understandability. All of the three dot stuff in there... To do that,

I do think you need automated systems that you can understand. And in 2012, we didn't have those. We just had rules that we had to write, and they got super complex. And we couldn't deal with semantics. We might have those tools pretty soon. There are systems for understanding that are coming online in terms of AI that could make it possible to automate testing something like WCAG.

I'm looking at the people in this room to maybe carry the torch on that stuff. Or out there. I could imagine in four or five years, rebooting the automated testing of WCAG and having the tools to finally do it. Five hands up at once.

It's not a question, per se, but just as a parallel... We built a component library in Angular, because we had a lot of these issues. And we individually rated these components as AA compliant, and then we handed them off to be built. But because we had done the components, the only thing we needed to do was the quality assurance work needed to check for usability. With the whole page.

JESSE: That's great.

So since we didn't have the tools, we had go through this longer way. But we kind of achieved similar results.

JESSE: Yeah.

I'll pass it -- hi, Jesse. I'm also curious about the follow-up from this graph. Which is... You implemented code that would basically generate these code mod diffs to fix the errors. But I'm wondering if over time the existence of the eslint rules actually cut down on people shipping these problems in the first place. Because... Over time, as the codebase continues to grow, these code mod diffs become more difficult to actually land. They get enormous. There's conflicts, et cetera. And I would like to know a little bit about whether you saw a trend that way, downward, just for the existence of the rules in the codebase, as devs work, and whether building out the initial Facebook design system actually was a way to remove that from most of the devs, pushing that into the building of the design system

and how you saw these trends go from there. JESSE: I think the best model to think about this is the Swiss cheese model. Where, when you stack a bunch of Swiss cheese, one layer on top of another, the holes probably won't align perfectly. Like... It's unlikely you're gonna get one hole that goes through the entire stack, straight through. So we have different systems in place that are catching different types of problems. You'll notice on this graph, we've got the orange line that goes up. Code mod. Line continues to go up. So in this case... We eliminated a bunch of problems. But developers continued to add them. Another case... We have a blue line that goes up.

Code mod. And then trend goes down. So there are certain types of problems that I think once we eliminated some of the broken windows, people started to copy and paste the right code. And now we're getting that good pattern. The good bacteria in the system is now winning over the bad bacteria. So it depends. But this wouldn't be the only layer in your system. At the very end, we have humans who are doing our evaluations. At dev time, we have linting rules. At render time, we built a system called Redblock that essentially puts a giant red box over the UI if you violated WAI-ARIA and doesn't let you interact with it. This is highly interruptive, friction filled, people post in the group all the time: How do I make this go away? And the answer is: Fix it. So

we have very good conversations when that happens with our web developers, and we teach them about ARIA. We go in. We help them get past the error. I just had one the other day, where someone had added... ARIA role description. To an element that didn't have a role. And that's a violation. It's something that would be very hard to catch. And we were able to go in and eliminate that property and move on. So we're catching super esoteric stuff as it comes on. That's another layer in the system.

To answer your question, the component library is yet another layer. We bake in a lot of really good focus management. We've updated and fixed focus management in Android, so that our libraries can utilize that in our frameworks. We made the label property on our web button required. So you cannot add a button to the page without adding text to the label. That took months and months to fix, because we had to go and fix all the existing errors before we could ship it. But those things in your component library can be another layer to eliminate all sorts of issues.

One last question. Hi. From a business storytelling perspective, were you able to make a correlation between those graph plummets and a reduction in people hours and thus dollars?

JESSE: Thus dollars? We didn't have that story. But this was the first time we quantified the growth of accessibility errors. And I was able to take this to my director and say... These graphs are going up. If we don't invest in arresting this trend, we're going to have that much more to fix. In that many more months. Right? So... Five or six months later, the problem has doubled. And that was enough for me to win the time to go and spend the next six months working on this Open Source project, to get it out. Yeah. So... It wasn't like... Company changing. But it was life changing for me. After a year and a half of failed projects, you know, my trajectory started going up too. Yeah.

I failed to take any livestream questions. But... Let's do one at least. Here, Thomas. I'll let you take one of the questions.

THOMAS: We have one from our livestream. Again, thank you, Jesse. So the question here is: Any recommended resources when writing custom accessibility for prebuilt components? JESSE: Accessibility for prebuilt components? You know... There are so many libraries out there that do this so well. I'm thinking of the lightning library from SalesForce -- it's a really good reference to go to. Looking at the docs for iOS and Android, some really good information. And honestly, the way ARIA spec and the implemented guidelines that are part of that ecosystem provide examples for each of the roles with code examples. They outline all of the different variations of states and properties. How to implement stuff. Even for some of the trickier things. So... Internally, we point people at the WAI-ARIA documentation when they're building these sorts of components,

and we've contributed at Facebook extensively to those documents as we were learning how to build the components. So we represent sort of the best that we know how to build of those components.

Great. Thank you so much. Let's give a round of applause to Jesse.