Advancing Accessibility: Strategies for Continuous Improvement

Hi. I'm Andrew. Tonight we're gonna be talking about the intersection of accessibility maturity models and continuous accessibility. My slides are up on noti.st.

So if you find it easier to follow along that way, it's noti.st/segdeha. Which is kind of my handle all over the socials. It's actually my first initial and last name reversed. Might make it make more sense.

So a little bit about me. I've been building the web professionally since 1998. Which coincidentally was the year that section 508 was passed and one year before version 1.0 of the world wide web consortium's Web Content Accessibility Guidelines were published. I first learned about section 508 soon after moving to Washington, DC, at the end of '98, and it really struck a chord with me.

Kind of by reinforcing my idealism around kind of the early web. Which was for a lot of us... We all very kind of earnestly bought into that being a force that was gonna democratize access to information.

Fast-forward kind of two and a half decades, and I'm now co-founder at Assistiv Labs, and after kind of an eventful career as a developer and engineering leader at a bunch of companies, including Disney, Apple, and Zapier. Working at Zapier is where I fell in love with the power of automation. It's also where I contributed to an accessibility guild. We were doing kind of grassroots work to improve the accessibility of Zapier's core product.

And it's there that I met Weston Thayer, the other founder of Assistiv Labs. So I joined full-time in the summer of 2022 to work aside Weston and the team to fill in what we saw as gaps in accessibility -- in the tooling available for making the web more accessible. So in terms of what we're gonna cover tonight, my agenda is pretty simple. We're gonna talk about accessibility maturity models, we're gonna talk about the concept of continuous accessibility, and then we're gonna talk about how continuous accessibility can contribute to accessibility maturity.

In a larger sense, my agenda really centers on advocating for the application of automation to make accessibility testing more efficient. So as I mentioned before, I'm a big believer in automation. From my time at Zapier, which is a company that has the mission of sort of automating all the things. When I was there, I saw Zapier help small and medium sized businesses really punch above their weight.

[audio drop] at the same time... I'm a firm believer that the gold standard for how we [audio drop] humans is to have humans validate the system. That starts with the actual users of the software, including and especially disabled people. But in the realm of digital accessibility, that should almost always include skilled auditors as well.

That said, it's not enough to get an audit, no matter how thorough or high quality, on a yearly or even quarterly basis, and rely on that alone to validate whether your product is accessible. So I believe both of the following are true. So, I believe one line of JavaScript will not fix 100% of your product's accessibility problems within 24 hours. I think we can all agree on that.

And there's a ton of work that we can offload to the machines to enable us to be more efficient and make better use of our... The time we do have as humans to make sure that what we're building is as inclusive as possible. All right. So let's first talk about accessibility maturity models.

So I bought and recently started reading Devon Persing's book, the Accessibility Operations Guidebook. Thumbs up in the crowd. Her guide to making accessibility work for sustainable. It's comprehensive.

I highly recommend it for anyone who is trying to take a more systematic approach to accessibility. And in her chapter on organizations, she explains that accessibility maturity models help us do the following. Identify an organization's current stage of maturity, identify gaps between the current stage and the next stage, and then develop a plan to move from the current stage to the next stage. So it's a type of capability maturity model.

There are lots of those that have different specializations. And they all tend to be pretty process-oriented. So I personally think that process gets kind of a bad rap. So done poorly, it can inhibit innovation by institutionalizing maybe restrictive practices or something.

But done well, process is what can increase the repeatability of behaviors that lead to better outcomes. So an example of this... In the late 20th Century, that's fun to say... I managed university residence halls.

So each year, we would hire dozens, sometimes hundreds of undergraduates to serve as resident advisors. So these folks tended to come in, they were bright and well-intentioned, but they had widely varying levels of skill around the kinds of things they needed to be able to do to be effective. So process is one of the tools that we used to kind of ensure a solid baseline of outcomes. So, for example, if there's a roommate dispute or something, and the RA for the students followed our processes, we were pretty well...

We were pretty confident that we were gonna get to a decent outcome. Even if they mostly followed our processes, it was gonna be better than if they just winged it. Right? Which would be just a little too unpredictable.

So I'm not super rigid about it. But I think solid processes are a great starting point.

So, Let's talk about some examples of some of the prominent accessibility maturity models, so AbilityNet in the UK, they have a Digital Accessibility Maturity Model, DAMM. This is the one I have the most experience with. We were using this at Zapier to do a self-assessment while I was there. Level Access has a DAMM as well.

Microsoft came up with what they call their accessibility evolution model, and the W3C has a Working Group that is pretty far along in developing what's called a group note. It's not a specification, exactly. But an accessibility maturity model.

The models all tend to be structured pretty similarly. For the most part, they define dimensions, for which the current stage is assessed by collecting proof points as evidence. So that's kind of how these things work.

There are several models devoted to accessibility. They all have pluses and minuses, unsurprisingly. And we're gonna focus on the W3C's model for the purpose of this talk. So the W3C's model actually outlines seven dimensions.

Communications, knowledge and skills, support, ICT, which stands for information and communicate technology, development life cycle... So that's a mouthful. And it's probably where we're gonna spend all of our time. Personnel, procurement, and culture.

So you see it covers a lot more than just technology. And this is true of all the models. They go into sort of organizational culture and all these things. But ... With these models, they're trying to take a holistic approach to how you assess the maturity of the organization.

So in my analysis, kind of looking at the model, the proof points related to continuous accessibility, which I'll talk about next. All happen to fall under the information and communication technology development life cycle dimension, which I'm gonna be calling ICT from now on. Because it's too many words to say. So the W3C model defines -- this is a lot of text.

Sorry about that. The W3C model defines the following stages of maturity. Inactive, launch, integrate, and optimize.

Across the various dimensions, the organization hopefully will progress through these four stages. So, Something to note is that according to people much more expert than I am in this kind of thing, you can't skip ahead. You have to go from inactive to launch to integrate to optimize. In that order.

Each stage sort of sets the table for the next one. So let's just run through an example to kind of give you a sense of what we're... A more concrete example of what we're talking about. Under the ICT dimension -- the W3C model includes a category called user research.

Emily, you'll be happy to hear that. For which one of the proof points is that user research includes disabilities. And that's how they word it. So if your organization doesn't consider disability at all in their user research, that's unfortunate.

But no judgment. We have to start somewhere. In that case, you would rate this proof point as no activity. If you've made it as far as making plans to do this in the future, you might rate this proof point as started.

If you usually but inconsistently consider some subset of disability in your research, for example, maybe screen reader users only, you might rate this one as partially implemented. And then if you're consistently considering a broad range of disabilities, in your user research, you can go ahead and mark this proof point as complete. So kind of four levels. You can also mark them not applicable.

But we're not gonna talk about ones that aren't applicable. So the point of all this is to give us a framework we can use to better understand whether we're getting better or worse at accessibility. Okay. So that's the models.

Let's shift gears and talk about continuous accessibility.

Continuous accessibility is an approach that's widely advocated for by Melanie Sumner and others for making the most of automation as a way to accomplish the following. So having greater confidence in the quality of our code, more easily delivering accessible experiences at scale, and responsibly reducing our organization's risk around accessibility. So one of the ways that Melanie says we can achieve these outcomes is by having the computers do the repetitive tasks, so that humans can solve the hard problems.

And that's very in line with Zapier and other automation companies' philosophies. So before we continue, I'm gonna go on a little side quest and talk about the technologies that enable all of this continuousness.

You may have seen these terms. I don't know how technical the audience -- everyone in the audience is. So I'm just gonna spend a minute on continuous integration/continuous deployment. Usually referred to as CI/CD.

These are techniques that have accelerated modern software development by enabling the automation of processes that used to be manual. So back in the bad old days, like 15 years ago, we used to push changes to a staging server, manually QA them, and then FTP the files to a production server. And I'm not gonna lie. It was terrifying at times.

Continuous integration, CI, refers to systems that bundle software into a known working version. And on the web, continuous deployment is what updates your servers to use the new version of the software. So again, CI creates the new version of your software. CD deploys it into production.

The Cs in CI and CD are there to indicate that these are automated processes. In other words, there's software that's doing the work of bundling up the new version of the application and deploying it to a production environment. So why is this so important? CI/CD enables a bunch of things, including the use of packages as a way to isolate system components, and high confidence deployment to production, specifically because the process is automated, and not relying on a systems administrator to remember the exact sequence of updating the application to persistence layers of the site or whatever -- the way we used to do things.

And yes, I have in fact personally seen things go very sideways multiple times in my career, from doing things in the old kind of manual way. CI/CD is also what enables us to run automated tests against our software prior to deployment. And this is where it starts to connect with what we're talking about tonight. The earliest of these tests really were intended to ensure the correct inputs and outputs for a system.

But eventually there was tooling that appeared that could test the UI as well. The problem was that the vast majority of the tooling was built in a culture dominated by ableism. So those tools didn't -- and they still don't in some cases -- take disability into account. So EJ Mason wrote an amazing article on the Assistiv Labs website about how ableism becomes the default for our work unless we actively sort of swim against the tide of the dominant culture.

So in that article, EJ argues that our tools in many cases reinforce ableism itself. But how, exactly? So examples include... Assuming all users are using a mouse.

So there's lots of libraries out there that are mouse-based. There is alSo, there are libraries out there that simulate keyboard events. But that's not the same as faithfully emulating user input. So that's just a couple of examples.

So, Thankfully... Automation friendly tools, specifically designed to address accessibility, started appearing over the last few years, including Axe-core by Deque, Site Scanner by Evinced, and others.

In a recent article reminding us why we can't just have nice things Bogdan Cerovac -- not sure if I'm pronouncing that right -- cautions us about using the tools used to test accessibility. He says that automatic accessibility testing can detect inaccessibility, but can't detect accessibility.

I agree with that. I like the fact that he used the word automatic and not automated. Because there is such a thing as automated tools that can do things like take into account design intent. Which the automatic tools can't do.

The automatic tools are great for a lot of things. And as an industry, we put a lot of faith into automatic tools, including scanners and linters, that apply rulesets to websites to report on accessibility errors. And it's helpful to an extent. A large extent.

But we need to understand what the tools can and can't do.

The primary thing scanners and linters can't do that humans can is understand the intent behind a design. A specific UI flow, what's the path through it for an assistive technology user to accomplish the task. And irrespective of any WCAG failures, how good is the experience of attempting to accomplish the task, for example, using a screen reader or voice control? The automatic tools aren't really able to tell us that.

So, another category of automatable... Which isn't actually a word, my spellchecker tells me... But another category of automatable tests is end to end, or sometimes called E2E tests. Those validate flows through a user interface.

So, for example, logging in, creating a project, creating a task, that kind of thing. This is a way of encoding human judgment about the design into a test that then can be automated. Okay? So, Unfortunately, most end to end testing frameworks suffer from the ableist traits that I mentioned earlier.

So, Some of the end to end testing frameworks suffer from these same ableist kind of traits, and offer only limited support for accessibility out of the box. I'm gonna refrain here from making a full pitch, except to say that at Assistiv Labs, where I work, we've created an end to end testing that treats accessibility as a first class concern. So I know it can be done.

Before I go on and continue, I want to acknowledge that manual QA and periodic audits are super valuable tools also. Essential, really, to ensuring accessibility. Not just the not accessible accessibility of our digital products. But they're not what we're talking about tonight.

We're really talking tonight about what we can responsibly offload to the machines. All right. So what makes continuous accessibility so compelling? And it's something that I call attention decay.

So, Attention decay is the idea that the sooner after it's made that an engineer finds out that their change broke something, the more likely they are to fix it. So, Say you're a developer. You find out something that you shipped six months ago is broken. You've moved on.

Users have been living with the bug for months. At best, you're probably gonna add it to the buck tracker and maybe ship a fix at some point in the future when it gets scheduled. But imagine you find out you shipped a bug earlier that day. Right?

The context of the change is still fresh in your mind. You have the opportunity to fix the bug before it affects very many people. So you're likely to jump on it and ship a fix right away. And this dynamic actually applies to any type of bug.

But it helps explain why tests in CI/CD are so popular and so valuable. Just in general. Because it means catching problems before they impact your users. All right?

So let's talk next about the types of automatable tools available. So in the automatic category, we have linters, such as Axe Accessibility Linter by Deque, scanners including Evinced and WAVE Testing Engine. Flow based tests with Axe integrated. I know Deque has a tool like that, Cypress has a similar tool.

And there are automated tools -- like I mentioned, accessibility first end to end tests, like we're trying to do at Assistiv Labs. So both automatic and automated tools are using and should probably be part of your process. But it's important to understand the differences in how they apply to your use case. All right.

Okay. This is where we get down to... That should be a multiplication sign, not a plus sign.

Caveats aside, automated solutions for validating the accessibility of digital products can contribute significantly to improving our accessibility maturity. So earlier I mentioned Devon Persing's Accessibility Operations Guidebook On the website for Devon's book, which is linked in the slides, she links to a handy spreadsheet for calculating scores for the different dimensions of the W3C accessibility maturity model. So we're gonna use that spreadsheet as a way to explore the difference that automated tools can make.

And again, we're using the W3C's model not because it's objectively the best in some way, but because... By virtue of originating from a standards body, it's the one that's least proprietary. That was the criterion. As I covered earlier, that model defines 7 dimensions, dozens of individual proof points.

We're gonna narrow it down to what's relevant to continuous accessibility.

All right. This is another kind of wall of text. I apologize for that. We'll go through them individually, though.

So relevant proof points for our purposes fall under the ICT dimension and include the following. In the development category, developer's accessibility checklist, consistent approach to implementing accessibility features across products. In the quality review through release category, consistent approach to testing and releasing products. Testing process includes automated accessibility testing.

That seems pretty applicable. And accessibility identified as product release gate. So next we're gonna talk about how continuous accessibility applies to each of those proof points. And before I go on, I just want to kind of be transparent about my methodology.

So the first thing I did was I made a copy of Devon's spreadsheet, I then duplicated the ICT tab, and set all the proof points back to no activity. So Devon's spreadsheet actually was... She had it all set up for kind of a past project that she had done, just as an illustration. I set everything back to no activity.

So we could see... Start from zero and see what difference -- and isolate the effect of continuous accessibility. Then I updated the evidence dropdown for each of the relevant proof points for each stage. By doing my best to log the minimum state that must have been true if continuous accessibility is applied.

So I was trying to be very conservative about my estimates of what the actual impact was. So I'll start with development. Specifically the proof point developer's accessibility checklists. So, I hope this isn't too much of a stretch.

But you can think of linters as a kind of checklist of heuristics that are automatically applied to code within the context of an integrated development environment, IDE, such as VSCode. If you're not a coder, with a linter installed, a developer will see warnings right there, where they're coding, about certain types of accessibility issues. So ideally these same heuristics are enforced in CI/CD. If you're using Axe linter, it makes sense to use it in your CI/CD as well, to make sure you're getting consistent results.

Okay. So, There are other more manual checklists that you can and probably should implement, that cover areas that Axe and similar tools can't. For example, you may want your team to evaluate new UI using Microsoft accessibility insights self-guided audit feature. So you can apply human judgment to things like tab order.

So while linters partially satisfy this proof point, they're not enough by themselves to get us to 100% complete. So when we say partially implemented across all four stages, that's good for eight points. Okay? I'll explain the points in a second.

Next proof point under development is consistent approach to implementing accessibility features across products. So automated tools can validate code before it's released. So they can be used to enforce a consistent baseline of functionality. But maybe this proof point should really involve shifting well left of the development phase into the UX research design.

So continuous accessibility gets us part of the way there but not 100% complete. That's another 8 points. We're up to 16 out of a possible 300 points. Doesn't sound too good just yet.

But it will get better, I promise. Let's move on to quality review through release category. The first proof point here is consistent approach to testing and releasing products. So implementing continuous accessibility tooling should be considered I think essential but not sufficient for meeting this proof point.

Because while automated testing can help us find certain classes of bugs basically in realtime, manually validation by humans is still necessary for areas that the machines can't test, or where they don't give us 100% confidence. So, Another example where continuous accessibility partially satisfies a proof point. So we're up to 24 out of 300. Okay.

There's a slide on the screen of a stack of Swiss cheese slices. This is a little different than all the text-based slides that I've had up to now. So before I continue, I want to go a little deeper on that last proof point. Cam, who introduced me a little earlier, first turned me on to this analogy of layers of testing being like a stack of Swiss cheese slices.

So any one slice would allow... Water? I don't know. Barbecue sauce...

Light... To leak through. But by stacking the slices, you put at least some cheese between the top and bottom of the stack. Does that make sense?

So I bring this up, because... Unlike the other proof points that I'm talking about... Consistent approach to testing and releasing products is not explicitly tied to accessibility. And that's because you need to test for accessibility whether or not the change being made was supposed to be related to accessibility.

And it happens all the time that unrelated changes end up breaking things for accessibility. And that's why continuous accessibility is I think the only realistic way to stay on top of the usability of a complex, rapidly changing application. So any mature software organization is already gonna have other types of testing in their CI/CD. And so adding accessibility-specific testing is kind of like adding another layer of cheese to the stack.

So that's what that slide's all about. All right. Continuing with quality review through release. We have the proof point: Testing process includes automated accessibility testing.

That sounds good. This proof point is worded as a binary. We'll say that it satisfies it completely. But I wouldn't be surprised to see some nuance added to this point in the future.

In future iterations of the model. As the tools get more mature. I think it's not enough just to say you have something in there. There probably needs to be a little more differentiation.

But for now, we'll bank that sweet 12 points, bringing our total to 36 so far. All right. The next proof point, under quality review through release, is accessibility identified as product release gate. So in case that term is unfamiliar to anybody, in software, a product release gate is anything in the process that might detect a problem severe enough to sort of stop the line and prevent the software from going out.

So product release gates can be both manual -- so a QA person finding a bug -- or automated. Now, some of our customers at Assistiv Labs are very sophisticated in how they approach accessibility. And I don't know a single one that would fail a build due to an automated accessibility test not passing. Yet.

I think that's in the future. By applying continuous accessibility to your product development efforts, you kind of set yourself up to be able to implement a true product release gate. At whatever point you determine that the tools are reliable enough to allow you to do so. So, We'll call this one started.

Which is one step up from no activity in the spreadsheet. It's good for another 4 points. Bringing us to... 40/300.

13.3%. So, You know... Makes a difference. It's not enough by itself.

Right? So continuous accessibility alone was enough to get one of the five proof points, sufficient to justify partially completed for three and started for one more. That's 40/300. Now...

We can do better. We can have a little bit more going on here. So I'm sure some of you who have seen that automatic tools like linters and scanners tend to identify a high volume of issues, which is good. It's kind of what you want.

You want it to identify all the problems. Right? When implementing tools that scan for problems, it's a good idea to consider in advance how to integrate those results into your existing prioritization processes. So the following are the related proof points.

So under the development category, documented way to triage and prioritize fixing accessibility issues and address customer reported feedback on accessibility. Under the quality review through release category, prioritization and refinement system for accessibility defects and bug tracking system includes an accessibility category. So, I didn't include these in my base calculation, because remember, my methodology was to include only the proof points that must be true, or must be affected if you implement continuous accessibility.

But I think these are worth talking through. So you could theoretically throw a bunch of tooling in your CI/CD and hope for the best, but I wouldn't recommend it. So while these ones didn't make the initial cut, you'll want to consider these proof points to make sure you don't get sort of swamped by noise in the form of duplicate issues or issues your organization might consider low priority. So let's just say we get to partially implemented across all four stages of the first two.

And completed for that binary one. That third one. That brings us to... A total of 68/300.

So that's a little better. A little better. And then there's a couple other things to watch. So, There are a ton of advancements being made around automation at the moment.

Including shifting automated accessibility checks left into design. So I haven't played with it directly. Which is why I didn't include it in my calculations. But as an example, Figma now supports triggering a type of CI/CD for what they're calling continuous design.

And I'm just like... Trying to connect the dots here... Stark makes a popular Figma plugin for checking accessibility. And that has automatable tools as part of it.

So does that combination get us to at least partially completed for these additional proof points? I'm not actually sure. But if we include all those additional proof points from the last two slides, theoretically you could use automatic checkers to check for certain types of accessibility problems. For that last point -- on things like draft blog posts.

So if we were to hit all of those, then we get to 108/300. 36%. Which is... With a pretty reasonable level of effort...

And I think that's pretty good. You know? That's just... Automating some things.

Get to 36%. So, Let's talk about some conclusions. So accessibility maturity models are sort of the best way we have. They're not a perfect way.

But they're the best way we have to track improvements in our organizations' accessibility programs. Continuous accessibility enables us to deliver accessible experiences at scale, helping us improve our accessibility maturity. But it's all with the caveat that it's not enough just to throw tools at the problem. You have to understand their capabilities and limitations.

And back them up with adjustments to your existing processes. To get the most out of them. And that's what I've got. Thank you so much.

All right. We've got a bit for questions. Anything in the room? Thanks.

You mentioned that you don't feel like any of the QA folks that do this kind of testing would ever pull the cord to stop a deployment for a failed accessibility test. I'm curious what you feel like is standing in the way of that. Even these automated tests... Do we feel like at that point...

If they fail, why is it any different than any end to end test failing that would just stop the deployment and have you fix it before it goes out? Yeah. That's a great question. So, Just to be super clear, a manual QA person -- absolutely -- I think there are lots of organizations where they would be like...

No, we can't ship this thing until we fix this problem. But for the automated tools, I think there's not quite enough trust in the tools yet or something. People are kind of aware of... It's testing these things.

But we really need to manually validate that before we actually stop the line. So I just haven't seen it yet. It may be happening. I mean...

None of our customers, as far as I'm aware, I could be wrong, are doing that yet. Where a test fails. Stop the line. I see Cam has his hand up.

So he may have an opinion or some insight here on this issue. But... Does that make sense? I don't know.

I think the day is coming where the same... I don't know. It's partly probably organizational too, in some cases. Where the company -- they don't value accessibility to the same degree they do just...

Oh, like a user can't log in or something. That's... Okay. Gotta stop the line.

Our end to end test failed on that. But if it's... Oh, people make these terrible arguments about it. Oh, it's only this percentage of users that can't do the thing.

Well, we'll ship it and fix it later. It's like... Well... It's not how it should be done.

But it's how... I've kind of seen it done. I have a theory on this. Which is that most end to end test suites -- CI integration suites -- are authored by engineers, explicitly saying: I expect the software to do this thing.

And when it fails, it comes back to them. And it's very clear, I own this behavior,I implemented it in my test. When you introduce some automation tooling that is not authored by the developer, that might be kind of inscrutable to them, or a little bit foreign... It's really easy to just kind of turn off somehow.

I think there is a place where something like Assistiv Labs -- you're running end to end tests that you've authored that use assistive technology. If that fails, absolutely you should fail the build. But like... There's this psychological bit, I think, that if you haven't written...

Using Axe-core, for example. I didn't author the Axe-core assertions. I might not understand them if I see them failing in CI. That may be a little foreign to me.

And I don't take immediate ownership over it. It's unfortunate, but I think that's true. That's a good question. Thanks.

Question from Carl. There's also a bit of a difference between... Let's say you're writing functional end to end tests and a complete scenario fails. Right?

So, It represents a failure of a task, whereas accessibility tests don't necessarily represent that. And going back to Cameron's point, when you're dealing with a large volume of issues, which are at least on the surface sort of superfluous noise... As opposed to something that rises to the occasion of being a complete failure for a user to perform a task. And so I think that's kind of the big difference that people see.

So at Tenon, we had people -- they would put it in their builds and they would create a dashboard out of the results and sift through the results to determine but would never turn a build red because of the mix between what Cameron said and the whole issue of severity and all that sort of stuff. I'm gonna turn it over to Dylan. I'm gonna change the subject. [laughter] I could go on about that, but you know...

Did I miss something? Is it possible to get to 36% in that maturity model without fixing a single issue? Oh, that's an interesting question. I didn't see anywhere in there a requirement to actually -- Actually remediate anything?.

You got to 36% without actually doing anything? Yeah. It's detection. It's process.

So by authoring a test suite using assistive technology, you're declaring that something works, with that assistive technology. So if you have a failure, you're regressing. So by failing to regress, you're maintaining a level of accessibility. Right?

We're gonna serialize the back and forth. But in his assessment of his score, it got him to 36%. He didn't force the build to not progress. So he's actually getting 36% without committing to fixing anything.

It's just an observation. You know... You're not wrong. People criticize this, that, or the other measure of whatever accessibility you've achieved.

And I'm just pointing that out. But Cameron, are you implying that having the test assumes that it's also making sure that you're fixing the stuff that you find? Right? Yeah?

So if I write a test that says... A mouse-based test... That exercises a login flow, I am explicitly asserting this should work with a mouse. And it does.

Because the test passed. To your point, if you're not failing CI in your end to end tests, you're doing something wrong. But I think this implies spiritually that you would fail those tests. Understandably, for reasons we've talked about, I think the industry is not to a point where...

First of all, we don't have... Historically, we haven't had the tools to do this effectively with end to end tests. And secondly, our imagination as engineering organizations has not caught up to what's possible today. So, Yeah.

So we actually have an article coming out with Asana in a couple of weeks. Where we're gonna talk a lot about how Asana uses our end to end testing framework. Asana is not currently failing builds. Right?

Correct? Okay. So, And Asana is not authoring the tests. So maybe there's a thing there where at some point we're able to open up our library to be more platform as a service and you're writing your own tests, and maybe there's a little bit of a mental switch that goes on for developers, where it's like...

I wrote the test. So if it fails, yeah, we shouldn't ship this thing. Right now it's still a little bit in a gray area, although we can do the right kind of assertions. There still may be a little path to walk there, to get to that.

But you're right. The 36%, there's no hard requirement to actually fix anything. You've got to call Carl for that. Right?

Not in baseball. You make $100 million a year in baseball, 36%. Question from Stephen. In my experience that folks are most interested in and affected by the results of improvements in the maturity model -- have been management and folks not quite in the weeds.

What's your experience? Yeah. I mean, that matches my experience. So my main experience came at Zapier, where we went down the path of building that sort of self-assessment.

I was saying to Dylan before the talk that one of the things that I found valuable about that was just the conversations that it prompted. You know, some people who... In leadership, who clearly hadn't thought too deeply about accessibility were then talking about it. Right?

And they were in a breakout room with people who are kind of on the front lines of trying fix things. And they get at least some level of conversation happening. But yeah. So one of the things that Melanie does in her continuous accessibility work is talk a lot about metrics and what you should be paying attention to.

You know, metrics are the kinds of things that leadership are gonna pay attention to. You can consistently track the number of defects as going down or the time to resolution as getting shorter. Like some of these kinds of things. Then those are numbers that might catch the attention of leadership.

Another question from Crystal Scott. As a web engineer, how can I help clients and organizations maintain accessibility long term? I often see sites regress after remediation. What strategies ensure sustained conformance?

Well, we have a little tool for that.

I think that... You know... Devon Persing actually talks about this in her book.

The way to do this stuff right -- really needs to be across the entire organization. It's not something you can do piecemeal. She talks in her book about... Sort of the friction of all these little ad hoc efforts.

And once you can kind of institutionalize accessibility across the organization, the friction and people paying attention to it... And it not happening automatically, but you know what I mean? It just kind of starts to happen as part of the course of doing business. But it really takes a lot of education, a lot of laying groundwork, a lot of being patient.

She talks a lot about organizational change in her book. And the interesting thing about... The book is great. I just started reading it.

But she talks a lot about how you don't expect to be doing organizational change as an accessibility person. You think you're gonna be fixing a website. Making it so it works with a screen reader. But you end up doing all these things that are maybe outside of your actual skill set.

And that's where a lot of people burn out. Because you're trying to do everything at once. So if you're lucky enough to be in a big enough organization, where you have somebody who can manage the program, and then subject matter experts who are separate from that person, you're in a really good spot. But yeah.

Anybody who hasn't yet, I can also second the recommendation. Devon's book -- I got through it -- first read this weekend and now I'm circling back to kind of go through it in deeper dive. And it's very good. So, Thank you, everybody, for coming.

We appreciate your time in person and appreciate your time online. Thank you to Andrew. Thank you so much everyone.