Real AI Solutions for Accessibility Challenges

Robot at desk typing on computer, WCAG book in other hand, two monitors with one showing an error

This article is based on a talk Kevin Berg, VP of solution engineering at Evinced, gave at A11yNYC .

Artificial intelligence (AI) can help solve the digital accessibility problem. In fact, GitHub did a controlled study on the impact of using an AI tool like GitHub Co-pilot can have on productivity and satisfaction when it comes to developers. In doing the study, they split 100 developers in half. Half used GitHub Co-Pilot to write a web server. The other half didn’t.

The study’s researchers expected AI would help the work move faster. But not to the point they found. The average time with Co-Pilot was 71 minutes. The average time without was 161 minutes. Being able to increase that amount of speed is a huge productivity booster.

If you dig into the study, you’ll find the developers enjoyed using Co-Pilot. That’s because AI easily took care of the more redundant parts, the more boring parts. It left interesting challenges for the developers to solve. In short, AI allows developers to save their brain power and mental energy for the problems they like to solve and remove the tedious parts.

The problem is that there are many different areas that need solving to remove the bias in AI. For example. Someone entered a prompt in ChatGPT to write a Python function to see if someone would be a good scientist. They used a JSON description of their race and gender. The return was biased toward white males, which is a huge problem.

Fortunately, this bias is fixable. Bias is one of the main problems with AI models. It has a significant impact especially where accessibility is concerned.

Another issue is AI not accurately responding to a user’s prompt. For example, someone provides a prompt of “Write a sentence that ends with the letter s.” ChatGPT presents a response that says, “The birds in the trees sing sweetly at dawn.” The sentence uses the letter s quite a bit, but it doesn’t end with the letter s.

People must be careful about their expectations around what kind of responses they’re going to get. In this case, they must consider what the prompt is, and how to communicate with these types of models to get the answers that they want.

3 Ways to Improve Generative AI for Accessibility

Evinced wants to use AI to to reduce the amount of tedium and increase satisfaction in working on accessibility issues. The goal is to do that in three different ways. First is intent detection. One of the great things that people can do with AI is use it to understand the context of the code that they’re developing or the type of design that they’re creating.

The second thing is to dig into these AI models with the expertise and understand how to handle prompt engineering. When it comes to creating prompts and trying to get responses that are both deterministic and correct, these prompts end up being extremely complex and often very long to get the types of answers people need to make productive changes.

The third is guardrails. We need to make sure to constrain the results from the AI and that they are accessible.

With these three areas — intent detection, expertise, and guardrails — people can use these tools in a way that’s helpful. Start with a design. How can people use AI to improve the process of design in general, and make sure that designers are always designing and creating designs that are accessible from the very start?

Design Assistant and Code Assistant Demos

How is it possible to ensure that AI models and tools like GitHub Co-Pilot return code that makes people’s jobs easier and faster as well as accessible that can be used by anybody? How can AI be used to find existing accessibility problems? Once found, how can it be used to fix them in a meaningful way? Evinced has developed a new set of tools to do that.

Design Assistant demo

Evinced took the underlying data from within Figma, applied the visual aspects, and used both AI and machine learning models to determine what type of design was being created. Then, it mapped it to an ARIA pattern and provided the designer with guardrails that helped them to create accessible designs every time they were developing a new component.

Code Assistant demo

Find Accessibility Issues Faster

How can developers find more accessibility issues when looking through the code that’s already developed? How can developers do it in a way that gives them the ability to reduce the amount of manual effort involved in going through every page of a website with a screen reader and a keyboard?

Modern technology makes it possible to find a lot more than any other tool. Like with Co-Pilot, it makes the developer’s job easier and less tedious. They’re finding the harder problems that are better to find and fix quickly. The next demo shows how the technology is used to detect more issues.

Finding and fixing problems demo

After identifying an issue, a developer can get a fix suggestion from AI. The developer would send the existing code snippet along with any prompts to AI and receive a new code snippet.

In the example, the suggestion for fixing a problem is to change an interactive element from a <div> to a <button>. At this point, the developer returns to the source code and makes the change to create a button and resolve the issue.

Video Highlights

Watch the Presentation

Transcript: Real AI Solutions for Accessibility Challenges - Kevin Berg

KEVIN: So today we're gonna talk about AI and how we can use it to help solve the digital accessibility problem. And how the future of accessibility tools can take advantage of the help that these types of generative AI models can provide, while not falling into some of the pitfalls that have equally been found. So to just jump right in, AI is clearly a tool that's going to help us. And in ways that we've measured so far.

GitHub did a controlled study of the impact that using an AI tool like GitHub Co-pilot can have on both the productivity and the satisfaction when it comes to developers. They did a study last year, where they had about 100 developers. They split them in half. Half of them used GitHub Co-Pilot to write a web server.

And the other half didn't. And what was really interesting is: We expected to help make things go faster. But to the point... Over 90 minutes was the amount of time that -- was the delta.

The average time with Co-pilot was 71 minutes. And the average time without was 161 minutes. So to be able to increase that amount of speed is awesome. And if you dug into the study a little bit, some of the more interesting parts that I found were that the developers really enjoyed using Co-Pilot.

Because it didn't necessarily take the challenge away from the development process. What it did is it took and kind of handled the more redundant parts, the more boring parts, that are easily done by AI. And it left the interesting challenges for the developers to solve. So it essentially helped the developers kind of save their brain power and mental energy for the problems that are more fun.

So I think the bottom line here is that, you know, these types of tools, they help make people faster at developing code. But also, developers are gonna want to use them. It makes their job easier. It makes their job more fun.

They get to do more of the fun part of coding, as opposed to the kind of tedium. So... However... This is the fun part.

The problem is that there are a number of different areas in which... Well, at least challenges to work through, when using AI models like this. Now... Thankfully, the problems like this are becoming less.

However, there's a lot of work to do, to make that bias go away. So here we have a prompt that someone wrote into ChatGPT. Where they were asked to write a Python function to check if someone would be a good scientist, based on a JSON description of their race and gender. Excuse me.

And the return was clearly biased toward white males. Which was a huge problem. And fortunately that seems to be fixed. So that's good.

But bias is one of the problems that we have with these AI models. And unfortunately, that impacts us significantly, when considering accessibility as well. And we'll dig into that a little bit more, as we look at the hands-on tools. And then another one is hallucinations.

So, for example, if we provide a prompt -- for example, "write a sentence that ends with the letter s". ChatGPT has presented a response that says "the birds in the trees sing sweetly at dawn". So it's a sentence that uses the letter s quite a bit. But it doesn't end with the letter s.

So... We have to be careful about our expectations around what kind of responses we're gonna be going to get. And in this case, we also have to consider what the prompt is, and how we need to communicate with these types of models to get the answers that we're looking for. So Evinced has taken an approach where we want to be able to take this and take the velocity increase, take the ability to reduce the amount of tedium and increase satisfaction, and do that in a way that helps solve the digital accessibility problem.

And so our goal is to do that in three different ways. First is intent detection. One of the great things that we can do with AI is use it to understand the context of the code that we're developing. Or in the case of design, in the type of design that we're creating.

The second thing is to really dig into these AI models with the expertise and understand how to handle the prompt engineering, which, when it comes to creating prompts and trying to get responses that are both deterministic and correct, these prompts end up being extremely complex and oftentimes very long, in order to get the types of answers we need to make productive change. And then third is guardrails. We need to make sure to constrain the results from the AI, to make sure that they are accessible.

So with these three areas, intent detection, expertise, and guardrails, we can use these tools in a way that's extremely helpful. So we will go ahead and start with a design. And we want to do this at every step. So starting with design, how can we use AI to improve the process of design in general, and making sure that designers are always designing, creating designs, that can be made accessible from the very beginning?

How can we help developers, folks that are using these AI models, how can we make sure that the help that they're getting from these tools like GitHub Co-Pilot is not only returning code that makes their jobs easier and faster and gives them better satisfaction, but how can we make sure that what type of code is returned is also accessible and can be used by anybody? And how can we use AI to find existing accessibility problems?

Which of course are always gonna happen. How can we use it to find problems? And then also at the end, once we've found the problems, how can we use it also to fix -- help fix those problems in a way that is meaningful? And so Evinced has developed our second generation of tools, which we're extremely excited about, that do exactly that.

So I'm gonna start by taking a look at our design tool. And the way this works is: The AI and the design is somewhat under the hood. So on the screen, on the left side, we have... It's an accordion pattern.

Which is essentially just a dropdown. And if you click on it, it will expose some more information. And so the first thing that we do with the design assistant is we select the component that we're designing. And then we're gonna scan that design to determine which ARIA pattern it maps to.

And in this case, the AI has determined that it's most likely an accordion pattern. That happens to be correct. So we can select that. And then go on to mapping the components.

And this is really the key. If you know the type of component that you're developing, or if the Evinced tool -- or the tool itself -- knows which type of component you're developing, then we know exactly what states are required in order to make it accessible from the start. So that way, if it's missing a particular state, then when you go to development, there's not a chance that the developer can really make it accessible, because it's missing the state at the design level.

So... Oops. Go back to that. So, again, we're analyzing the component.

And then once the component is analyzed, and we understand that it's an accordion pattern, we then analyze the current states that are there. So in this case, we have a default collapsed state and a default expanded state. But we are missing the required focus states. And so that is our next task.

We need to go through and create a copy of these. And create the focus states. And now we've fulfilled the requirements for the accessibility label. We can scan it for all the traditional things, like color contrast and text spacing, and all of the things like that.

And then we're able to provide information to developers and designers, in order to understand how to develop that in an accessible way. So the key here is that with AI, we were able to take the underlying data from within Figma, use the visual aspects as well, use both AI and machine learning models to determine what type of design that we're creating, and then map it to an ARIA pattern and provide the designer with guardrails that helps them to create accessible designs every time they're developing a new component.

So that's how we can help from the design perspective. Next we'll look at how we can approach it from the code perspective. And for that, I'm gonna jump into a live demo. Where I'll just pull up my IDE.

Which is VSCode, in this case. And I am today going to put my shoes in as a frontend developer. Where I'm developing a tablist of orchestral instruments. A secret about me is I used to be a professional musician, playing in symphonies.

And so I'm creating my website here. Where I'm listing out different types of instrument families. So... I have GitHub Co-Pilot linked in here.

And the nice thing about this is, if I'm trying to create a list of orchestral instruments, all I can do is -- I've left a comment. And if I just hit the enter button, it will create a tablist for me. Now, as we look at this, we can see that it's been built with divs. I'm not seeing any sort of ARIA roles or anything that would make this accessible.

And this is kind of the underlying problem. As more developers are using tools like GitHub Co-Pilot, or Amazon Code Whisperer, or many of the various others, while much of the code that has trained this type of tool is accessible, and does provide good examples, unfortunately the overwhelming majority is not. And so the training that they provide and the overwhelming answers that they... That are fed back to the user are with code that is not accessible.

And so this is actually a really huge problem. Because as... From the outside of our talk here, if both developers really want to use these types of tools, which nobody can blame them for -- it makes their job easier and more fun -- and they're able to develop codeine faster, the prospect that this code won't be accessible really compounds our current problem that we have today. Because the code features, applications, will be developed at a faster pace.

Potentially with just as many or more accessibility issues than we're finding today. And for folks that are on the front lines, trying to kind of mitigate these problems, day in and day out, that is not a proposition that we want to deal with. To have more problems to find at all times. So in order to mitigate this, we can continue.

We just... We generate the code. We can even generate more options, and so perhaps we'll get lucky and Co-Pilot will provide us something a little bit better with the kind of additional options. But unfortunately the options that are provided -- none of them really have any sort of accessibility in mind at all.

We've just got divs and h1, h2, list items, nothing interactive. It's all very, very inaccessible. And so this inherently shows the underlying problem. And not only just with HTML.

We can also add script tags, and it can generate the JavaScript as well. And this time ChatGPT is funny. It just... In this case, put a comment of "write your code here".

Again with the hallucinations that we discussed earlier. So now we can take a look. So if we were to build an application that uses... With just GitHub Co-Pilot, taking charge, we would see that...

We would create an application that behaves as following. So on the screen now, I have just a general template with a tablist that lists, you know, string instruments with a description and an option to learn more. We've got a wind instruments tab. As well as a percussion tab.

And if we were just to tab through this, we would go from -- first we're on the first tab. And now in an accessible application, we would expect it to enter into the tab panel and then go to the "learn more" button or link. But unfortunately, it goes straight from the string tab to the next tab over, which is the winds tab. And then the percussion tab.

And then it jumps into the tab panel. And shows "learn more". And so if I was a screen reader user, I would hear "String, wind, percussion, and learn more". And I would think I was learning more with percussion.

And so here is the underlying problem. So... At Evinced, what we've done is we have created an intermediary. So I'll go ahead and revert this.

I'm going to go ahead and add our Evinced Co-Pilot. I'll just save that. And what I need to do is just close out the VSCode, just to clear any cache. Anything like that.

And so now I'm gonna do the exact same thing I did before. I've got our initial tablist. It's got the comment with string, winds, percussion. And when I hit the enter key this time, I'm expecting to see something very, very different.

And in fact, we do. This time, instead of a ton of list items and divs and things like that, I'm seeing HTML buttons with ARIA roles, ARIA controls. ids. ARIA selected -- all of these types of things that we're expecting to see.

And in fact, if I were to create some broader options, we would be able to show... Once GitHub responds here, we're seeing much more of what we're excited about. First the answer is being provided by Evinced. We've got roles, tabs, all sorts of things, where we can review the different suggestions, find the one that works for us, and accept it and move over.

And so this way Evinced is both using the intent detection piece, where we can tell that we're looking at a tablist, so we can provide the HTML in an accessible way for that type of component, we're using the expertise as far as the intent detection, of communicating -- because we know it's a tablist, communicating to ChatGPT all of the different areas that are required for a tablist. And throughout this process, I really don't want to be throwing developers under the hood.

I mean, many great developers know about accessibility, and are doing the right thing. And doing the best they can to create accessible code. The problem is that, like all of us, developers have a lot on their mind. And remembering what to do, when it comes to accessibility, is actually a really tall order.

If we go look at the different ARIA role models, what we've got here is an insane spiderweb of correlations, based on the type of element we're creating. Where even the tablist -- we have to know about ARIA level, ARIA multiselectable, ARIA orientation, all of these things. And let alone knowing how to be a developer, remembering all of these things is just almost too much for anyone. And so what this allows us to do is create the code, as you normally would, and it helps guide developers to creating the accessible code in a way that's equally as fast and equally as enjoyable for them.

They just don't have to think about -- one less thing to think about. We help them create accessible code as we go. So, jumping back into our example site here, what we can now do is try again. Let's try our tab.

First we go to the first tab. That's excellent. We're on the strings tab. And then instead of going to the next tab over, we're going into the tab panel.

I've highlighted that. We go to the "learn more". Then we go to the next tab. We can use the arrow keys to get to the next tab, as expected.

We can get in, and everything works the way it is supposed to here. Except for my tabbing ability. So... We're able to do this.

So with the guardrails provided by the Evinced tool, we're able to then provide accessible code and create accessible features. All right. So the next area we want to talk about is: How are we able to find more accessibility issues, existing issues, when looking through the code that's already developed, already out there, when we're finding problems? How can we do that in a way that is -- gives us the ability to reduce the amount of manual effort that it takes to go through every page of your website with a screen reader and a keyboard and what have you?

And now, of course, I'm not going to sit here and claim that we have the ability to find all problems. But using modern technology, we can find a lot more than any other tool, which, again, similar to Co-Pilot, it makes your job easier. Less tedium. You're doing this stuff, you're finding the harder problems that are better to find and fix quickly.

And so... What we'll do here is... We'll just jump into a quick demo to show how the technology is used, to be able to detect more issues. So just as a baseline, what we'll do is we'll just -- I've loaded up a dummy website here.

It's meant to mimic a travel site, where you can select the type of vacation you want to have, where you want to go. There's a date picker, so you can select when you want to go. And then there's also a search button, so that you can complete the search, and get your results. And so the first thing we can do to just set a baseline here is just run a legacy tool.

We'll open up Lighthouse, and use Google Lighthouse. This is a fine tool. It is able to find some interesting accessibility issues. But the problem is really in the way that it works.

Lighthouse uses axe-core under the hood. There are some nice issues you can find there. But the way that axe works is it looks at the HTML, analyzing the syntax, and really just essentially checking to see if anything is missing. And so what happens is, because you don't have an understanding of how the user is meant to interact with the website, you don't have enough context to find a lot of accessibility issues, especially some of the critical ones involving critical screen reader issues and keyboard accessible issues that might be the ones that are really preventing users from using your website in an appropriate way.

So even though we've got a score of 100 from Lighthouse, if I were to tab into this website, and hit the tab key one time, unfortunately I'd skip over the first dropdown, the second dropdown, I can get through the date picker okay, but unfortunately, I'm not able to get to the search button either. And that's because these elements have been coded incorrectly. From an accessibility perspective and honestly, a development perspective as well.

And unfortunately, while this is obviously a demo site, these are the types of issues that we see over and over, across the web. Where these types of things are -- they make elements that are meant to be interacted with impossible to get to. And so in the case of a site like this, if I were a keyboard-only user, the only thing I can do on this site is select a date. Which is not good for anybody.

So what I'll go and do is I'll just fire up Evinced. And the Evinced approach is a little bit different. And the way that we analyze the page is, again, using the modern technology. We use computer vision, machine learning, as well as AI.

To gain additional context. So we analyze the page. And we're able to determine that, hey, okay. We've got a top level heading at the top of the page that says "travel: Book your stay".

And Evinced understands that you've got a form with two dropdowns. Where you want to go. What type of vacation. The date picker to choose when.

And then a search button. So the key is, for example, the search button -- Evinced knows ahead of time that this search button is a button and a user is meant to interact with it. And so in that particular case, when we then look at the HTML code, we have a set of expectations, as opposed to just looking if something's missing. So we're expecting to see an HTML button tag with all of the lovely accessibility features built in.

Or at the very least, we're expecting to see the various ARIA attributes that would be required in order to, say, gain keyboard focus or declare control to the screen reader. What have you. And so it's when we don't see those -- it's when we can go ahead and scan the page very quickly. And we can determine that there are accessibility issues on these elements.

Where other tools can't. Because we have used the technology to determine that... How this page is meant to be used, and so then we know how to test it. So this is how we can use these technologies to detect more issues.

Now, jumping back to our slides, so... Great. We can find more issues faster. What do we do once we've found them?

How do we fix them? Well, in almost exactly the same way as we saw before. What we do is, of course, like tools do, we provide all of the information around the type of issues that we've detected. And then we take all of that information, and we have the broken DOM snippet, the severity, the type of issues, we have -- since we have access to the HTML and the visual aspects, and the analysis we've done, we can use all of that to send a prompt to a generative AI tool, to get an explanation of how we should fix this issue.

And so, when we do that, we just simply would click the Get Fix Suggestion. And what that would do is return us an answer. So it takes the existing code snippet, and with our prompts, it returns a new code snippet that is going to be accessible. In this case, we've taken a div that was designed...

Excuse me. As an interactive element. A button that was meant to be clicked on. Which of course comes with no accessibility.

Doesn't declare control as a screen reader and you can't get to it with a keyboard. And we've changed that to a button tag. And so at this point, we would just go back to our source code, make the change to make sure that knowing exactly we need to change that div to create a button, and that way, it will resolve that issue. Yeah.

So... Thank you all very much. We're excited to share here how we are using this modern technology in a place that's sorely needed to make sure that, as the technology advances, we are advancing with it, and making sure that accessibility is easy for folks to add into part of their process, without a huge amount of effort, knowledge, and time to do. So...

Thank you all. I would be happy to take any questions.

CAMERON: Thankyou Kevin. I am super exceited about this. Yes, right here at the front.

Kevin Andrews: This is Kevin Andrews from Georgetown University. Really appreciated your talk this evening. I lead our IT accessibility compliance program. And this is really interesting, because we're kind of having some conversations at the IT level about AI, and how, for better or for worse, we can use it at the university.

I think they're still trying to figure that out, as we all are. And I was wondering if you could speak to... It sounds like... I'm not...

Unfortunately, I don't know much about your organization. But I was wondering if you could maybe speak for a minute or two about how teams have successfully integrated your product into their development processes. Obviously without getting too into the weeds. I would be curious to hear about that.

We're agile. So I would just be interested in hearing about that. And how we might be able to use that in our sprints.

KEVIN: Yeah. Absolutely. So... To keep it somewhat high level, there's several areas in which that can happen.

As an agile part... I mean, regardless, we want to get in as early and as often to hit accessibility problems at the source. One of the ways that a colleague of mine puts it is that... You know, a bug is a bug.

Whether it's a functional problem or an accessibility problem or a security problem or just a flat-out infinite loop, and we want to approach everything that way, and make sure that accessibility is part of that. So whether that's starting early with the design process, our customers are wanting more and more, earlier and earlier, so that these problems aren't created at the very beginning, or they're caught extremely early. So that they -- by the time their applications get to production, you know, they're as close as we can make them, when these, of course, applications are ever-evolving.

And so our customers... We're fortunate at Evinced to have so many different tools that fit in different places in the pipeline. What we do is we work with our customers to kind of customize how they would best do it. With both an initial phase of how we can start today to make the problem better, and then also how we can work towards tomorrow to make sure that the problem is more solved, going forward.

So... You know, we would probably want to sit down and chat about what would work better for you all. But we've kind of seen just about everything these days. So I'm confident we would know how to help.

CAMERON: And Mike is over here in the front. If you want to chat afterwards. Otherwise in the room? Ben was someone else.

But you can follow up. The mic is right in front of you. Got it. Here you go, Ben.

Ben Ogilvie: Hey, Kevin. It's Ben, at ArcTouch. Correct me if I'm wrong. I think I remember you saying this a few weeks ago.

As of right now, the V2 tools that you demoed today focus on web at the moment. Not just kind of taking native into account, right?

KEVIN: That's correct. Yep. That's phase 2 of our gen2 tools, is gonna be mobile as well. Because obviously that's where we go to next.

Ben Ogilvie: Awesome. Cool. Within the web stack, is a developer able to tailor the suggestions to different frameworks as of right now? Or is it kind of pure native HTML for now?

KEVIN: Yeah. The fix suggestions -- the first phase was more of the raw HTML. So that you take that general understanding, take it back to whatever your framework is, React, Flutter, Angular, what have you, and kind of apply it there. Our initial state will be to then provide examples in those most popular frameworks.

But then as we're refining our prompts, we want to be able to just have those return directly in the languages. Because that would be one step easier. But you know... With everything, that's a bit of a challenge too.

So we're working on that as well.

Ben Ogilvie: Thanks.

CAMERON: Yeah. Another question in the room.

Jonathan Pool: Hi, Kevin. This is Jonathan Pool from CVS Health.

KEVIN: Mr. Pool, great to hear your voice again.

Jonathan Pool: Yeah. I was wondering... Is there a model where the AI that detects accessibility defects can tell the AI that creates applications about its defects, and teaches it to avoid such defects?

KEVIN: Yeah. So the idea of training AI models is really an important one. And it's really an area where, as leaders of accessibility in the space... We need to do everything possible as we can to make sure that what we put out is accessible.

Because if you're a Microsoft or a Google or an Asana, your footprint is going to be incredibly impactful on the code that these models are trained on. How the continued learning happens with the prompts that are returned. And so... It's a really interesting approach that we need to take.

So... You know, these models are trained on extremely large sets of data. It's difficult to pivot quickly. But it's really a long-term goal for the more good code that we put out there -- the more of that is trained to return, and when these models are asked for simple -- you won't have to specify that it needs to be accessible.

That will be an assumption that needs to be made. And so hopefully... It'll probably be some time in the future, but our goal would be -- at Evinced, if our tools work and we're able to provide more accessible code out there, well, then, we'll further train the models, and someday you won't need tools like ours to make code accessible. These generative AI tools will know how to do that on their own.

So I'm not sure if I answered your question directly. But those are the thoughts.

CAMERON: I'm getting a nod.

KEVIN: Okay.

CAMERON: So that seems good. Any other questions in the room? Yes? Jack?

One second.

JACK: Hi, Kevin. I really enjoyed your presentation. Jack here. I'm a UX designer.

That's why I want to know what kind of designer plugins you are developing for Figma or Sketch or other tools. Yeah. That's my question.

KEVIN: Yeah, absolutely. So our design assistant is built for Figma at the moment. In a quick poll of our customers, Figma was overwhelmingly the most popular. But we are planning to add support for Sketch and those as well.

But yeah. The design assistant -- it's really exciting. Any component you're developing, whether it's a tablist or chording -- whenever we match those with the ARIA pattern guide, as well as some other sources as well, to make sure that those designs can be made accessible from the design standpoint. So...

Yeah. We're very excited about it.

CAMERON: If people on the stream have questions, feel free to ping Meryl in the chat, and I would be happy to route them here. I have a question, Kevin, about scanning of actual source code. So you showed a Co-Pilot model, where something was identified at the authoring time. And you showed a scanning model in the DOM that addressed code in production, essentially.

Right? Or in a local development environment. Are you thinking about ways to basically do static analysis on a codebase that would take some of the benefits of the GitHub Co-Pilot kind of model, but do it, like, wholesale, over an entire codebase?

KEVIN: Yeah, absolutely. That's one of our goals, to be sure. And there's significant challenges with that problem. One is...

In order to be able to cover accessibility to a certain extent, we have to be able to kind of automatically detect what type of element it is. Like... For example, in HTML, it can be difficult to tell whether an element is a modal or a pop-up or different types of areas. If you know that...

So if a tool knows that this element is a modal, we can scan it for almost every accessibility issue there is. And right now we use multiple data streams in order to detect that information. So of course we look at the code if it's available. We look at the syntax.

We also... One of the areas we do -- if it's rendered, whether it's in JS DOM or in the browser -- we use additional data there. And so we have a plan for how we're gonna do that. That's just a little bit further out.

But we're excited when that tool is gonna be available. Because essentially what you'll be able to do is point is towards the code repository and get an accessibility output for all of your repositories.

CAMERON: I'm looking forward to that.

KEVIN: Me too!

CAMERON: Questions in the room? Yes?

PAOLO: Thank you. Hi, Kevin. This is Paolo from Hearst Magazines. I'm in Product.

I'm just curious to hear... Never worked with accessibility. But now I think that I will start. Thank you very much for the presentation.

That was very, very clear. I wonder... Like, during your presentation, you mentioned Lighthouse. What you are going to do if Google decides to improve that Lighthouse?

KEVIN: I mean... First thing we would do is celebrate. Because the way I feel in this industry is: The more people doing good to solve problems, the better. But you know...

Great! There's a ton of tools out there. The more... I guess that's my answer.

The more help out there for folks to make the more things accessible, the better.

CAMERON: Okay. There's a YouTube question. Sorry. Give me one second.

I'm reading through here. Emendip on the screen, you asked... Can we take that as an input from developers, when they're authoring these UI widgets -- kind of taking the guess away? I'm not sure what you're referring to specifically.

Can you add a little more context to that question? And I will take another pass at it? And then Anandip, sorry if I'm pronouncing that incorrectly... Can Evinced help with the service design or the creation of service blueprints?

I don't know what service blueprints are. Maybe you do, Kevin.

KEVIN: I apologize. I'm not familiar with the service blueprint. I can speak a little bit about Evinced in general, if that would help.

CAMERON: I'll let Ana maybe do a follow-up on that. There's a question in the room here.

RYAN: Hi, Kevin. Thank you for your demo and your talk today. It was really great. My name is Ryan.

I'm an accessibility frontend engineer. And I'm curious about the example that you showcased today -- was for a tablist pattern. Which is a pretty standard ARIA pattern guide example. I'm curious about how the Evinced tool would handle any component that has non-trivial interactions or things that are not so easily fit into any of the ARIA practice guide examples.

KEVIN: Yeah. That's a great question. Because... You know, while...

I would say the majority of components fall into one of these categories, there's always custom. Everybody... Everything... Every website is different, to some extent.

Somebody has got a good idea on how to improve it. And that's something to configure, to absolutely consider. So our approach to that is: First, we have built out our tool to recognize all of the different ARIA patterns, match them to the guides, plus some additional feedback in areas where they're lacking. To map those.

The second thing that we've noticed is a lot of the, say, custom components that we see are really just kind of combination of existing patterns, and then we see components that are truly custom, that have requirements that we would need to focus on. And so that's also our pattern. The way we're approaching it. We will start with the most common, hit the analysis for combinations, and then work on the ability to address pure custom components as well.

So... Tricky problems. Exciting ones. But a great question.

CAMERON: Amandip followed up in the YouTube channel, the question about taking input from the developer, about what they want to try to do. It's a question about intent. Like... Asking...

Setting the intent that this is a dialogue or tab or accordion, while asking Co-Pilot to create code for them. Can you refresh us? I think that's something that you showed. There's a comment mechanism that says, like, make me an accordion?

KEVIN: Yeah. It's really interesting, the way GitHub works. The Co-Pilot. You can essentially leave a comment of...

What kind of general guidelines. And then you can ask GitHub Co-Pilot to create some code for you. So... What's interesting is...

When we created this tool, you know, just when we started getting into it, we thought... Okay. So what we'll do is we'll say... Hey, Co-Pilot.

Develop us a tablist. And the Evinced tool wouldn't have to do anything else, besides analyze the ten results and then pick the one that's made accessible. But the problem was: When we looked at ten results, none of them at all were accessible. And so we had to actually take a much deeper approach, which involved the two areas of technology that we've focused on.

One of them is the intent detection. Where we're detecting that that's a tablist. Some of the detection is user input. We take that.

Some of it is kind of the comment as well. Some of it is just contextual. And then the second part is: Okay. You can't just say, you know, please return me a tablist.

You need to be extremely clear with the prompt to say this needs to be returned as... This is a tablist, it has to be accessible. You have to have the screen reader areas. The prompt itself has to be extremely complex to make sure that you're getting the answer you need.

And you have to be able to work with the generative AI model to make sure that the results are deterministic. You have to make sure that it's going to return you an answer that is the same every time. Which is another challenge that we've worked through with these particular tools.

CAMERON: Okay. Anyone else in the room have a question? Or concern? Thought?

Yes?

Jonathan Poole: Jonathan Poole again. Does AI have a role to play in helping large organizations with thousands of pages that have been created by dispersed autonomous groups arrive at accessibility in the form of consistent representation across all of those pages? Does it... Can it play a role in identifying candidates for, let's say, the canonical representation that should then be propagated across the organization?

KEVIN: So let me try to rephrase, to make sure I understand. Can AI detect... Like, a website amongst thousands that is like a great example of accessibility to then propagate outwards to the other pages?

Jonathan Pool: I meant that... Let's say... Buttons or links can be all accessible, and yet they're not accessible, because they're inconsistent across all of the pages of a site. And so to solve that problem, you need...

You can't use page level analysis. And I'm wondering whether AI can be brought to bear, to find the... Let's say some kind of consensus, or modal patterns that can be promoted into some kind of a design system for the organization?

KEVIN: Sure. The nice thing about... I mean, large numbers of pages are easy. Evinced can scan a thousand pages in four minutes.

Using all of the AI modeling that we do, to create that intent and detect how each interactive element is developed. I think the guidelines that you're referring to are kind of already established, with the ARIA pattern guide, the WCAG criteria, and that sort of thing. So this may be something we want to discuss in a little more detail. I'm not entirely sure...

I mean, the short answer is... We don't even know what-all AI can do. And so one wouldn't really want to close the door on anything.

CAMERON: Okay. Well... Oh, yes. Great.

I think we'll do this question and one more question, if there is one.

Daniela Decker: Hi. Thank you again. My name is Daniela Decker. This is my first Meetup, so I'm excited to be a part of the group here in New York City.

I'm an independent consultant for the International Finance Corporation at the World Bank. And a few other clients as well. And I'm particularly interested not so much from a developer or designer kind of standpoint on the technical piece, but more so on how AI impacts hiring and retention, specifically for disabled employees, or disabled job seekers. I think that is a really important fundamental element of how we talk about AI and how it is used to streamline and support HR teams.

Yet it is often discriminatory. So I would love to hear your perspective on those challenges, as well as opportunities to ensure that AI is not biased, when we're looking at disabled job seekers and people seeking employment.

KEVIN: Yeah. The bias instilled in AI is an incredibly serious problem. And the problem lies... The data used to train these models out there is implicitly and explicitly biased as well.

And so... Yeah. This is an extremely complex topic that I don't think... I think everyone is trying to work through.

Unfortunately, I don't have a great answer, except for that it should be everyone's goal to identify and put forth examples of non-bias. Call it out where we see it. And act for change. So that, again, just like when we're looking at accessible code, the more good content out there that can be used to train these models, the better.

The more improvement we'll see. But of course, this is gonna take a lot of time.

CAMERON: I've also seen a handful of organizations publishing AI ethics guidelines that the organization is committed to. So... If you're in a position to enact that kind of policy, I encourage you to research and see what other organizations are doing. I know Google and Microsoft -- the usual suspects -- are publishing these kinds of ethical guidance.

Asana does too. So that's somewhere to look for inspiration. And also a call to action. There are also a handful of other events that we have recorded on our YouTube channel, that relate to AI and bias.

And I think it's a great idea for a topic for another Meetup, to really do a deep dive there. So thank you for the question. Did you have a question? Yes.

Natalie Austin; Hi. My name is Natalie Austin. I'm an accessibility products specialist at Edward Jones. I had a question relating to kind of these...

Or let me go back a step. I'm hearing a lot about ARIA. And I have a bit of pause. Because I come from an environment where it's so overused to a point where it's not useful at all.

I know that some of the W3C patterns -- for instance radio buttons -- those are pure ARIA. They're not actually just using good semantic HTML. So I guess I'm curious how these AI tools can suggest the right patterns. Because I guess I'm just concerned of...

Promoting so much ARIA and just getting away from good semantic HTML.

KEVIN: Yeah. Absolutely. And I think... I think in my opinion, that's keeping these tools in perspective.

These tools are not replacing good old-fashioned human judgment, knowledge, and understanding. And we need to treat them as well. Like... I kind of think of...

It's silly. But I kind of think of these generative AI tools as like a really great shovel. They make the job easier, but they don't tell you where or how to dig or how to solve your problem. So...

My only point being... We have to use judgment when using any new tools. Anything that's still evolving, to make sure that we're using it in a way that makes sense. And I couldn't agree with your comments more.

We need to use a lot of judgment and use it in a way that helps instead of hinders.

CAMERON: Thank you to Kevin for joining us tonight. Thank you again!

Bio

Kevin Berg is the VP of Solution Engineering at Evinced, where he is dedicated to helping leading enterprises improve and accelerate their digital accessibility efforts. Kevin trained originally as a classical musician at the University of Cincinnati and the University of Texas, where he got his doctoral degree.

He spent six years at Sauce Labs, a leading functional test automation platform, where he eventually led global solutions engineering before joining Evinced in a similar role. When he is not helping Evinced customers through challenges, Kevin is home in Montana making stuff (he is a hobby machinist), playing his oboe, and enjoying family life with his wife and two kids.