What Automated Testing Misses: Six Accessibility Problems That Pass a Scan

An automated accessibility scan checks a page's code for known failures, such as missing alt text or low color contrast, but it cannot tell you whether a customer can finish a task on your site. During an accessibility review for a small business customer, we found that no one could tap the Purchase button on an iPhone, because of a plugin the customer did not know about. This post shows six problems that pass a scan but still cost you customers, with a fix for each, and ends with steps you can take this month.

On October 20, 2026, I spoke at "Accelerating Accessibility as a Competitive Advantage," an online evening for small and medium business owners hosted on Zoom by New York State Assemblymember Stefani L. Zinerman. Marc Safman of Safman Consulting organized the event, and Peter Slatin of Slatin Group moderated. My part of the program was a talk titled "What Automated Testing Misses."

At Equal Entry, the digital accessibility consulting company I run, we see this pattern regularly: an organization says it has done accessibility work, and then a screen reader user still cannot complete a purchase, submit a form, or find the information they came for. I have also seen it from the other side. From 2009 to 2012, I was in charge of product management and then Chief Technology Officer at HiSoftware, which made Compliance Sheriff, a product that scanned web pages for accessibility issues. I saw firsthand what a scanner can tell you and what it cannot.

This is not an argument against the Web Content Accessibility Guidelines (WCAG). WCAG is essential, and conformance is the right baseline. Much of what I describe below is exactly what WCAG intends. The problem is that using only code to decide whether something meets WCAG is too mechanical, and not human enough for what WCAG actually requires. A green report is not proof that people can accomplish real tasks. The distance between those two outcomes is what I call the gap between technical compliance and real-world usability.

Key points

• Passing an automated scan is not proof that your site meets WCAG.
• Automated tools can check the code and identify specific failures, but knowing whether content is perceivable, operable, understandable, and robust requires human evaluation, ideally including real assistive technology use.
• Many customers who hit a barrier never report it and simply take their business elsewhere.
• Just because it works for you does not mean it works for your customers.
• Passing automated checks can lower some legal risk, but it does not prove that people with disabilities can use your site.
• The law might be how you were introduced to accessibility, but the business value is why you'll want to keep it.
• Try your most important task with only a keyboard and a stopwatch, and count the Tab presses.
• Put an Accessibility link in your footer and make sure someone is listening.
• Writing down and testing your critical workflows turns accessibility into everyday quality assurance.
• Conformance is the baseline, and listening to your users is where accessibility becomes useful.

You are not your user

I have spent years telling clients to respect user settings. Then I started needing glasses, and the change happened quickly. I was taking a bus to a friend's house in New Jersey, and the transit app showed the bus number in very small text. It did not respect my font size preference. I misread the number, got on the wrong bus, and ended up somewhere in the middle of New Jersey where I did not want to be.

That was not a theoretical accessibility issue. A small typography decision turned into a transportation failure, and it changed what I trusted that app to do for me. It is also a reminder that people do not always think of themselves as disabled before accessibility becomes relevant to them. You are not your user, and one day you may be the user who depends on the settings you ignored.

If you own a small or medium business, the most important thing to understand is your customers and how they actually use your site. Your customers are not thinking about your code. They are thinking about what it feels like to navigate, perceive, and understand your content, and whether they can finish what they came to do.

A scan checks the code. People use the site.

Automated tools are useful. They catch missing attributes, broken markup, and low color contrast. Technology-only accessibility solutions have been around for a long time, and I am not arguing against using them. The point is to understand what they miss and what else needs to be part of the process.

WCAG success criteria are written to be testable, which is their greatest strength. A tester can verify that an image has a text alternative, that a control has an accessible name, or that text meets a 4.5:1 contrast ratio. Automated tools are good at these checks, and they are objective and repeatable.

A scanner mostly confirms that something is present in the code. Only a person can tell you whether it works. A scan usually cannot try to buy something. It cannot get confused by an error message or become trapped inside a pop-up with no way out. Real customers can, and when they do, they leave. The table below shows how a passing check and a usable result can diverge.

In each row, the left column can pass while the right column fails. That is not a flaw in WCAG, which calls for meaningful alternatives and understandable instructions. The gap appears when teams substitute a checklist for human judgment. A user does not experience success criteria. A user experiences the time, effort, and confidence it takes to finish a task.

Six problems that pass a scan

Each of these problems can pass a scan and still block a customer. To illustrate each example in the talk, I used Knomo, the Equal Entry mascot. The name stands for "knowledge monster," and Knomo is a round, bright yellow character with a big smile, arms raised in a cheer, and a blue Equal Entry baseball cap. Each of the six problems below is one my company has seen on small and medium business websites, and none of them is something technology alone will reliably catch. They come up repeatedly in our manual testing with keyboards, zoom, voice control, and screen readers.

1. The alt text is there, but it is wrong

Perhaps the best-known accessibility requirement is that images need a text alternative, so a person who is blind or has low vision still knows what an image shows. That description is called alt text, and a screen reader reads it aloud in place of the image.

Picture an online clothing store with three products for the cold weather ahead: a red sweater, a blue cardigan, and a green hoodie. Every product photo has alt text, so the scan shows no errors. At most, some tools add a warning. But the screen reader announces "product image," "product image," "product image." Nobody is trying to buy a product image. The shopper is trying to buy something specific, and now they cannot tell the three items apart.

A scan can tell you that alt text exists. It does not know your business or what you are showing your buyers. Done right, each photo names what a buyer needs to know: "Red crewneck sweater," "Blue cardigan with buttons," "Green hoodie with front pocket." Now the shopper can compare the three, choose one, and add it to the cart.

The same thing happens with charts. An image of a sales chart has the alt text "Bar chart." A text alternative exists, so the tool passes it. The information the chart communicates, such as which quarter performed best, is still unavailable.

This is a small fix, a short description of a product you already sell. It is also one of the most common problems we see, whether the alt text is too generic or the same text is reused across many images.

2. The checkout only works at normal zoom

You run a scan on your checkout page, and it finds no errors. But some customers need larger text, so they zoom in. A typical scan does not do that. It checks the page at one size and stops.

In my example, a customer at a flower shop is ready to pay. The order total is $48, and they have entered their card number. All that remains is the submit button. At their zoom level, that button runs off the edge of the screen, and tapping it does not register. The customer has their card out, and you lose the sale at the final step.

Done right, the page reflows. The layout rearranges itself to fit the screen instead of running off the side. The card field fits, and the button reads "Pay $48" right where the customer can tap it. WCAG expects text to be resizable to 200 percent and content to reflow without horizontal scrolling at the equivalent of 400 percent zoom. A page that reflows also works better on a small phone for everyone.

3. The error message nobody understands

Every field on a form can have a label, and the scan will pass. But a scan never fills out the form, and it never makes a mistake. Customers do.

In my example, a customer completes a delivery checkout with their name, street address, city, state, ZIP code, and delivery notes. Everything looks right. They press submit, and a small message appears at the top: "Invalid input." It does not say which field is wrong or what to change. The ZIP code has only four digits, but nothing on the page says so. A confused customer may decide the site simply does not work and leave, or pick up the phone and call you instead.

A scan has no way to judge that message. The message also falls short of WCAG, which asks you to name the field with the error and, when the fix is known, to suggest it.

Done right, the message names the field and the fix: "The ZIP code needs 5 numbers, for example 11201. You entered 4." It offers an example of a valid ZIP code, and the ZIP field itself is marked with an error label so the customer finds it right away.

4. The pop-up you cannot escape

Many websites are designed with only a mouse or a finger in mind. Many of us have seen the newsletter or coupon pop-up that covers the page the moment you arrive. A scan of that page may find no errors, because a scan never tries to close anything.

A customer who cannot use a mouse or a touch screen may be using a keyboard or a switch device. In my example, a shopper at an online boutique presses Escape, and nothing happens. They press Tab, looking for a close button, and focus circles through the pop-up with no way out. The products are right behind it, and they cannot reach a single one.

Done right, Escape closes the pop-up and focus returns to the page, so the keyboard picks up where the shopper was instead of losing their place. The newsletter still gets its chance. Anyone who wants to sign up still can. You have simply stopped blocking everyone who does not.

Two more keyboard problems a scan passes

Keyboard users run into two other problems a scan passes. A checkout page places sixty focusable elements before the payment form. The focus order is logical, but a keyboard user must press Tab dozens of times to reach the one field they need. And a custom menu uses valid ARIA (Accessible Rich Internet Applications) roles and states, yet behaves differently from the pattern the role promises. The screen reader announces a menu, the user presses the arrow keys as expected, and nothing happens. Predictability is itself an accessibility feature.

The page may look accessible to a scan, but the only way to know is to sit down with the keyboard and not touch your mouse.

5. The links that are not descriptive

This is one of my favorite examples because it involves someone who does not use a mouse or a keyboard at all. They use their voice to control the software. That could be a customer with carpal tunnel syndrome, an amputee, someone with another motor disability, or simply someone who prefers voice control.

In my example, a bakery website shows bread, croissants, and a birthday cake, and under each one is a link that says "Read More." The customer says "Click Read More." Because the page has three of them, the software places a number on each link, and the customer has to work out which number belongs to the croissants. A screen reader user who pulls up the list of links hears "Read More" three times with no idea which is which. Some scans flag vague link text, but usually only as a warning, and warnings are easy to skip.

It gets worse at scale. A screen reader user moving through a product listing hears "Add to cart, button" forty times. Each button has an accessible name, so the check passes. But they cannot tell which product each button belongs to without navigating back and forth.

Done right, each link names where it goes: "Sourdough bread," "Chocolate croissants," "Birthday cakes." The customer says "Click Chocolate croissants" and goes straight there. This example is about efficiency. You can learn a lot by trying to use your own site with voice control, the same way you would try it with only a keyboard.

6. The captions that are low quality

A video can have captions, and the scan will pass. At most, a scan checks that a caption track exists. It does not watch the video or read what the captions say.

In my example, a local yoga studio posts a class video. A deaf or hard of hearing customer, or anyone watching with the sound off, follows along on their mat. The automatic captions read: "breed in and raise your arms over your hand then fold four word into down word dog." The customer cannot follow the class, so they do not book one.

Done right, someone reviews the captions before the video is posted: "Breathe in and raise your arms. Then fold forward into downward dog." Captions need to be accurate, and the fix takes one step. Review and correct automatic captions before you publish.

None of these issues is exotic. They are the ordinary result of treating a checklist as the finish line rather than the starting line.

The most expensive barrier is the one nobody reports

Many customers who hit a barrier do not file an accessibility ticket. They give up, use another business, or avoid the site in the future. On a checkout page the stakes are simple: the customer is trying to give you money and cannot. If people cannot operate the site, they leave.

Reporting a barrier also takes extra effort. When a customer makes that effort and nothing changes, trust erodes, and they become less likely to report the next problem. When a company responds quickly, listens, and fixes the issue, the customer comes away seeing a business that understands them. How you respond to accessibility feedback becomes part of your brand.

So the absence of complaints is good news, but it is not evidence that there are no barriers.

Why organizations stop at compliance

Most teams that stop at a clean report are not indifferent to users. They are responding to the incentives in front of them.

• Legal risk drives the budget. Accessibility work is often funded in response to a demand letter, a procurement requirement, or a regulatory deadline. Those drivers ask for conformance, so a report becomes the definition of done.
• Checklists are easier to schedule than experiences. A list of success criteria fits neatly into a sprint. Task-based usability is harder to estimate and harder to close as a ticket.
• Automated tools create a false sense of coverage. Automated scanners catch real issues, but only some of the issues a person finds. A green dashboard can suggest the work is complete when the most consequential barriers have not been examined.
• Disabled users are absent from the process. When no one on the team uses assistive technology daily, the difference between "announced" and "understandable" is easy to miss.

Accessibility testing finds bugs that affect every customer

Testing your site the way a customer uses it finds bugs that affect everyone. Testing with people who use assistive technology catches problems that scanners miss.

In the small business review I described at the start, it was not just screen reader users who could not tap the Purchase button. Nobody on an iPhone could, because of a plugin the customer did not know about. Without accessibility testing, no one would have known why those sales were not coming in.

An accessibility audit can bring a small business the same disciplined workflow testing that mature technology companies use. Start by writing down the workflows that matter most to your business, such as checking out, creating an account, booking an appointment, requesting a quote, or signing up for a newsletter. Break each one into steps, and define what success looks like.

Any code change can break one of those workflows. When they are written down and tested, your team knows what success looks like, progress becomes measurable, and regressions are easier to catch. Accessibility stops being an isolated specialty and becomes part of operational quality.

Measure effort, not only presence: the number of steps, Tab presses to the primary action, time to complete the task, errors, and backtracking. The more steps a task requires, the more friction you have introduced. A screen reader user may legitimately take longer than a mouse user, so the goal is not identical times. The goal is to find unnecessary friction and remove it.

What to do this month

I closed the talk with three of these steps: run simple self-tests, watch someone who uses assistive technology, and write testing into your vendor contract. The other three round out what any business can do in the next few weeks.

1. Run an automated scan and fix what it finds. A free tool such as Microsoft Accessibility Insights is a good first pass. Zero automated findings is the starting line, not proof of accessibility.
2. Run simple self-tests on your top three pages. Pick the three pages that get the most traffic or the most complaints. Zoom in to 200 percent, try voice control by saying "Click" and a link's name, and turn on a screen reader to hear your image descriptions and error messages. You will find something. Everyone does.
3. Use your own site with only a keyboard, and time one critical task. How many times do you press Tab before you can do the thing you came to do? Can you see where focus is? Can you reach and activate every control, and escape every menu and dialog? Do you get stuck or have to backtrack?
4. Watch someone who uses assistive technology complete a real task on your site. Not a demo. Ask them to buy something, book something, or contact you, and watch. You will learn more from watching than from any report.
5. Put an Accessibility link in your footer. Give people a clear way to tell you what is not working, and make sure somebody actually checks it and responds.
6. Write testing with real users into your website vendor contract. If testing is not in the contract, it is easy for it to get skipped.

If you are just starting out

If you are just starting out, two things help most.

First, choose your platform with accessibility in mind. Popular platforms such as Shopify and WordPress already provide accessible building blocks to start from. When you select a platform or a vendor, look for those building blocks. Customizing a platform can break accessibility, which is one more reason to test.

Second, try the keyboard test. It requires no special knowledge. Do not touch your mouse, and try to do everything a customer needs to do on your site. Getting through that test sets you up well for much of the other accessibility work.

Go beyond the basics with people who use assistive technology

Once the basics are in place, add a second question to every review: can a person actually complete the task?

• Include people with disabilities in testing. An expert testing with NVDA, JAWS, VoiceOver, TalkBack, or voice control is valuable. A person who depends on those tools every day brings learned strategies, expectations, and priorities that an auditor does not experience the same way. Compensate participants fairly for their expertise.
• Follow established interaction patterns. When a component uses an ARIA role, it should behave as the ARIA Authoring Practices Guide describes.
• Write for comprehension. Treat alt text, labels, instructions, and error messages as content that deserves editorial review, not as fields to populate.
• Train the whole team. Designers, developers, content authors, and product managers each make decisions that widen or narrow the gap.

An AI code scan does not replace human testing

AI makes it easier to scan websites and send accessibility demand letters at scale. I think it works in the other direction too. It is very easy for a business to receive a letter, run an AI tool, and conclude that everything is fixed because the code scan now passes. AI coding tools will increasingly fix syntax and common technical violations, and that can speed up remediation.

But AI does not know what your brand is trying to communicate, which information matters most to your customers, whether a task has too many steps, or whether a real person trusts the experience.

AI may become good at checking boxes. But even when a scan says your site is passing, what matters is whether a person with a disability can sit down and use it. I am concerned about overreliance on, and hype around, AI as a fix for accessibility. When we stop thinking about why we are making design decisions, the result is less likely to work for people. As tools solve more of the technical checks, human judgment becomes more important, not less.

Conformance is the baseline

Automated testing is a useful starting point. It is not the finish line. Each of the six examples in this post can get past a scan, and every one of them can cost a business customers.

WCAG conformance tells you that the building blocks are in place. It does not tell you whether the building is livable. Accessibility is the discipline of designing beyond yourself: testing with people outside yourself, and listening to how other people process information, navigate, and make decisions.

If your team has recently completed an audit, treat it as a foundation. Choose one critical task, test it with people who use assistive technology every day, and listen closely to what they tell you. It is a way of finding friction before your customers simply disappear.

If you would like help testing your site with people who use assistive technology, contact me at [email protected]. You are also welcome to join us at the monthly A11yNYC (Accessibility New York City) meetup.