What Automated A11y Tests Miss

Adding accessibility (a11y) checks to CI makes it tempting to relax once the checkmark turns green. But automated tools are generally understood to catch only a portion of all accessibility problems. This article sorts out, for the implementation and testing stages, what to leave to machines and what a person should look at.

Think of automated checks in three layers

Each layer catches a different kind of problem.

• Static checks (while you write): for example eslint-plugin-jsx-a11y. It reads source code only, so it runs without starting the app.
• Runtime checks (after rendering): for example @axe-core/playwright. It inspects the real DOM, so it can also catch color contrast and duplicate IDs.
• Manual checks (a person operates the page): keyboard use and screen reader output, which machines cannot judge.

What automation catches and what it misses

Usually caught

• Images with no alt attribute
• Form fields with no associated label
• Text and background contrast below the threshold
• Misspelled or invalid aria-* attributes
• Buttons and links with no accessible name

Usually missed

• Whether the alt text actually describes the image (a tool will accept "image1")
• Whether the Tab order matches the visual reading order
• Whether focus returns to where it was after a modal closes
• Whether an error message is announced to screen reader users
• Whether some action can only be done with a mouse

In short, a machine can check that an attribute exists, but only a person can confirm the page is usable.

A minimum manual check (about 5 minutes)

Whenever you change a screen, a person should do at least these four things.

1. Put the mouse aside and use only the Tab key. Check that every field and button is reachable and that the focus outline is visible.
2. Open and close a modal or menu. Check that Esc closes it and that focus goes back to the button that opened it.
3. Zoom the browser to 200%. Check that text is not cut off and that no horizontal scrolling is needed.
4. Let a screen reader read one screen. On macOS, use VoiceOver (Command+F5) and confirm that button names and error messages are announced.

An example for CI

For runtime checks, a short test per key screen is a good start.

import AxeBuilder from '@axe-core/playwright';

test('contact page has no a11y violations', async ({ page }) => {
  await page.goto('/en/contact');
  const results = await new AxeBuilder({ page }).analyze();
  expect(results.violations).toEqual([]);
});

Do not start with every screen. Adopt the rule that every new screen must pass, and you can keep going without being blocked by old violations.

Common failures

• Treating green as done: the four manual checks get skipped. Add them to your pull request checklist.
• Muting violations in bulk: nobody looks at them later. If you must ignore one, leave the reason and a deadline in a comment.
• Reports never reach developers: when a tester notices a problem, "the screen reader sounds wrong" alone cannot be reproduced.

Putting it into practice with Bugoon

Problems found in manual checks fade from memory unless they are reported on the spot. Embed the Bugoon widget in your site and testers can mark up a screenshot right on the page where they noticed the problem. The operation steps are recorded too, so you keep a clue such as which key press made focus disappear. Reports link to GitHub Issues, and you can follow their status on the kanban board.

If contrast ratios and focus state could also be captured along with a report, it might become easier for non-engineers to communicate accessibility problems.