Accessibility Testing: What axe-core Cannot See

I worked on a customer portal where consumers signed up for mobile and internet subscriptions. WCAG conformance was in the requirements. It was written down, agreed, signed. Nobody on the project had any way to demonstrate it.

A requirement that exists on paper and nowhere in the pipeline is one you find out about at the worst possible moment, usually when somebody outside the team asks for evidence. And this was a consumer telco portal. The share of your users who need an accessible interface is roughly the share of the population that needs one, and a self-service portal for mobile and internet sits inside the European Accessibility Act's scope.

So you add a gate. Axe-core in CI, on every pull request, failing the build on violations. That is the right first move, and this article shows how to wire it up against a real user journey. It is also where most teams stop. The number people quote to justify stopping there does not mean what they think it means.

The Proofreader Who Checks Spelling, Not Meaning

Every draft gets a proofreading pass before it ships. It is fast, mechanical, and applied to every page without exception. It catches misspellings, missing punctuation, a doubled word. Errors with a fixed, checkable shape that a rule can be written against. What it does not catch is whether the piece says what it means. A wrong argument, a confusing structure, a claim that does not follow from the one before it. All grammatically flawless. All invisible to a spelling pass, because a spelling pass was never built to look for that category.

Axe-core is that proofreader. Missing alt text, thin contrast ratios, unlabelled form controls, ARIA used wrongly: structural defects with a fixed shape. Then there is the other category. Focus order that traps a keyboard user. A screen reader announcing controls in a senseless sequence. Whether the page actually works for the person sitting in front of it. None of that has a fixed shape, so none of it is detected.

A proofreader catches nearly every misspelling and almost none of "this doesn't say what it means." Axe-core sits in roughly the same place.

Push the analogy further and it starts to flatter axe-core. A proofreader is a person who could read for meaning if you asked. Axe-core cannot, no matter how you ask. And a proofreader's spelling pass is close to perfectly precise, where axe-core's has real false positives and a catch rate that moves with how you configure the rules. That last part is why nobody agrees on the number.

Why This Landed on Your Backlog This Year

The deadline passed, and it came with a penalty clause.

The European Accessibility Act became enforceable on 28 June 2025 for new products and services, with existing ones having until 28 June 2030. It covers e-commerce, banking, transport, telecoms, e-books and SaaS sold into the EU regardless of where the vendor is based. Penalties range from EUR 5,000 in Estonia to EUR 500,000 in Germany, or up to 4% of annual revenue. The first lawsuits were filed in French Commercial Court in November 2025.

In the US, the DOJ's ADA Title II final rule sets WCAG 2.1 AA for state and local government digital services. An April 2026 Interim Final Rule pushed the deadlines out by a year, to 26 April 2027 for entities serving 50,000 or more people and 26 April 2028 for smaller ones. The standard itself did not change. WCAG 2.2 remains the operative W3C Recommendation and the compliance baseline; WCAG 3.0 is still a Working Draft and is not expected to be final before 2028, so build nothing on it.

Under that pressure a team does the right thing first and wires in the automated pass. Almost nobody skips the gate. They stop at it, because testing every journey with a screen reader does not scale to a fixed date the way a CI job does. The more reliable the automated pass looks, the easier it is to make that swap without anyone deciding to.

The Coverage Number Nobody Agrees On

Three figures circulate, and all three are defensible.

Figure	Source	What it counts
~57%	axe-core's own repository, from Deque's coverage study of 2,000+ audits, 13,000+ pages and ~300,000 issues	Issue volume: what share of individual defects found in real audits axe-core flags
80%	Deque's DevTools marketing page	Broader tooling claim, unstated denominator
20-30%	Widely repeated industry figure	WCAG success criteria: what share of the numbered requirements can be machine-checked at all

They do not contradict each other, but use different denominators. Issue volume is dominated by defects that repeat across thousands of pages, and repetitive defects are exactly the ones with a fixed structural shape. Success criteria weight "images have alt text" the same as "the reading sequence is meaningful", so the moment you count requirements instead of instances, automated coverage collapses.

So a coverage number quoted without its denominator tells you very little. It is the same arithmetic that makes an 80% code coverage dashboard feel like proof of something, a problem I have written about on mutation testing.

Getting Your Hands Dirty

We're now going to build a working accessibility gate: a Playwright suite that walks a signup journey, runs axe-core at each state the user actually passes through, and fails a pull request when it finds a violation nobody has signed off on. We gate a journey rather than a page, because nobody has checked the states a scanner never reaches.

Prerequisites

Two things, both worth getting right first, because the failure modes later are confusing rather than obvious.

Node 18 or later. Check what you have:

node --version

If that prints nothing, or a version below 18, install a current release: brew install node on macOS, winget install OpenJS.NodeJS.LTS on Windows, and NodeSource rather than apt install nodejs on Debian or Ubuntu, where the distribution packages lag badly:

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
sudo apt-get install -y nodejs

An application under test, serving on http://localhost:3000 with a route at /signup. The gate has to scan something. If the app you care about already runs locally, start it and skip ahead.

If not, here is one. It is a single file with no dependencies beyond the Node you just installed, and it is deliberately imperfect: four defects are planted in it, spread across the three states the journey visits. A clean page teaches you nothing here, because you never get to read a report.

It lives in the same project as the gate, under app/, so there is one package.json and one thing for CI to check out. Create that project now; Step 1 installs the tooling into it:

mkdir a11y-gate && cd a11y-gate
npm init -y

Save this as app/server.js:

// app/server.js
// A deliberately imperfect signup flow. Three states, four planted defects.
const http = require('http');

const page = (title, body) => `<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>${title}</title>
  <style>
    body { font-family: system-ui, sans-serif; margin: 0; background: #fff; color: #1a1a1a; }
    main { max-width: 32rem; margin: 0 auto; padding: 2rem 1rem; }
    label { display: block; margin-bottom: .25rem; font-weight: 600; }
    input { width: 100%; padding: .5rem; font-size: 1rem; border: 1px solid #767676; }
    button { margin-top: 1rem; padding: .6rem 1.2rem; font-size: 1rem;
             background: #1a4fa0; color: #fff; border: 0; cursor: pointer; }
    .error { color: #d00; margin-top: .5rem; }
    /* Planted defect 1: 2.1:1 contrast, needs 4.5:1 (WCAG 1.4.3). */
    #legacy-promo-banner { background: #f0f0f0; color: #a6a6a6; padding: .75rem 1rem; }
  </style>
</head>
<body>
  <div id="legacy-promo-banner">Save 20% when you bundle mobile and internet.</div>
  <main>${body}</main>
</body>
</html>`;

// Planted defect 2: an <img> with no alt attribute (WCAG 1.1.1).
// Planted defect 3: the heading level jumps from h1 to h3. Axe-core tags
// heading-order as best-practice, not WCAG, so the tag filter in Step 3 skips it.
const landing = page('Sign up', `
  <h1>Create your account</h1>
  <img src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" width="120" height="40">
  <h3>Step 1 of 2</h3>
  <form method="POST" action="/signup">
    <label for="email">Email address</label>
    <input id="email" name="email" type="email">
    <button type="submit">Continue</button>
  </form>`);

// Planted defect 4: the error is announced by colour alone and is not in a
// live region, so a screen reader user never hears it (WCAG 1.4.1, 4.1.3).
// No rule detects this one.
const withError = page('Sign up', `
  <h1>Create your account</h1>
  <img src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" width="120" height="40">
  <h3>Step 1 of 2</h3>
  <form method="POST" action="/signup">
    <label for="email">Email address</label>
    <input id="email" name="email" type="email">
    <p class="error">Enter an email address.</p>
    <button type="submit">Continue</button>
  </form>`);

const stepTwo = page('Choose a plan', `
  <h1>Choose a plan</h1>
  <h3>Step 2 of 2</h3>
  <form method="POST" action="/done">
    <label for="plan">Plan</label>
    <input id="plan" name="plan" type="text" value="Mobile 10GB">
    <button type="submit">Continue</button>
  </form>`);

const send = (res, html) => {
  res.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' });
  res.end(html);
};

const server = http.createServer((req, res) => {
  const url = new URL(req.url, 'http://localhost:3000');

  if (req.method === 'GET' && (url.pathname === '/' || url.pathname === '/signup')) {
    return send(res, landing);
  }

  if (req.method === 'POST' && url.pathname === '/signup') {
    let body = '';
    req.on('data', (chunk) => (body += chunk));
    return req.on('end', () => {
      const email = new URLSearchParams(body).get('email');
      send(res, email ? stepTwo : withError);
    });
  }

  if (req.method === 'POST' && url.pathname === '/done') {
    return send(res, page('Done', '<h1>You are signed up</h1>'));
  }

  res.writeHead(404, { 'Content-Type': 'text/plain' });
  res.end('Not found');
});

server.listen(3000, () => console.log('Listening on http://localhost:3000'));

Start it from the project root:

node app/server.js

The three states match the journey in Step 3. A GET of /signup is the landing state, submitting empty re-renders it with the error that a static scan never reaches, and submitting with an email address moves to step two.

Four defects are planted, and the gate sorts them into three groups. The landing state reports exactly two:

[serious] color-contrast: Elements must meet minimum color contrast ratio thresholds
   at #legacy-promo-banner
[critical] image-alt: Images must have alternative text
   at img

Those are the structural ones, caught every time.

The third is the heading jump from h1 to h3. Axe-core has a rule for it, heading-order, but the rule is tagged best-practice rather than wcag2a, so the tag filter in Step 3 never runs it. The defect is real, the engine knows how to find it, and your configuration decided it did not count. That is the "catch rate moves with how you configure the rules" problem from earlier in this article, sitting in your own terminal rather than in a coverage study.

The fourth is the validation error, conveyed by colour alone and never announced. No tag filter brings that one back, because no rule has a fixed shape to check it against. A keyboard user submits the form, the page changes, and nothing is said. You find it by pressing Tab and listening, which is the manual check the closing section asks you to give an owner.

The banner carries the #legacy-promo-banner id, the selector the allowlist in Step 4 suppresses, so you can watch a deferred violation drop out of the report and come back when its expiry passes.

Leave the app running in its own terminal. Everything below happens in a second one.

Step 1: Install the tooling

From the a11y-gate directory, install the test runner, the axe integration, the engine itself, and the browsers Playwright drives. Bringing your own application instead of the demo one? Run mkdir a11y-gate && cd a11y-gate && npm init -y first.

npm install --save-dev @playwright/test @axe-core/playwright axe-core
npx playwright install --with-deps chromium

The browser download takes a minute or two on first run, finishing with a line naming the Chromium build. Confirm the engine version, because rule behaviour changes between releases:

node -p "require('axe-core').version"

That prints 4.13.0 at the time of writing. axe-core, @axe-core/cli and @axe-core/playwright are versioned in lockstep, so pin all three together.

Step 2: Configure the runner

Create playwright.config.js in the project root. The baseURL is what lets your test navigate with relative paths, and webServer is optional if you start the app yourself.

// playwright.config.js
const { defineConfig } = require('@playwright/test');

module.exports = defineConfig({
  testDir: './tests',
  reporter: [['list'], ['json', { outputFile: 'a11y-results.json' }]],
  use: {
    baseURL: 'http://localhost:3000',
  },
});

Step 3: Scan a journey, not a page

This is the step that matters. A static scanner sees the landing page. Your users see a form with validation errors, or an expanded accordion, or a modal. Scan at each meaningful state.

// tests/signup-journey.spec.js
const { test, expect } = require('@playwright/test');
const AxeBuilder = require('@axe-core/playwright').default;

const scan = (page) =>
  new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa', 'wcag21aa', 'wcag22aa'])
    .analyze();

test('signup journey is free of detectable violations', async ({ page }) => {
  const findings = [];

  await page.goto('/signup');
  findings.push(['landing', await scan(page)]);

  // Submit empty to force the error state a static scan never reaches.
  await page.getByRole('button', { name: /continue/i }).click();
  findings.push(['validation errors', await scan(page)]);

  await page.getByLabel(/email/i).fill('test@example.com');
  await page.getByRole('button', { name: /continue/i }).click();
  findings.push(['step two', await scan(page)]);

  for (const [state, results] of findings) {
    expect(results.violations, `state: ${state}`).toEqual([]);
  }
});

Run it with npx playwright test. Against the demo app it fails, which is the point: there are planted defects to find.

Look at how it fails, though. Asserting on results.violations hands the whole axe object to the reporter, so the contrast violation alone prints about fifty-five lines of nested JSON. A page with a dozen violations buries what the developer needs: the rule, the element, the fix. Worse, the assertion sits inside the loop, so the first failing state throws and you never learn about the other two. Step 4 replaces that final for loop and leaves the rest of the file alone.

Step 4: Handle false positives honestly

Axe-core has real false positives, and a team that cannot suppress one will suppress the whole gate instead. Two mechanisms are enough: rule configuration for genuine engine disagreements, and a triaged allowlist with an expiry date for known defects you have not fixed yet.

// tests/a11y-allowlist.js
// Every entry needs an owner and an expiry. Expired entries fail the build.
module.exports = [
  {
    rule: 'color-contrast',
    selector: '#legacy-promo-banner',
    reason: 'Third-party banner, vendor ticket VEN-4471',
    owner: 'web-platform',
    expires: '2026-12-01',
  },
];

Then apply it and let the calendar do the enforcement. The same module also formats the output, since a gate nobody can read is a gate somebody eventually switches off:

// tests/a11y-report.js
const allowlist = require('./a11y-allowlist');

const today = () => new Date().toISOString().slice(0, 10);

// A missing expiry is the failure the allowlist exists to prevent, so reject it
// at load time rather than treating it as a permanent exemption.
for (const entry of allowlist) {
  if (!entry.expires || !entry.owner) {
    throw new Error(
      `a11y allowlist entry for ${entry.rule} on ${entry.selector} needs both an owner and an expires date`
    );
  }
}

function filterKnown(violations) {
  const now = today();
  return violations.filter((v) =>
    !allowlist.some(
      (a) =>
        a.expires >= now &&
        a.rule === v.id &&
        v.nodes.some((n) => n.target.join(' ').includes(a.selector))
    )
  );
}

function formatViolations(state, violations) {
  const lines = [`${state}: ${violations.length} violation(s)`];
  for (const v of violations) {
    lines.push(`  [${v.impact}] ${v.id}: ${v.help}`);
    for (const node of v.nodes) {
      lines.push(`    at ${node.target.join(' ')}`);
    }
    lines.push(`    ${v.helpUrl}`);
  }
  return lines.join('\n');
}

module.exports = { filterKnown, formatViolations };

The load-time check earns its place. In JavaScript, comparing a missing expires against today's date yields false, so an entry someone forgot to date does not fail loudly. It quietly becomes the permanent exemption the expiry column was meant to prevent.

None of that reaches the gate until the test uses it. Two changes to tests/signup-journey.spec.js from Step 3: one new require at the top, and the for loop at the bottom swapped for the one below. The journey itself does not change. Here is the whole file afterwards:

// tests/signup-journey.spec.js
const { test, expect } = require('@playwright/test');
const AxeBuilder = require('@axe-core/playwright').default;
const { filterKnown, formatViolations } = require('./a11y-report');  // added

const scan = (page) =>
  new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa', 'wcag21aa', 'wcag22aa'])
    .analyze();

test('signup journey is free of detectable violations', async ({ page }) => {
  const findings = [];

  await page.goto('/signup');
  findings.push(['landing', await scan(page)]);

  // Submit empty to force the error state a static scan never reaches.
  await page.getByRole('button', { name: /continue/i }).click();
  findings.push(['validation errors', await scan(page)]);

  await page.getByLabel(/email/i).fill('test@example.com');
  await page.getByRole('button', { name: /continue/i }).click();
  findings.push(['step two', await scan(page)]);

  // Replaces the expect() loop from Step 3.
  const failures = [];
  for (const [state, results] of findings) {
    const violations = filterKnown(results.violations);
    if (violations.length) failures.push(formatViolations(state, violations));
  }

  expect(failures.join('\n'), 'accessibility violations').toBe('');
});

Collecting into failures and asserting once at the end is what reports all three states in a single run, rather than stopping at the first.

A failing run now names the state, the impact, the rule, the element and the page that explains the fix, in four lines per violation:

landing: 1 violation(s)
  [critical] image-alt: Images must have alternative text
    at img
    https://dequeuniversity.com/rules/axe/4.13/image-alt?application=playwright

The contrast violation is absent, suppressed by the allowlist, and returns on 1 December 2026 when the expiry passes. Defining filterKnown and forgetting to wire it in is worse than not having it at all: the build then fails on the exact violation the team just agreed to defer.

If you need to disable a rule engine-wide rather than per element, new AxeBuilder({ page }).disableRules(['color-contrast']) does it, and it should be rare enough to require a comment explaining why.

Step 5: Wire it into CI

A gate that does not block a merge is a report. Two prerequisites first, both run in a11y-gate alongside everything else, and both missing because npm init -y does not add them:

npm install --save-dev wait-on
npm pkg set scripts.start="node app/server.js"

The start script is what the workflow calls to boot the application under test. Here it points at the demo app from the Prerequisites; for your own application, put its start command there instead. Without it the workflow fails with Missing script: start, a confusing way to find out your pipeline was never going to run.

This assumes the gate and the application share a repository. If yours is a separate service, npm start becomes whatever stands your environment up and baseURL points at that rather than localhost.

# .github/workflows/accessibility.yml
name: Accessibility gate
on: [pull_request]

jobs:
  axe:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npm start & npx wait-on http-get://localhost:3000/signup
      - run: npx playwright test
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: a11y-results
          path: a11y-results.json

wait-on keeps the suite from racing the server. Note the http-get:// scheme and the path. Given a bare http://localhost:3000, wait-on sends a HEAD request to the root and waits for a 2xx, so an app serving nothing at / never satisfies it. The job then times out saying it could not reach the server, while the server runs perfectly the whole time.

That uploaded JSON is what the telco portal never had: a dated record, per commit, that you export rather than reconstruct.

Step 6: Tear down

Stop the server first, with Ctrl+C in the terminal it is running in. Then:

cd .. && rm -rf a11y-gate
npx playwright uninstall         # removes the browsers this project downloaded

Leave off --all unless you mean it. That flag removes browsers used by every Playwright installation on the machine, which is a rude thing to do to your other projects.

Gate Now, Name the Remainder

Add the gate. It costs a day, it catches a real recurring class of defect, and it keeps that class from regressing. The proofreader catches the same errors on every draft, forever, which is why you keep the pass even knowing what it cannot do.

Then write down what it cannot see and put a name next to each item.

A green pipeline means you may continue. It is not a conformance claim, and it should never be quoted as one in a tender response.
Keyboard-only traversal of each critical journey is a manual check. Give it an owner and a cadence.
Screen reader verification belongs in the definition of done for the flows that carry revenue.
Every allowlist entry has an expiry, because a suppression without a date is a decision nobody has to revisit.
Quote coverage numbers with their denominator, or do not quote them.

A remainder nobody has named is a remainder nobody owns. Most of the teams that get caught by the EAA will have had a scanner, and it will have been green. Their definition of done ended there.

Worth asking at your next planning session: if a regulator wanted evidence of conformance tomorrow, would you export it or reconstruct it? And who owns the part axe-core was never built to see?