fix(atomic-a11y): resolve criterion conformance from axe-core results by y-lakhdar · Pull Request #7686 · coveo/ui-kit

y-lakhdar · 2026-05-28T14:47:59Z

Problem

The a11y reporter marked all WCAG criteria as notEvaluated in a11y-report.json — even when axe-core clearly detected violations or clean passes. Two bugs in the downstream OpenACR pipeline made things worse:

buildCriterionAggregates() treated affectedComponents as the "violating" set, but this field actually contained all components with coverage (passes included) — so every component was incorrectly flagged as violating.
The pipeline ran without a11y-report.json on disk, producing placeholder not-evaluated for every criterion.

Solution

Resolve conformance directly in the reporter using per-criterion violation/pass tracking:

axe-core signal	Resolved conformance
All covered components have violations	`doesNotSupport`
Some components violate, others pass	`partiallySupports`
No violations, explicit passes exist	`supports`
Only incomplete/inapplicable coverage	`notEvaluated`

Design decisions

incomplete = no conformance signal — contributes to coverage but doesn't resolve to pass or fail
axe-core pass = supports — even for criteria with known limited automated coverage (e.g., 2.1.1 Keyboard). Manual/interactive overrides correct later

https://coveord.atlassian.net/browse/KIT-5756

KIT-5756 The a11y reporter now tracks per-criterion violations and passes separately, resolving conformance directly in a11y-report.json instead of deferring to the OpenACR pipeline. Changes: - Add criteriaViolated/criteriaPassed to component automated results - Replace affectedComponents with coveredComponents/violatingComponents on criterion entries - Resolve conformance in report-builder (violations -> doesNotSupport, passes -> supports, partial -> partiallySupports) - Summary counters now reflect resolved conformance values - Merge-shards re-resolves conformance post-merge - Simplified OpenACR buildCriterionAggregates to read directly from report criterion fields

changeset-bot · 2026-05-28T14:48:07Z

⚠️ No Changeset found

Latest commit: ecdd33d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

svcsnykcoveo · 2026-05-28T14:48:15Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9b6b3edaad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-05-28T14:51:34Z

+
+  // No violations — check if any component has an explicit pass for this criterion
+  // (not just incomplete/inapplicable coverage)
+  return 'supports';


Require explicit passes before reporting supports

For a criterion that appears only via axeResults.incomplete or axeResults.inapplicable, collectCriteria adds it to criteriaCovered but not criteriaPassed; this resolver never reads criteriaPassed, so it reports supports whenever there are no violations. That contradicts the intended notEvaluated case and inflates summaries/OpenACR for rules Axe could not actually pass; please require at least one explicit pass before returning supports.

Useful? React with 👍 / 👎.

chromatic-com · 2026-05-28T14:55:07Z

Important

Testing in progress…

🟢 UI Tests: 367 tests unchanged
⏳ UI Review: Comparing 367 stories…
Storybook Publish: 367 stories published

chromatic-com · 2026-05-28T14:55:08Z

Tip

All tests passed and all changes approved!

🟢 UI Tests: 367 tests unchanged
🟢 UI Review: 367 stories published -- no changes
Storybook Publish: 367 stories published

pkg-pr-new · 2026-05-28T14:55:30Z

coveo-atomic-pkg-new-template

@coveo/atomic

npm i https://pkg.pr.new/@coveo/atomic@7686

@coveo/atomic-hosted-page

npm i https://pkg.pr.new/@coveo/atomic-hosted-page@7686

@coveo/atomic-legacy

npm i https://pkg.pr.new/@coveo/atomic-legacy@7686

@coveo/atomic-react

npm i https://pkg.pr.new/@coveo/atomic-react@7686

@coveo/auth

npm i https://pkg.pr.new/@coveo/auth@7686

@coveo/bueno

npm i https://pkg.pr.new/@coveo/bueno@7686

@coveo/create-atomic

npm i https://pkg.pr.new/@coveo/create-atomic@7686

@coveo/create-atomic-component

npm i https://pkg.pr.new/@coveo/create-atomic-component@7686

@coveo/create-atomic-component-project

npm i https://pkg.pr.new/@coveo/create-atomic-component-project@7686

@coveo/create-atomic-result-component

npm i https://pkg.pr.new/@coveo/create-atomic-result-component@7686

@coveo/create-atomic-rollup-plugin

npm i https://pkg.pr.new/@coveo/create-atomic-rollup-plugin@7686

@coveo/headless

npm i https://pkg.pr.new/@coveo/headless@7686

@coveo/headless-react

npm i https://pkg.pr.new/@coveo/headless-react@7686

@coveo/shopify

npm i https://pkg.pr.new/@coveo/shopify@7686

commit: ecdd33d

Copilot

Pull request overview

Fixes the a11y reporter so that WCAG criterion conformance is resolved at report-build time (and during shard merge) from per-criterion violation vs. pass tracking, instead of being deferred and always emitted as notEvaluated. Per-component automated results now carry criteriaViolated and criteriaPassed sets, and the criterion-level shape replaces the ambiguous affectedComponents with explicit coveredComponents / violatingComponents. The OpenACR aggregate builder is simplified to read straight from the merged criterion data.

Changes:

Extend A11yAutomatedResults and ComponentAccumulator with criteriaViolated / criteriaPassed; have VitestA11yReporter.collectCriteria route axe buckets into them.
Replace A11yCriterionReport.affectedComponents with coveredComponents + violatingComponents, resolve conformance in report-builder.ts and merge-shards.ts, and populate summary counts accordingly.
Simplify OpenACR buildCriterionAggregates to derive aggregates directly from criterion-level data, and update tests accordingly.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
packages/atomic-a11y/src/shared/types.ts	Adds `criteriaViolated`/`criteriaPassed` and renames `affectedComponents` to `coveredComponents`/`violatingComponents`.
packages/atomic-a11y/src/reporter/reporter-utils.ts	Adds violated/passed sets to the component accumulator.
packages/atomic-a11y/src/reporter/vitest-a11y-reporter.ts	Tags collected criteria as violated or passed and seeds the new sets.
packages/atomic-a11y/src/reporter/report-builder.ts	Tracks covered/violating components per criterion and resolves automated conformance.
packages/atomic-a11y/src/reporter/merge-shards.ts	Merges new component sets and re-resolves criterion conformance after merge.
packages/atomic-a11y/src/reporter/summary.ts	Computes `supports`/`partiallySupports`/`doesNotSupport`/`notApplicable` from criterion conformance.
packages/atomic-a11y/src/openacr/report-builder.ts	Builds aggregates directly from `criterion.coveredComponents` / `violatingComponents`.
packages/atomic-a11y/src/tests/merge-shards.test.ts	Updates fixtures for the new shape and asserts `supports` conformance.
packages/atomic-a11y/test/interactive-reporting.test.ts	Updates fixtures and assertions to the renamed fields.

+function resolveAutomatedConformance(
+  criterion: A11yCriterionReport
+): A11yCriterionReport['conformance'] {
+  const coveredCount = criterion.coveredComponents.length;
+  if (coveredCount === 0) {
+    return 'notEvaluated';
+  }
+
+  const violatingCount = criterion.violatingComponents.length;
+  if (violatingCount >= coveredCount) {
+    return 'doesNotSupport';
+  }
+  if (violatingCount > 0) {
+    return 'partiallySupports';
+  }
+
+  // No violations — check if any component has an explicit pass for this criterion
+  // (not just incomplete/inapplicable coverage)
+  return 'supports';
+}



 describe('interactive report aggregation', () => {
-  it('does not duplicate affected components covered by automated and interactive results', () => {
+  it('does not duplicate covered components covered by automated and interactive results', () => {


y-lakhdar temporarily deployed to PR Artifacts May 28, 2026 14:48 — with GitHub Actions Inactive

y-lakhdar temporarily deployed to pr-workflows May 28, 2026 14:48 — with GitHub Actions Inactive

y-lakhdar temporarily deployed to PR Artifacts May 28, 2026 14:48 — with GitHub Actions Inactive

chatgpt-codex-connector Bot reviewed May 28, 2026

View reviewed changes

Merge branch 'fix-open-acr-generation' into fix-rule-conformance

ecdd33d

y-lakhdar temporarily deployed to pr-workflows May 28, 2026 14:55 — with GitHub Actions Inactive

y-lakhdar temporarily deployed to PR Artifacts May 28, 2026 14:55 — with GitHub Actions Inactive

y-lakhdar changed the title ~~fix(atomic-a11y): resolve criterion conformance from axe-core results j:KIT-5756~~ fix(atomic-a11y): resolve criterion conformance from axe-core results May 29, 2026

y-lakhdar requested a review from Copilot May 29, 2026 18:55

Copilot started reviewing on behalf of y-lakhdar May 29, 2026 18:55 View session

Copilot AI reviewed May 29, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(atomic-a11y): resolve criterion conformance from axe-core results#7686

fix(atomic-a11y): resolve criterion conformance from axe-core results#7686
y-lakhdar wants to merge 2 commits into
fix-open-acr-generationfrom
fix-rule-conformance

y-lakhdar commented May 28, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented May 28, 2026 •

edited

Loading

Uh oh!

svcsnykcoveo commented May 28, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot May 28, 2026

Uh oh!

chromatic-com Bot commented May 28, 2026

Testing in progress…

Uh oh!

chromatic-com Bot commented May 28, 2026 •

edited

Loading

All tests passed and all changes approved!

Uh oh!

pkg-pr-new Bot commented May 28, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

y-lakhdar commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Design decisions

Uh oh!

changeset-bot Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

svcsnykcoveo commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 28, 2026

Choose a reason for hiding this comment

Uh oh!

chromatic-com Bot commented May 28, 2026

Testing in progress…

Uh oh!

chromatic-com Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

All tests passed and all changes approved!

Uh oh!

pkg-pr-new Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

y-lakhdar commented May 28, 2026 •

edited

Loading

changeset-bot Bot commented May 28, 2026 •

edited

Loading

svcsnykcoveo commented May 28, 2026 •

edited

Loading

chromatic-com Bot commented May 28, 2026 •

edited

Loading

pkg-pr-new Bot commented May 28, 2026 •

edited

Loading