Testing Claude Dashboards and Motion: Visual, A11y, and States

What Breaks Visually in Generated Dashboards

• Encoding drift. A regenerated chart quietly swaps a bar chart for a stacked bar, a linear axis for a log axis, or a zero-based axis for one that starts at 40%. Every number is still correct, but the chart now tells a different story.
• Color drift. "North America" is blue on one chart and orange on the next. A category reorder reassigns the palette.
• Overflow and collision. A new, longer category name truncates the legend, overlaps axis labels or pushes a tile off-screen at laptop width.
• Data-shape breakage. A null renders as a zero, a negative breaks a stacked area, a new 13th category falls off a 12-color palette, or an empty week renders a blank card with no message.
• Theme breakage. Light-mode colors carried into dark mode leave low-contrast text and invisible gridlines.
• Animation end-states. In Motion, the last frame should land exactly where the static chart would. Off-by-one timing leaves a bar at 92% of its height or a label mid-fade in the exported MP4.

How to Test It Without Drowning in False Positives

• Pin the data for visual tests. Run snapshot comparisons against a fixture dataset or a frozen query, not live production numbers. Live data belongs in outcome checks (surface 3), not pixel diffs.
• Mask what is meant to move. Refresh timestamps, "last updated" badges and live counters get masked or excluded from the comparison.
• Freeze motion before you capture. For browser snapshots, disable or settle animations first. Playwright's toHaveScreenshot({ animations: "disabled", mask: [...] }) does both. For Motion exports, compare specific frames (first, key beats, last) rather than whole videos.
• Snapshot the matrix, not one screen. Light and dark, plus at least desktop, laptop and mobile widths. Generated layouts fail at the breakpoints nobody looked at.
• Diff with tolerance, review with humans. Use a visual-diff tool with anti-aliasing and layout-shift tolerance (TestMu AI's SmartUI is built for this). Route real diffs to a person; a model shouldn't approve its own chart.
• Assert structure in the DOM as well as pixels. Many visual regressions are cheaper to catch structurally: the chart has 12 bars, the legend lists 4 series, the y-axis starts at 0, the refresh timestamp is present. That's where Kane CLI fits (section 6).

The principle: pixels for layout, structure for meaning, humans for judgment.

The WCAG Criteria Generated Dashboards Fail Most

The Animation-Specific Criteria

Claude Motion and animated dashboard transitions add a second set of obligations:

• 2.2.2 Pause, stop, hide. Anything that moves, blinks or scrolls automatically for more than five seconds, alongside other content, needs a way to pause it.
• 2.3.1 Three flashes. Nothing flashes more than three times in any one second. Fast count-up numbers and strobing highlights are the usual culprits.
• 2.3.3 Animation from interactions (AAA). Motion triggered by interaction can be turned off. In practice, respect the prefers-reduced-motion media query and test with it on.
• 1.2.1 Prerecorded audio-only and video-only, and the captions criteria. An exported MP4 that carries information needs a text alternative. If it has narration, it needs captions. A Motion explainer pasted into a slide deck or email loses everything the original version page could have provided, so ship the transcript with it.

How to Test It

• Automate the floor. Run an automated accessibility scan (an axe-based scanner or TestMu AI's accessibility testing) on every generated dashboard, in both themes. That catches missing names, contrast failures and broken ARIA. It won't catch the problems that need judgment, like a meaningless alt text or a confusing focus order.
• Test with the OS settings real users have. Reduced motion on, 200% zoom, forced colors or high contrast, mobile width.
• Walk it with a keyboard. Tab order, focus visibility, every filter operable, no keyboard traps in tooltips or modals.
• Spot-check with a screen reader. Does the chart announce a takeaway or just "graphic"? Does a filter change announce anything?
• Re-run on every regeneration. An accessible dashboard can become inaccessible after one "make it prettier" prompt.

Note: TestMu AI covers both scans for a generated dashboard: SmartUI compares screenshots while filtering anti-aliasing and dynamic-content noise, and the accessibility suite runs axe-core checks against WCAG 2.2. Create a free account to run them on your next dashboard.

The States to Cover

For Claude Motion, the states are temporal: first frame, each key beat, final frame, and the editable version after a one-word change. Anthropic says you can change a word, number or timing while the rest stays the same.^[3] That promise is exactly what a regression test should hold it to.

Setup

npm install -g @testmuai/kane-cli
npx @testmuai/kane-cli-skill        # lets Claude Code, Codex or Gemini CLI call Kane
kane-cli login --username "$LT_USERNAME" --access-key "$LT_ACCESS_KEY"

1. Loaded State: The Numbers Mean What They Say

kane-cli run "Open the Q3 revenue dashboard. Confirm it shows revenue by region for exactly \
  4 regions, displays a last-refreshed time, and that the Total Revenue tile equals \
  the sum of the four regions" \
  --url "$DASHBOARD_URL" --agent --headless --final-validation on

This catches the failure no screenshot diff can see: a correct-looking chart with the wrong aggregation.

2. Filter Propagation: Every Chart Responds

kane-cli run "Set the region filter to EMEA. Confirm every chart and the Total Revenue \
  tile now show EMEA only, and the active filter is visible on the page" \
  --url "$DASHBOARD_URL" --agent --headless

3. Empty State: No Data Is a State, Not a Crash

kane-cli run "Set the date range to a week with no orders. Confirm the dashboard shows \
  a clear no-data message and no chart shows NaN, empty axes or last week's numbers" \
  --url "$DASHBOARD_URL" --agent --headless

4. Error and Stale States

Point the same outcome at a staging dashboard whose connector credentials have been revoked:

kane-cli run "Open the dashboard. Confirm each chart that cannot load shows a readable \
  error with a way to retry, and no chart presents old data as current" \
  --url "$STAGING_DASHBOARD_URL" --agent --headless

5. Look Beyond the Stated Path

kane-cli run "Review the Q3 revenue dashboard as a finance user would" \
  --url "$DASHBOARD_URL" --agent --headless --bug-detection continue

Issues Kane notices along the way, like a truncated legend, a broken tile or a console error, are reported separately with a category and a confidence signal. They don't silently widen what the test checks.

6. Make It a Suite That Gates Every Regeneration

Save the outcomes as markdown test files in the repo, so a change to what "correct" means is reviewed in a PR:

kane-cli testrun run --tags dashboards --parallel 4 --on-failure fail-fast

Exit codes are 0 pass, 1 fail, 2 setup error and 3 timeout, so CI can block a dashboard from being shared until it passes.

7. Hand the Board Meeting an Answer, Not a Promise

Every run writes a sha256-hashed .evidence pack with every step, screenshot, DOM snapshot, console line and network call. Open it at evidence.lambdatest.com. When someone asks "was this chart checked before we used it?", you can show exactly what was checked and what the browser showed.

Where Kane CLI Sits Next to the Other Tools

• Kane CLI - interaction states, data meaning and structural checks, as plain-English outcomes, replayed on every change.
• Visual diffing (SmartUI or Playwright toHaveScreenshot) - pixel-level layout, color and theme regressions against fixture data.
• Accessibility scanning - automated WCAG checks on every generated view, plus human review.
• Frame comparisons - for Claude Motion MP4 exports, compare first, key and final frames. Kane CLI tests web and native mobile apps; it doesn't inspect video files.

Honest Limits

• Private dashboards need access. Claude Dashboards start private. To test one in CI, test it where it's shared or handed off (Hex, Sigma, Grafana and others), or wherever your team publishes it, with test credentials.
• Kane CLI doesn't inspect video. It tests web and native mobile apps in a real browser or emulator. For Motion MP4s, compare frames, and test the interactive version in the browser before export.
• Kane CLI is not a pixel-diff tool or a WCAG scanner. It checks outcomes and structure, and its evidence pack captures screenshots for review. Pair it with a visual-diff tool and an accessibility scanner. Don't ask one tool to do three jobs.
• Semantic drift is harder than moved elements. If a regenerated chart keeps its title but now plots a different metric, a structural check catches it only if your outcome names the metric. Write outcomes that say what the chart means, not just what it's called.
• Live data makes visual baselines meaningless. Pixel-compare against fixture data. Check live dashboards with outcome assertions that tolerate the numbers changing but not their meaning.

Note: Kane CLI is free to install, and local runs are free on a TestMu AI account. Start with the Kane CLI docs and claim 10,000 free Kane CLI credits for your first month.