UX RESEARCH CASE STUDY · 2026

AI Tool Evaluation for Heuristic Analysis & Usability Testing

AI Tool Evaluation for Heuristic Analysis & Usability Testing

Where can AI meaningfully support evaluative UX research—and where does human judgment remain essential?

Where can AI meaningfully support evaluative UX research—and where does human judgment remain essential?

ROLE

UX Research Intern

TIMELINE

Summer 2026

METHODS

Interviews, Heuristic Evaluation, Usability Testing Analysis, AI Benchmarking

SCOPE

4 AI tools, 3 task flows, 2 user perspectives, 24 evaluation conditions

THE CHALLENGE

AI promises speed. Research requires trust.

AI promises speed. Research requires trust.

Before testing the tools, I spoke with four UX researchers to understand how AI was already showing up in their workflows.

Researchers were using AI to reduce effort in tasks such as drafting discussion guides, organizing notes, refining writing, and creating first-pass outputs.

But evaluative research raised a different set of concerns. Researchers were hesitant to rely on AI to understand context, recognize emotional or behavioral nuance, prioritize severity, account for edge cases, or make final decisions about whether a product was ready.

LARGE TAKEAWAY

How accurately can AI identify usability issues—and what happens when evaluation requires context, interpretation, and judgment?

How accurately can AI identify usability issues—and what happens when evaluation requires context, interpretation, and judgment?

RESEARCH APPROACH

AI should be evaluated against human research, not in isolation.

AI should be evaluated against human research, not in isolation.

I designed the study around one principle: AI should be evaluated against a human baseline rather than judged on its outputs alone.

I designed the study around one principle: AI should be evaluated against a human baseline rather than judged on its outputs alone.

01

Understand

Learn the product and domain, review existing research, and interview UX researchers about current AI workflows and concerns.

02

Establish a human baseline

Conduct a heuristic evaluation and consolidate existing usability-testing findings to create a set of human-validated issues.

03

Evaluate AI

Give multiple AI tools consistent product context, tasks, interfaces, and evaluation instructions.

04

Compare

Map every AI-generated finding back to the human baseline and evaluate what the tools identified correctly, incorrectly, and not at all.

HUMAN BASELINE

Before evaluating AI, I evaluated the experience without it.

Before evaluating AI, I evaluated the experience without it.

For heuristic evaluation, I reviewed the CGW portal using Nielsen’s 10 Usability Heuristics, focusing on navigation and structure. I documented usability issues, severity, and recommendations, then reviewed the findings with the product researcher to validate my interpretation.

For heuristic evaluation, I reviewed the CGW portal using Nielsen’s 10 Usability Heuristics, focusing on navigation and structure. I documented usability issues, severity, and recommendations, then reviewed the findings with the product researcher to validate my interpretation.

For usability testing, I reviewed existing findings across three flows:

For usability testing, I reviewed existing findings across three flows:

Onboarding → Prework → Command

Onboarding → Prework → Command

These flows progressed from familiar interaction patterns to more context-dependent technical work, allowing me to compare how AI performed as task complexity increased.

These flows progressed from familiar interaction patterns to more context-dependent technical work, allowing me to compare how AI performed as task complexity increased.

I consolidated findings from existing usability testing materials, including spreadsheet trackers, reports, and FigJam notes, to create the human baseline.

I consolidated findings from existing usability testing materials, including spreadsheet trackers, reports, and FigJam notes, to create the human baseline.

LARGE TAKEAWAY

This baseline became the reference point for determining what AI identified correctly, what it missed, and what it incorrectly flagged.

This baseline became the reference point for determining what AI identified correctly, what it missed, and what it incorrectly flagged.

EXPERIMENT DESIGN

4 AI tools × 3 task flows × 2 user perspectives = 24 evaluation conditions

NotebookLM

Gemini

TIADA

Claude Code

NOTEBOOKLM

58.3% F1

14 real issues identified · 13 false alarms · 7 real issues missed

Even the strongest tool produced almost as much noise as valid evidence.

ANALYSIS

Not every AI finding is actually a good finding.

I compared every AI-generated finding against the human baseline and classified it as a True Positive, False Positive, or False Negative.

AI identified a genuine usability issue that should be fixed.

AI FINDING

“I expected a way to dismiss the welcome message or skip the tour so I could immediately start my work.”

HUMAN FINDING

Experienced users may become frustrated by repetitive instructional content and longer task-completion times.

✓ VALID FINDING

AI identified an issue that was also supported by human usability research.

ANALYSIS

Not every AI finding is actually a good finding.

I compared every AI-generated finding against the human baseline and classified it as a True Positive, False Positive, or False Negative.

AI identified a genuine usability issue that should be fixed.

AI FINDING

“I expected a way to dismiss the welcome message or skip the tour so I could immediately start my work.”

HUMAN FINDING

Experienced users may become frustrated by repetitive instructional content and longer task-completion times.

✓ VALID FINDING

AI identified an issue that was also supported by human usability research.

FINDING 01

AI was better at reading screens than understanding systems.

Across usability testing, AI repeatedly identified issues that were directly visible in the interface, including ambiguous labels, missing controls, insufficient system feedback, and small inconsistencies.

But it struggled with issues requiring an understanding of navigation across the product, information architecture, mental models spanning multiple screens, hidden system behavior, and design rationale.

AI FREQUENTLY IDENTIFIED

Ambiguous labels and terminology
Missing back or skip options
Progress indicators and insufficient feedback
Visually disconnected information

AI RARELY IDENTIFIED

Navigation across the product
Information architecture flaws
Mental models spanning multiple screens

AI could inspect what was visible. It struggled to infer what existed beyond the screen.

FINDING 02

The more context a task required, the less reliable AI became.

AI performed best on the onboarding flow, which used more familiar and predictable interaction patterns.

Performance dropped on Prework and Command, where the tasks required more technical knowledge, hidden system context, and understanding of previous steps in the workflow.

Without that context, AI sometimes confused intentional product behavior with usability problems.

Onboarding → Prework → Command

Onboarding

Familiar, linear interaction patterns

Most reliable AI performance

Prework

More technical and context-dependent

More incorrect assumptions

Command

Required hidden system and cross-flow context

AI struggled most to distinguish normal behavior from actual usability issues

Increasing complexity did not just create more misses. It created more false confidence.

FINDING 03

AI could identify a problem without reliably knowing how much it mattered.

Different AI tools often agreed that a usability issue existed, but assigned very different severity ratings to the same finding.

One issue, for example, was rated 2 by one tool, “Low” by another, and 5 by another.

AI could explain why an issue might matter, but it could not reliably determine the level of user impact.

SAME ISSUE

Claude Code — 2

TIADA — Low

NotebookLM — 5

Severity ratings reflected each tool’s interpretation rather than a consistent measure of user impact.

Severity remained a research judgment, not just another field for AI to generate.

FINDING 04

AI’s distinctive value was breadth and consistency.

AI was particularly useful for repetitive, detail-oriented inspection.

AI WAS GOOD AT:

Reading every label

Comparing details across screens

Identifying inconsistencies

Catching small defects designers may overlook

Up to ~24% more validated findings

In heuristic evaluation, combining the researcher’s review with an additional AI pass expanded the validated finding pool.

Claude surfaced 5 validated issues beyond the original evaluation, increasing the validated pool from 21 to 26 issues.

The gain varied substantially by tool—from no increase to roughly 24%.

AI worked best as a second set of eyes—not as the final decision-maker.

Human validation was still necessary before AI-generated issues could be included.

METHOD COMPARISON

Rules gave AI boundaries. Human behavior introduced uncertainty.

Heuristic evaluation and usability testing produced different AI performance patterns.

HEURISTIC EVALUATION

Higher precision

AI performed better when verifying explicit usability rules.

Lower recall

The tools still missed subtle, context-heavy violations involving interaction, navigation, state changes, and information architecture.

Key implication: AI became more selective, but selectivity created blind spots.

USABILITY TESTING

Higher recall

When asked to analyze task execution, AI cast a wider net and identified more potential friction points.

Lower precision

Trying to predict human behavior also created substantially more false alarms.

Key implication: Simulating behavior increased coverage, but also increased noise.

AI was more reliable when evaluating explicit rules than when predicting how real users would behave.

RECOMMENDATION

AI should extend researcher judgment—not replace it.

The findings shifted the question from “Can AI conduct usability evaluation?” to “Which parts of evaluation are appropriate to delegate to AI?”

USE AI TO EXTEND

Consistency checking · Visible, rule-based violations · Detailed interface scanning · First-pass review · Expanding the evaluation surface · Generating candidate issues for researchers to investigate

KEEP RESEARCHERS RESPONSIBLE FOR

Understanding context · Interpreting behavior · Connecting interactions across a system · Evaluating design rationale · Assessing severity · Prioritizing findings · Validating evidence · Making final research judgments

AI became most useful when its role was bounded and every output remained subject to human validation.

OUTCOME

Translating the research into an AI-assisted evaluation framework.

I translated the findings into a framework for AI-assisted heuristic evaluation.

01 When AI adds value—and when it should not be used · 02 Which parts of heuristic evaluation can be supported by AI · 03 The minimum context and input AI tools should receive · 04 A recommended prompt and output structure · 05 Guidance for selecting the appropriate model or tool

The goal was not to automate the researcher. It was to create a repeatable way to use AI within clear research guardrails.

REFLECTION

How I grew through this project

Gaining confidence in my judgment — I became more comfortable questioning AI outputs, defending research decisions, and distinguishing evidence from assumptions.

Sharing work before it was complete — Early conversations helped me validate findings, understand stakeholder needs, and strengthen the direction before the final analysis.

Discernment and storytelling — I became better at moving beyond presenting everything I found to identifying the most meaningful insights and building a clear research story.

Evaluating AI is not only about measuring what it can produce. It is about deciding what we should trust it to do.