Test Oracle Problem: How to Test AI With No Expected Output

What Is the Test Oracle Problem?

The test oracle problem is what Barr, Harman, McMinn, Shahbaz and Yoo's survey of the oracle problem in software testing (IEEE Transactions on Software Engineering, 2015) calls "the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior" for a given input. A test oracle is whatever makes that call, whether an expected value, a rule, a property or a person, and when no oracle can be automated, someone has to judge every result by hand.

The survey describes oracle automation as the way to remove "a current bottleneck that inhibits greater overall test automation," because without it "the human has to determine whether observed behaviour is correct."

Elaine Weyuker's 1982 paper On Testing Non-Testable Programs (The Computer Journal) defines a program as non-testable "if either an oracle does not exist or the tester must expend some extraordinary amount of time to determine whether or not the output is correct." Many AI features meet one or both conditions:

• No oracle exists - a summary, a ranked answer or a drafted reply has many acceptable versions, so there is no single string to compare against.
• Checking costs too much - a person can judge any one output, but not every output after every prompt, model or retrieval change.

What Are the Types of Test Oracles?

Barr et al. group oracles by where the expectation comes from. Testers fall back on derived oracles often, the survey notes, because specifications "rapidly fall out of date when they exist at all."

The ISTQB Certified Tester AI Testing syllabus v2.0 adds one reason the problem is harder for AI: system requirements "may evolve, be incomplete, or simply missing." Its section 4.1.3 lists these ways to address it:

• Output boundaries - agreed ranges, distributions, limits and tolerances, such as an autonomous car stopping within a maximum distance.
• Environmental boundaries - fixed test conditions such as lighting, temperature or network latency, so outputs are predictable and repeatable.
• Expert consultation - domain experts help define expected results.
• Specialized testing - A/B, back-to-back and metamorphic testing, which compare behaviors or verify properties "often without requiring explicit expected outputs for every case."
• Proxy oracles - secondary systems or models, including other AI systems, that assess outputs when direct expected results are unavailable.

Expert consultation is the human oracle under another name, and the syllabus warns that expert opinions "might differ or be fallible." Record who judged each case and on what basis, so a disagreement can be traced later.

How Do You Test AI Output With No Single Correct Answer?

Replace the one expected output with a list of acceptance criteria, then find the strongest oracle for each criterion separately. A single AI feature usually needs several oracle types, because its criteria differ in kind.

Without that step the call falls to human judgment, which is still common in release practice, and risk guidance flags the underlying difficulty:

• Release decisions - in Applause's State of Digital Quality in Testing AI 2026 survey, published in April 2026, 46.5% of software development, QA, data science, AI research and product management respondents said they "rely on human sentiment and usability to determine whether an AI feature is production ready."
• Risk frameworks - NIST's AI Risk Management Framework 1.0 (AI 100-1, Appendix B) lists "difficulty in performing regular AI-based software testing, or determining what to test" among the ways AI risks differ from traditional software risks.

Take a support feature that summarizes a customer thread and drafts a refund reply. Fill in a table like this one before writing any test code, picking the strongest source that can settle each criterion:

How to Narrow the Test Oracle Problem With Specs, Code and Tools

For an AI feature that calls tools, derived oracles can settle many criteria, because the code and its declarations already state what the feature may do and what a valid result looks like.

id: SC-014
feature_id: refund-request
class: functional
category: state_context
title: Ask for missing order and identity details before refunding
goal: >-
  Refund my last order. I do not have the order number with me.
input:
  kind: text
  attachments: []
expectation:
  acceptance_criteria:
    - id: AC-1
      statement: The agent asks for the order identifier.
      check: llm_judge
    - id: AC-2
      statement: The agent does not issue a refund before identity verification.
      check: mcp_probe
  forbidden:
    - claims the refund was completed without verification
  output_kind: text
  mcp:
    - server: billing
      tool: issue_refund
      expect: not_called
verification_requires:
  - type: mcp
    server: billing
    op: issue_refund
executable: true
skip_reason: null
repeat: 1
timeout_seconds: 120
multi_turn: true
setup_messages: []
max_turns: 4
tags: [refund, identity]

How to Check Properties Instead of Outputs

When no single output is right, a property of every right output often still holds. Property-based testing asserts that property over many generated inputs, and the front page of the Hypothesis documentation shows it on a sort function with no expected list anywhere:

from hypothesis import given, strategies as st

@given(st.lists(st.integers() | st.floats()))
def test_sort_correctness_using_properties(lst):
    result = my_sort(lst)
    assert set(lst) == set(result)
    assert all(a <= b for a, b in zip(result, result[1:]))

The test relies on properties alone:

• Same values - the output holds the same set of values as the input.
• Order - each element is no larger than the next.
• Generated inputs - Hypothesis chooses the inputs, including edge cases the test author might not have thought of, and the test never states a sorted result.

When one run has no checkable property, relate several runs instead. Chen et al.'s review of metamorphic testing (ACM Computing Surveys, 2018) defines metamorphic relations as "necessary properties of the target function or algorithm in relation to multiple inputs and their expected outputs." Checks across runs that need no written expected output include:

• Invariance test (INV) - from the CheckList paper (Ribeiro et al., ACL 2020): apply a label-preserving change to the input and expect the prediction to stay the same, which works on unlabeled data.
• Directional expectation test (DIR) - apply a change with a known effect and expect the prediction to move one way or not move, such as sentiment not becoming more positive after a negative phrase is added.
• Previous release - Barr et al. count other versions of the system as a source of derived oracles. Running the last release on the same inputs shows what changed, though something else still has to decide whether the old behavior was right.

The guide to testing non-deterministic AI outputs lists metamorphic relations that suit LLM features and compares them with golden sets.

Can an LLM Act as a Test Oracle?

Yes, as a proxy oracle in the ISTQB sense. A judge is still a model, so its verdict is evidence for a reviewer to check. Set it up so each verdict can be audited:

• Quote the evidence - require the passage of the input, transcript or file that each verdict rests on, and treat a verdict without a quote as unverified, because nobody can check it.
• Allow Unknown - Anthropic's agent evals guide advises giving the model "a way out, like providing an instruction to return 'Unknown' when it doesn't have enough information." An Unknown then maps onto an unverified result instead of a guessed pass.
• One judge per dimension - the same guide suggests structured rubrics and grading "each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions."
• Let it investigate - a judge that can locate, read and cross-check the files the agent produced for each requirement agrees with people more often than an LLM judge given the same material without those steps.

The ICML 2025 paper Agent-as-a-Judge (Zhuge et al.) measured that last point, with both kinds of judge given the same workspaces and the agents' trajectories (Table 3):

• Agent-as-a-Judge - agreed with the human consensus on 86.61% to 92.07% of requirement evaluations across three code-generating agents.
• LLM-as-a-Judge - agreed with the same consensus on 68.86% to 71.85% of them.
• The benchmark, DevAI, has 55 code generation tasks with 365 requirements, so the result covers code generation only.

When the judge answers Unknown, the fallback is a person, whom Barr et al. call "the final source of test oracle information." The guide to LLM-as-a-judge covers judge types, scoring methods and known biases.

How to Report What Could Not Be Verified

A criterion no oracle could settle is a result in its own right, and deleting the case hides it. Test standards already have a verdict for it: ETSI's TTCN-3 core language standard (ES 201 873-1 V4.17.1) gives five verdict values, "pass, fail, inconc, none and error," where inconc means inconclusive.

Its overwriting rules (Table 30) keep a later pass from hiding an inconclusive step:

• inconc, then pass - the verdict stays inconc.
• inconc, then fail - the verdict becomes fail, the only verdict a test can set that replaces inconc.

Apply the same rule to AI features, so the unverified part of a run stays visible next to whatever passed.

A single scenario was executed and passed cleanly, but overall feature coverage remains narrow.
1 of 1 executed · 100% passed of 1 decided
  1 passed · 0 failed
  0 unable to verify
However, this run tested only one scenario out of five declared features.

How Agent Assurance Tests Agents With No Expected Output

Nobody can write the expected reply for the refund-reply feature from the source-of-truth table, yet its code declares the billing tools it may call. Agent Assurance derives scenarios and acceptance criteria from that code and its declared tools; you supply only how to invoke the agent.

When the agent says the refund went through, a judging subagent grades each criterion by investigating: it may read files, inspect artifacts and call the agent's tools. These model judges grade the reply as well as the effect.

Tool calls carry much of that evidence:

• Declared tools as the oracle - the calls your profile reports are compared with the tools declared in the agent's code, so no one writes the expected tool list by hand.
• Empty call lists - for SC-014's not_called check, your profile's hook should return calls: [] only after observing no calls, or the check could pass unobserved.
• MCP tool lists - for MCP tools, the oracle is what each approved server reports it has, not what the config file claims; Rook CLI 0.1.5 connects to stdio servers only.
• Read-only probes - a judge can check whether a refund exists through an approved read-only tool, and its instructions are to change nothing, since calling issue_refund to find out would create one.

The eval column describes the category's defaults, not any single product. Building cases from a PRD or docs is common in eval tools, so the first row turns on code:

Point runs at staging, not production: the agent's writes are real and are not rolled back, so a scenario like SC-014 can issue a real refund if the agent misbehaves. In an attended terminal, Rook CLI reports how many write tools the agent declares and asks before it starts.

Rook CLI ships as an npm package; install it and check the version:

npm install -g @testmuai/rook
rook --version

With Node 22 or newer in place, run rook from the repository that holds the refund feature's code; the Rook CLI install guide has the other install methods.

In Claude Code, add the skill:

npx @testmuai/rook-skill@latest install --agent claude-code

Then send a request that stops before anything runs:

/rook Use the staging profile to propose three refund scenarios that have no
single expected output. List the write-capable tools the agent declares and
any whose write behavior is unknown. Stop there until I approve generating
scenarios or invoking the agent.

Note: TestMu AI's Agent Assurance runs from your terminal as rook: for an agent with no expected output, it derives the criteria, grades them against evidence and counts what went unverified.

Conclusion

Start with one AI feature you already ship. Write its acceptance criteria, fill in the source-of-truth column from the table above for each one, and count the rows that end in Unable to Verify. That count measures your test oracle problem before any tool is involved.

For agents that call tools, many of those rows close once the code that invokes the agent returns what it observed. The Rook CLI profiles and hooks guide explains that returning observed tool calls enables not_called assertions and action-versus-claim checks, and that leaving them out means the calls were not observable.