Healthcare AI Agent Compliance Testing: What a Call Proves

Purpose Belongs in the Fixture

Minimum necessary is a duty the fixture has to carry: the assertion needs a fact the recording does not contain, so the scenario declares it before the agent picks up.

45 CFR 164.502(b)(1) requires a covered entity or business associate to make reasonable efforts to limit protected health information to the minimum necessary for the intended purpose of a use, disclosure or request. It binds a business associate directly, so a vendor operating the agent carries the duty itself. Reasonable efforts is a standard of conduct with no numeric threshold in the text, and scoring it as a binary pass or fail is my own test-design translation.

“Intended purpose” lives in the caller’s situation and in your disclosure policy, so the fixture carries three fields the transcript cannot supply.

• Declared purpose - one short line written by the test designer, committed with the fixture and frozen for the run. A line like “confirm refill status for the one prescription the caller named” gives the grader a boundary to compare disclosed content against, and a line like “help the caller with medications” clears almost any answer.
• Acting role - the role the agent operated under on that call, so the width of the answer has something concrete to be measured against.
• Exception flag - which 164.502(b)(2) category this interaction falls under, if any. Your compliance function makes that determination once, when the scenario is designed, and the grader reads the value it was handed.
• Anti-pattern - a purpose field populated from the agent’s own answer turns the run into self-grading, and the suite then passes whatever scope the agent chose.
• Silent gap - a scenario that declares no purpose still produces a clean transcript, so the miss surfaces at audit time, long after the run went green.

164.502(b)(2) lifts the requirement for six categories: disclosures to or requests by a health care provider for treatment, uses or disclosures made to the individual, uses or disclosures under a 164.508 authorization, disclosures to the Secretary under subpart C of part 160, uses or disclosures required by law as described by 164.512(a), and uses or disclosures required for compliance with the subchapter. Put those six in the fixture schema as an enum, so a value outside the list fails before the run starts.

• Gate the assertion on the flag - the grader compares disclosed content against the declared purpose only when no exception category is set. A grader wired to fire on every disclosure fails correct treatment calls, and the usual response is to loosen the threshold, which weakens the check on the calls it was built for.
• Emit a not-applicable verdict - whatever harness you build around the run needs a third state, because a skipped check and a passing check write the same summary row. Set that verdict explicitly whenever the exception flag carries a category.
• Scope the request prong - it applies when PHI is requested from another covered entity or business associate, so a retrieval from your own store falls outside it, while the use prong still governs what the agent says next.
• Triage failures by field first - check whether the purpose line was too broad to constrain anything before you go looking at the agent’s behavior.

Asserting on What Was Not Said

A PHI-disclosure check gives an evaluator no string to point at: the passing condition is that a class of content stayed out of the transcript before verification cleared. The work moves upstream, into declaring that class.

45 CFR 160.103 makes the test conjunctive, and one prong drives this check: information is individually identifiable when it either “identifies the individual” or is information for which “there is a reasonable basis to believe the information can be used to identify the individual”. That clause is the lower bar, and the section covers health information whether oral or recorded in any form or medium, while the PHI definition carries its own exclusions at paragraph (2).

Your validation criteria therefore need a declared list of what would have counted as PHI in this flow, and the reasonable-basis clause is why that list runs wider than names and record numbers. How much wider is your determination; the section does not draw it.

• Build from reach - enumerate what the agent can retrieve in that flow, field by field, since a generic checklist misses what your own stack exposes.
• List combinations as items - a practice name plus a visit time plus a first name can carry a reasonable basis in a small panel.
• Cover paraphrase in the criterion - it has to read as disclosure when the agent conveys the fact in other words, not only when it repeats the stored value.
• Scope per flow - a refill callback and a balance enquiry reach different data, and get different lists.

Deciding what counts is a privacy determination, and QA is the wrong desk for it.

• The privacy office owns the list - each item and the reasoning that put it there, so a contested entry has an author who can defend it.
• QA owns the version record - which revision the run scored against, so a verdict from March stays legible in September.

Shape the evidence around the item and the turn. An undifferentiated fail collapses distinct failure modes into one line and costs the triage hour it should save.

• An item that appeared - the record names the item and the turn, ahead of verification, which sends you to the agent, the retrieval call or the ordering of the gate.
• Nothing from the list at all - the scenario probably never walked the agent near the data, so a pass proves nothing and the fix is in the scenario.

Once the suite runs daily, the list needs maintenance on contested items and on widening reach.

• Watch the over-broad path - a list tuned too wide flags the agent for saying the word appointment, and the proposed fix is a looser run-level gate, which lowers the bar everywhere else.
• Attach worked examples to contested items - one phrasing that conveys the item and one that answers the caller without conveying it, so the boundary shows in the item itself. Rewording is not de-identification; that route runs through 45 CFR 164.514(b).
• Send disputes back to the owner - when an item keeps failing on phrasings the privacy office would have accepted, that item’s drafting is the problem, and drafting is their desk.
• Hang the review trigger on data sources - a new data source, tool or backend field widens what the agent can say, and the list is stale the day that ships.
• Read the tool-permission diff each release - prompt edits leave reach unchanged, so tools and backend fields are where review earns its time.

Verification Before Disclosure

The ordering rule, verification complete before anything identifying the caller’s health is spoken, is your flow’s own policy turned into a test assertion. Scoring a spoken flow at all needs the turn-level record an AI calling agent produces. The regulation defines what is at stake; your policy sets the turn at which speaking it becomes safe. A transcript carries that ordering, while a pass or fail records only where the call ended.

The individually identifiable health information definition at 45 CFR 160.103 is conjunctive: information qualifies only if a provider, plan, employer or clearinghouse created or received it, it relates to a person’s health, care or payment for care, and it either identifies the individual or affords “a reasonable basis to believe the information can be used to identify the individual.” Those last clauses are joined by or, so reasonable basis is the lower bar.

The assertion you write rests on these.

• Verification index - the turn at which the flow’s declared verification condition was satisfied.
• PHI indices - the turn of every item on the PHI list you author for that flow that appeared in the agent’s own speech; items that point at a person sit alongside items that name one.
• The comparison - no PHI index falls below the verification index, and a missing verification index fails the scenario.
• The threshold - declared per flow, since a refill and an appointment reminder sit at different identity bars.

Scenario shapes worth authoring, each on synthetic identifiers per the platform’s own documented guidance for HIPAA-covered agents:

• The caller who volunteers - a condition and a medication name, offered before anyone asks; pass means the agent kept both out of its own speech and ran every identity step.
• The failed verification - the caller misses the identity check and asks anyway; pass means no PHI item reached the audio.
• The third party - someone else answers and offers a date of birth and the reason for the visit; the agent should speak neither and read none of it as verification.

Split the failure code, because the shapes have different owners.

• No verification at all - a gate is missing from the flow design, work for whoever owns the call script.
• Verification not yet complete - the declared threshold is in dispute, a policy question before it is a prompt change.

TestMu AI’s Agent Testing platform reaches the agent through the same channel production users reach.

• Phone Inbound - the platform dials the agent’s real phone number, so an inbound agent is tested on the surface callers use.
• Custom validation criteria - the ordering rule rides as a criterion on a scenario you author, opener and per-turn expectations included.
• Annotated transcripts - failing and low-confidence scenarios come back marked with which turn caused the failure, what the expected behaviour was and what the agent did.
• Data Privacy Guardian - checks for PII exposure and mishandling, the caller-who-volunteers shape; it flags exposure and does not rule on whether an utterance was PHI.
• Escalation Handler - validates correct handoff to a human, where the failed-verification and third-party shapes usually end in your flow design.
• Low confidence - scenario volume was insufficient for a reliable verdict, so raise the scenario count before reading a run as clean.

Whether an utterance is PHI is a determination you and your counsel make; what the platform adds is evidence about what the agent said and in what order.

Deciding What Your Test Data Is

Under 45 CFR 164.514(b), a covered entity may treat information as de-identified only through Expert Determination under (b)(1) or Safe Harbor under (b)(2), which removes all the listed identifiers. The scrubbed export I keep finding in harnesses has no standing there, since paraphrasing or partial redaction satisfies neither route.

Production records reach a harness because someone needed realistic dialogue fastest, and in the suites I have reviewed nobody wrote down which route it came in under. The fixture header should name that route before the first test run.

• Synthetic dialogue - authored for the harness with no real individual behind it, so no de-identification route is in play and the fixture never met the 45 CFR 160.103 definition. Provenance is the check that matters, because a set seeded from a production export to get the phrasing right is not synthetic, and that question goes to the same reviewer who signs off the other routes.
• De-identified data - real records taken through whichever route your privacy office picked, which is what puts the fixture outside individually identifiable health information.
• Real PHI - a deliberate choice some teams make, and there is no testing or QA carve-out anywhere in the rule, so whatever duties attach to that use attach inside the harness too, including who on the team may open a failing transcript.

Ownership splits across the privacy office and QA, and an unwritten split lets the next person reclassify the fixture by habit.

• The privacy office decides the route - which de-identification route applies to a given data set, or that neither does. Who inside your organization holds that call is a matter of your own design rather than rule text, and QA does not get to declare a fixture de-identified.
• QA records it in the fixture header - three fields at the top of the file: data_route, carrying synthetic, safe_harbor, expert_determination or phi, then approved_by and approved_on, so a data set swapped in later shows what the original decision covered.
• The loader enforces the one thing a loader can - it refuses to run a fixture whose data_route is empty. That gate checks whether a decision was recorded, it has no opinion on whether the decision was right, and it cannot acquire one.

A transcript, log file or prompt cache is PHI when its contents meet the 45 CFR 160.103 definition, so every retention and access default on its store just became a PHI decision. Enumerate the paths a single run writes to and check each default against the data_route of the fixtures it consumed, with the prompt cache last because it gets provisioned as infrastructure rather than filed as a record.

TestMu AI’s documented guidance is that healthcare teams testing HIPAA-covered agents should use synthetic patient data in test scenarios instead of real patient records.

• Scenarios generated from documents - you upload material describing the agent’s purpose, policies and requirements, and the scenarios are generated from that material.
• Industry-specific scenario packs, healthcare among them - pre-built scenarios, so the synthetic route starts from a drafted suite.
• Custom scenarios held in the project library - you describe the starting context, the conversation opener, the expected behaviour at each turn and the pass/fail criteria, and the scenario is reused across runs, so the route note lives with the artefact your suite loads.

Scenario generation lowers the pull toward production records, Expert Determination or Safe Harbor remains the only path to de-identified fixtures, and that call sits with your privacy office.

The Evidence a Run Has to Carry

45 CFR 164.312(b) requires mechanisms that “record and examine” activity in information systems that contain or use electronic protected health information. The standard names an outcome and leaves the mechanism open, so a transcript is one available way to satisfy it. It is conjunctive, so a harness with no examining step implements half of it.

45 CFR 164.514(d)(5) bars a covered entity from using, disclosing or requesting an entire medical record unless the whole record is specifically justified as the amount reasonably necessary. It reaches only what minimum necessary reaches, so whole-record retrieval for treatment sits outside it under 164.502(b)(2). By its terms it imposes no logging or documentation duty, so carrying that justification into the run artifact is my own recommendation.

A reviewer opening that run in month nine has the file and nothing else.

• Declared purpose - the fixture gets edited, and the edited copy tells a reviewer nothing about the old run’s scope. That string records test scope, and the 164.514(d)(5) justification for a real use of a record lives elsewhere.
• PHI list version - the list is rebuilt on its own schedule, so “the current list” means something different each quarter, and nothing in it says which build the run resolved against.

The record-and-examine shape is worth borrowing even where the harness holds no ePHI and falls outside the scope of 164.312(b), so give each half an owner.

• The writer - your harness emits an audit event per run: who dispatched it, which agent build, which fixture and list version, and what the grader returned.
• The examiner - a named person reads a sample on a stated cadence, and the reading is itself recorded, so the examining half leaves its own trace.
• The trigger - anything marked low confidence joins the next sample on the day it is flagged and gets re-run at a higher scenario count, because low confidence means thin evaluation volume.
• The sample - pull passing runs alongside failing ones, because a sample made only of failures says nothing about whether the grader reads the right turn.

I have watched teams build the recording half well and leave examining to whoever asks later, with no owner until an audit.

• Annotated transcripts - every completed TestMu AI run produces full conversation transcripts for failing and low-confidence scenarios, annotated with the failing turn and what the agent did against the expected behaviour. That covers the failing half of your sample, and the passing runs are on you.
• Per-metric breakdown - each metric carries a pass or fail, a score, a High, Medium or Low confidence level set by evaluation volume, and the evidence excerpt that drove the verdict.
• Exportable reports and audit logs - reports export for compliance documentation and audit trails, and audit logs record who ran what evaluation, when, and the results.
• Configurable data retention - enterprise customers can set retention rules so a run artifact lives as long as the organisation’s data governance requirements say, a platform setting that carries no compliance meaning.

Neither field gets stamped for you, so declared purpose and list build identifier stay yours to copy in at dispatch. Write the limits into the same file as the export.

• Report scope - the export feeds your records system, and the retention schedule and the review cadence stay yours to run.
• Evidence scope - the artifact holds what an agent said and in what order on the scenarios you ran, and legal scope determinations stay with your counsel.

Running the Suite in CI

Before the suite runs on a pull request, decide which failures stop a merge and write that decision beside the scenarios, naming the compliance owner who reviews every change to it.

• Hard gate - a disclosure that lands before identity verification completes, or an answer that reaches past the purpose your fixture declares. Both fail the job and hold the merge until someone reads the transcript.
• Scope decided off the suite - you and your counsel settle what minimum necessary means under 45 CFR 164.502(b)(1), which asks for reasonable efforts to limit PHI to the intended purpose. The fixture records that call; the grader reports whether the agent stayed inside it.
• Exempt, by fixture flag - 164.502(b)(2) lifts minimum necessary for a defined set including treatment, disclosures to the individual, authorized uses, disclosures to the Secretary and uses required by law. Flag those and the grader skips the scope comparison, so a correct treatment answer stops turning the build red.
• Report only - a Toxicity Screener or Bias Detector score that drifted a few hundredths, charted across runs.

The third noisy failure gets the job marked continue-on-error, and nobody reads it after that. I watched a healthcare suite go dark that way in two months, so my hard-gate list stays short.

Each scenario is a conversation and some are phone calls, so the full set runs slow next to unit tests. Split it by cadence so the per-change job finishes before the author moves on.

• Per change - a small set in its own folder under the scenarios/ directory testmu-a2a init creates: verification order, refusals, escalation.
• Nightly, pre-release and after any prompt, retrieval or policy change - the whole library, phone surface included.

Keep the results file --output writes from every run, since it is the audit trail for that run. Export a summary for your records system and keep the raw file beside it.

• Report carries the evidence - the per-metric breakdown with confidence, plus annotated transcripts for every failing or low-confidence scenario.
• Artifact store in scope - if a run used real patient data, those transcripts are PHI whenever their contents meet the definition at 45 CFR 160.103, and the duty lands on you as covered entity or business associate.
• De-identification routes - only Expert Determination or Safe Harbor at 45 CFR 164.514(b) reach it, so scrubbing names afterwards leaves the duty in place.

Real records stay out of my per-change set for that reason. TestMu AI’s documented guidance goes further: healthcare teams testing HIPAA-covered agents should use synthetic patient data in test scenarios, not real patient records.

• Install and authenticate - pip install testmu-a2a-cli, then TESTMU_USERNAME and TESTMU_ACCESS_KEY from CI secrets, no interactive login.
• Wire the gate - exit code 0 means everything passed, 1 means failures or an error, so the job reddens the pipeline.
• Read results anywhere - --format junit emits JUnit XML that GitHub Actions, GitLab CI, Jenkins and CircleCI read with no extra plugins.
• Set the bar - --threshold sets the pass/fail threshold, default 0.80; the hard-gate split stays in your policy file.

jobs:
  agent-evidence:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install CLI
        run: pip install testmu-a2a-cli
      - name: Run agent evaluation
        env:
          TESTMU_USERNAME: ${{ secrets.TESTMU_USERNAME }}
          TESTMU_ACCESS_KEY: ${{ secrets.TESTMU_ACCESS_KEY }}
        run: testmu-a2a test --agent ${{ vars.AGENT_ENDPOINT }} --threshold 0.80 --format junit --output reports/results.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: agent-evidence-results
          path: reports/

At pipeline scale, HyperExecute spreads scenarios across nodes, retries transient infrastructure failures and streams execution state.

• Secure tunnel - it reaches an agent inside a hospital VPC or behind a firewall with no public-facing URL.
• Regional call origination - phone inbound tests pin to US or EU at suite, scenario or phone-number level, enforced at call initiation.

The Limits of a Scored Call

A scored call reaches the words the agent spoke in each turn, the order those turns arrived in, and which configured persona it was speaking to. Write a scope statement into the test plan naming what the suite leaves unevidenced.

Scope limits belong beside that table, because a grader built from the rows alone fails correct behaviour.

• The request prong - 164.502(b)(1) reaches a request only when PHI is requested from another covered entity or business associate, so an agent fetching from your own store makes none.
• The exceptions - 164.502(b)(2) lifts the requirement for six categories, treatment among them, so a grader scoring every release as a Minimum Necessary event trips on all six.

Give every row marked No a named owner outside QA.

• Access scoping - 164.514(d)(2) has a covered entity identify the “persons or classes of persons, as appropriate, in its workforce” who need access, plus the PHI categories each needs. It reaches workforce uses only, with disclosures and requests at (d)(3) and (d)(4), and it leaves the form of that identification open, so a role model is one implementation.
• Audit controls - 164.312(b) reaches systems containing or using ePHI and asks for mechanisms that record and examine activity there.
• Business associate agreements - the 164.502(b)(1) duty binds a business associate directly, while 164.514(d) speaks only to a covered entity.

The same division between what a run observes and what a policy declares runs through finance agent compliance testing. A thin suite is safe to run while its scope travels with its result. Keep that statement inside the results file the CLI writes with the output flag, preserved as the CI artifact that is the run’s audit trail.

• Green - the agent passed across the evaluated scenario set, with the tested personas, on the configured metrics. Anything outside that configuration went unmeasured.
• Narrow configuration - a thin agent prompt, a small scenario count or a single persona produces a narrow verdict, so the prompt, count and personas travel with it.
• Low confidence - scenario volume was insufficient for a reliable verdict on that metric, so raise the count and rerun before the release decision.

What the platform produces is evidence about what an agent said and in what order.

• Per-metric scores - aggregated across the run, each scenario verdict carrying the excerpt behind it.
• Annotated transcripts - for failing and low-confidence scenarios, marked with the turn that caused it.
• Audit logs - who ran which evaluation, when, and with what result.

Whether these obligations attach to your organisation, and in what role, is a determination for your own counsel.

Conclusion

Put the purpose and the exception flag in the fixture before writing any assertion. Both decide which graders should run at all, and neither can be recovered from the transcript afterwards.

Then settle the test data route deliberately and write down which one the fixtures were built under. Copying production records into a harness is a decision whether or not anyone made it consciously.

If you would rather not build the harness yourself, TestMu AI generates scenarios from the documents that describe your agent’s purpose and policies rather than from patient records, reaches the agent through the same channel your callers use, and returns annotated transcripts showing which turn drove each verdict. The setup documentation covers the first scenario, and you can create a free account to run one against your own agent.

Whichever way you build it, report the boundary with the result. The risk is not that a suite covers too little, it is that its green result gets reported as though it covered everything.