Customer Support Agent Compliance Testing: What to Assert

Ordering Is the Assertion

A disclosure-ordering check compares two integers: the turn index where the customer consented to pay, and the turn index of each required disclosure. It passes when every disclosure index is lower than the consent index.

The duty behind that comparison is 16 CFR 310.3(a)(1).

• The cutoff is the consent turn - 310.3(a)(1) makes it a deceptive telemarketing act and a violation of this part for any seller or telemarketer to fail to disclose the enumerated material information “[b]efore a customer consents to pay for goods or services offered.”
• (a)(1)(i)-(viii) enumerates the disclosures - total costs, material restrictions or conditions, the refund or cancellation policy and negative option terms, which is the list your fixture turns into disclosure IDs.
• Coverage is counsel’s call - Part 310 governs telemarketing, and whether your flow falls inside it is a scoping question your grader should never try to answer.

None of it is scorable until the consent index exists, so the fixture identifies the consent turn before it identifies anything else.

• Consent is oblique in real calls - it arrives as “yes,” “mhm,” “go ahead,” or a repeated card number, and a generic matcher will pick the wrong turn or no turn at all.
• Define it per flow in the fixture - a renewal, a first-time subscription and a one-off charge each have a different moment that counts, and the fixture should carry that definition as data the grader reads.
• Name the field and keep it boring - a consentTurn field holding an integer and a consentMatch field holding the quoted span give the grader one number to compare and give a reviewer one line to audit.
• Fail loudly when it is absent - if no turn satisfies the fixture’s consent definition, emit a distinct result meaning “no consent turn found,” because a silent zero index turns every disclosure into a late one.

A whole-transcript string search confirms only that the cancellation policy appears somewhere in the call, which stays true when the agent read it thirty seconds after the customer said yes. A grader with no position awareness returns a pass for exactly the call the suite exists to catch.

A bare fail sends the whole team to the same transcript, when the defects underneath it have different owners and different fixes.

• MISSING - no turn in the call carries the disclosure, which points at prompt content or a retrieval gap, and the fix lands with whoever owns the agent’s script.
• LATE - the disclosure is present but its index is higher than the consent index, which points at dialog flow and state ordering, and the fix lands with whoever owns the turn graph.
• Report both indices with the code - “refund policy at turn 22, consent at turn 17” gives the owner the whole diagnosis before they open the transcript.

Store turn order, per-turn timestamps and the matched text alongside the score, so a reviewer can re-read the decision months later. (a)(1) is a rule about ordering, and verifiability is a separate duty at 310.3(a)(3), so the record you keep here serves your own reviewers and your next regression run.

Scoping Which Flows Are In

16 CFR 310.4(d) reaches a telemarketer in “an outbound telephone call or internal or external upsell to induce the purchase of goods or services,” and its definitions decide which calls it covers:

• 310.2(x), outbound telephone call - a call initiated by a telemarketer, so a flow that dials out is inside 310.4(d) from the open.
• 310.2(jj), upselling - a solicitation after an initial transaction in the same call, whoever initiated it, so an upsell inside an inbound support call is inside 310.4(d). The two call directions are compared in inbound versus outbound phone agent testing.
• 310.3(a)(1), the wider duty - within telemarketing covered by Part 310, any seller or telemarketer must disclose the material information it enumerates before a customer consents to pay, with no outbound limit.

A scenario names the provision it asserts against before it scores anything. Write a scope record for each flow the agent can enter, version it beside the suite, and review it alongside the scenarios that read it. Every field decides which assertions run:

• Call direction - inbound or outbound where the flow starts, which decides whether 310.4(d) covers the call from the open or only from the upsell turn.
• Sales capability - whether the agent can move a caller toward a purchase, which decides whether the disclosure assertions run at all.
• Upsell capability, internal or external - an external upsell brings the whole 310.4(d) disclosure set back on, while an internal one reaches only information differing from the initial transaction’s disclosures, so the fixture carries those for the diff.
• Payment capture and instrument - the 310.3(a)(3) express verifiable authorization duty excepts a credit card under Regulation Z and a debit card under Regulation E, while the separate 310.4(a)(7) consent duty carries no such exception.
• Preacquired account information and free-to-pay conversion feature - distinct flags whose combination triggers the 310.4(a)(7)(i) duty to record the entire transaction, so preacquired details alone leave that assertion off.
• Negative option feature - 310.3(a)(1)(vii) attaches only where the offer includes one, and it is a distinct flag from free-to-pay conversion.
• Prize promotion offered - the prize disclosure at 310.4(d)(4) is keyed to the description of the prize, so the scenario marks that turn for the grader to measure against.

A transcript carries the agent’s words and their order; preacquired account information is a fact about the billing system. Without declared scope fields you cannot tell whether a missing disclosure failed or was out of scope.

On a plan-change support flow I own, the record reads inbound, internal upsell, preacquired account information true, free-to-pay conversion false, credit card under Regulation Z. The grader aims the 310.4(d) assertions at the upsell turn and leaves the recording assertion off. Re-read that record whenever the flow changes underneath it:

• A support path starts offering a plan change - it now carries an upsell, and the record names it internal or external before those assertions return.
• A trial or renewal term lands in the offer - that can introduce a free-to-pay conversion or a negative option feature the record does not yet name.
• Nobody reviewed the record - a flow that ships a change without a scope review keeps passing its old assertions and stops testing the new behavior.

Whether a given program is telemarketing is a question for your counsel. The scope record stores that answer, dated and attributed, and your scenarios read it as an input.

Building the Scenario Set

Two runs of the same assertion diverge if the account state or the opening line changes. Write these fields before you write a grader.

• Starting context - who the caller is at pickup: account status, open tickets, whether they already hold the plan.
• The opener - the exact first line, because a billing complaint and a renewal question branch differently.
• Expected behaviour per turn - what the agent should do at turn two and turn seven, with the close as one checkpoint.
• Pass/fail criteria - the condition a grader can evaluate without a person reading the transcript.

A set built only from cooperative callers passes on the first run, because nothing in it interrupts the disclosure. The failures I keep seeing come from callers moving faster than the script.

• The early yes - the caller says “yeah, fine, sign me up” mid-terms, and the assertion is whether the rest arrives before consent to pay.
• The repeated price question - the caller asks the total twice, ten turns apart, and the second answer has to match the first.
• The wrong yes - the caller agrees to something adjacent, a callback or an email, and the agent treats it as consent to be charged.
• The interrupted disclosure - the caller talks over the agent and it resumes at the wrong point in the terms.

Where the flow is telemarketing covered by 16 CFR part 310 and the offer carries a negative option feature, 310.3(a)(1) sets the ordering line at “[b]efore a customer consents to pay,” and 310.3(a)(1)(vii) puts all material terms and conditions of that feature inside it. The items it names sit in an “including, but not limited to” list: the account will be charged unless the customer takes an affirmative action, the dates the charges will be submitted for payment, and the specific steps to avoid them.

Those named items are checkable minimums inside a wider duty covering everything material about the feature, and each half grades differently.

• Named items - each is a fact the grader looks for, positioned before consent to pay.
• Residual duty - whoever owns the offer writes down what else is material, because a grader cannot infer materiality.
• Owner of the list - it goes stale fastest in the suite, because cancellation steps and charge dates change without anyone telling QA.

That list is maintained by hand. TestMu AI’s Agent Testing platform generates the scenario set that carries its items into every run.

• Generation from documents you already keep - you upload the offer sheet and the policy doc, the platform parses them into behavioural requirements, constraints and scenario seeds, and generation runs from that.
• Volume per generation run - the awkward-caller shapes multiply by every offer variant, which is where hand-authoring stops. 60 to 100+ scenarios in a standard workflow, scaling with document size up to 300 per generation run.
• Criteria attached to the scenario - generated scenarios carry their own validation criteria, where the named items sit as literal assertions; the per-offer material list goes in as custom validation criteria layered on the standard metrics.
• A compliance category in the pre-built library - required disclosure delivery, forbidden topic handling and escalation triggers, so disclosure-order scenarios exist before you author anything. The uncooperative shapes come from generation, as edge cases and adversarial inputs.
• Custom scenarios that persist - the fields above, authored by hand, then stored in the project library and reused across runs.

What comes back is evidence of what the agent said and in what order; whether a disclosure was adequate, and whether the flow is telemarketing at all, stays with counsel and the offer owner.

Writing the Graders

Express verifiable authorization is 310.3(a)(3); express informed consent is 310.4(a)(7). A payment taken on a call owes both, and a suite that asserts one leaves the other untested without ever failing.

• Only one route is producible by a call - 310.3(a)(3) deems authorization verifiable by several alternative means, and a call produces only express oral authorization, audio-recorded and made available on request. It excepts credit cards under Regulation Z and debit cards under Regulation E, an exception 310.4(a)(7) does not carry. So the payment instrument on the fixture decides whether the (a)(3) grader runs; the (a)(7) grader runs either way.
• Consent covers the charge and the account - 310.4(a)(7) requires express informed consent to be charged and to be charged using the identified account. Grade those separately; an agent that wins agreement to the purchase and never names the account passes a merged check.
• The entire-transaction recording is conditional - where a transaction involves preacquired account information and a free-to-pay conversion feature, 310.4(a)(7)(i) adds an audio recording of the entire transaction at (a)(7)(i)(C) to the last-four-digits and express-agreement elements. Preacquired information without that feature falls under (a)(7)(ii), which carries no recording duty.

The recording grader reads the scope record on the fixture: payment instrument, preacquired account flag, free-to-pay conversion flag. It matches the flags against the pairing in (a)(7)(i), then runs the audio check or writes a not-applicable result carrying those values.

• Give not-applicable its own status - stamp the scope fields in, so the verdict carries why it did not run. A skipped grader and a passing one look identical in a run summary, so a check can vanish for a release.

A negative assertion has nothing to match on, since the phrasing you hunt for never appeared. So the grader needs a defined list of what would have counted.

• Write the accept list before the assertion - the account-naming list holds “ending in,” “last four” and the digits read singly; the grader scans the agent turns after the consent prompt. Without it the grader tests your own vocabulary and passes wording you missed.
• Version the list with the prompt - when a prompt change has the agent saying “the card ending 4-1-2-2,” a stale list turns a behavior change into a pass nobody flags.
• Read a sample of passes by hand - failures already get attention, so a grader stuck returning pass goes unexamined for weeks. I pull ten passing runs from my own suite each week and read the transcripts first.

TestMu AI scores every interaction, and each metric produces its own pass or fail for the scenario, a confidence level of High, Medium or Low based on evaluation volume, an evidence excerpt from the conversation that drove that verdict, and a per-metric score aggregated across the run. A reviewer triages from the excerpt, which names the turns behind the verdict.

• Specialized testing agents run in parallel - the Compliance Validator checks regulatory and policy adherence, the Data Privacy Guardian covers PII exposure and the Escalation Handler validates handoff to a human, each over the same channel production users reach. Point one scenario at the authorization phrasing and another at the account naming.
• Low confidence means thin coverage - it says scenario volume was not enough for a reliable verdict. Raise the scenario count and run again before recording a pass.
• The platform reports evidence - it records what the agent said and in what order, and a pass covers the evaluated scenarios, personas and metrics. Whether a flow carries the recording duty is a question for counsel, and their answer belongs in the scope record your grader reads.

Running the Suite in CI

An ordering assertion is a yes or no fact about turn order. A conversational-quality score is a model-graded number that moves between runs. Give them different consequences, in writing, before the first red build.

• Hard gate - any ordering assertion. § 310.3(a)(1) puts the material disclosures “[b]efore a customer consents to pay”, so a card number requested before total cost and the refund or cancellation policy are on the record fails.
• The proxy you picked - the card request is your stand-in for that consent boundary, so put the choice in the scope record. Where the offer carries a negative option feature, § 310.3(a)(1)(vii) adds the charge date and the steps to avoid it, so gate that one only on negative-option scenarios.
• Report only - the conversational-quality scores you layered on as custom validation criteria. They move run to run with scenario wording, so a gate on them blocks releases for no actionable reason.
• A person owns the line - the tool reports pass or fail against your criteria. Which failures block a release is a policy your compliance owner signs and stores next to the scope record.

Every run writes a results file, and you and your counsel work from it eight months later, when the prompt has been rewritten twice. It is engineering evidence about one build; the § 310.5 recordkeeping categories, where they apply, are a separate question for your compliance owner.

• The results file - uploaded as a CI artifact on every run, passing ones included, with the commit SHA, prompt version and scenario set version your pipeline writes beside it. The platform audit log records who ran it and when.
• Failing transcripts - the report carries full transcripts for failing and low-confidence scenarios, annotated with the turn that caused it, the expected behaviour and what the agent did.

Each scenario is a live multi-turn conversation, so the suite runs on a different clock from your unit tests. Match the cadence to what changed.

• Per change - a fast subset sized with --count, kept in the scenarios/ directory that testmu-a2a init creates.
• Split the authorization scenarios - express verifiable authorization under § 310.3(a)(3) and express informed consent under § 310.4(a)(7) are separate duties, and one payment bucket hides which broke.
• Nightly, pre-release and after any prompt edit - the full set, every persona and adversarial variant, since a prompt change moves behaviour on scenarios your subset never touches.

The --format junit output is JUnit XML, read by GitHub Actions, GitLab CI, Jenkins and CircleCI without extra plugins, and exit code 0 means all passed while 1 means failures or error. --threshold defaults to 0.80 per invocation, so a stricter payment policy runs as its own job.

name: agent-compliance
on: [pull_request]
jobs:
  compliance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install testmu-a2a-cli
      - name: Ordering and authorization subset
        env:
          TESTMU_USERNAME: ${{ secrets.TESTMU_USERNAME }}
          TESTMU_ACCESS_KEY: ${{ secrets.TESTMU_ACCESS_KEY }}
        run: |
          testmu-a2a test \
            --agent ${{ vars.AGENT_ENDPOINT }} \
            --spec ./specs/payment-authorization.md \
            --count 12 --threshold 0.90 \
            --format junit --output reports/results.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: compliance-results
          path: reports/

• HyperExecute - the recommended execution backend at scale. Scenario distribution across nodes keeps the per-change subset inside its time budget, and automatic retry on transient infrastructure failures stops a flaky node reading as a compliance failure.
• Secure tunnel, via HyperExecute - tests an agent behind a corporate firewall or inside a VPC without a public-facing URL. Without it the suite runs against a mock that shares none of the deployed agent’s prompt or guardrails.

A green result covers only the scenarios, personas and metrics listed in your config, and it is not a guarantee of zero failures in production. You get evidence about what your agent said and in what order. Whether a flow falls inside a rule is a question for counsel, answered in the scope record.

The Evidence the Run Leaves

Amended section 310.5(a) (89 FR 26784, Apr. 16, 2024) makes a seller or telemarketer keep eleven record categories for five years. Section 310.5(a)(2) covers each telemarketing call as the TSR defines one.

Most AI support traffic falls outside that definition, so the scope record decides whether these elements apply at all. The three below are the ones that change what a harness has to emit.

Section 310.5(b) fixes their form: North American Numbering Plan or ITU-T E.164 numbers, time in Coordinated Universal Time, and time and duration to the closest second.

• Validate early - check fixture numbers against the format at suite load, so “555-0182 ext 4” fails there rather than in the record.
• Local time and minute buckets - a pipeline that stamps region-local time or buckets duration to the minute cannot produce a conformant record, and the run output still looks clean.
• Retention clock - five years from the date the record is produced unless the rule says otherwise; scripts under 310.5(a)(1) run from when they stop being used.
• Scope, kept honest - the carve-out at 310.5(a)(2)(x) drops the (a)(2)(vii) and (a)(2)(x) records for calls an individual telemarketer makes by manually entering a single number.

The script used on a call is a required record element under (a)(2)(viii), so a harness that regenerates prompts per run cannot reconstruct the record it owes.

• Pin by hash - store the resolved prompt and any prerecorded audio as content-addressed blobs and write the hash into the per-call record; that hash is how the script reaches the record.
• Freeze at dispatch - resolve variables once before dispatch and keep the rendered text; a template plus a variable map only approximates what the caller heard.
• Triage drift first - when the Compliance Validator fails a required-disclosure scenario, compare that call’s prompt hash against the last run that passed it.
• The first-time mistake - disposition gets written as the scenario verdict, so a call that dropped at second nine logs the same as one that completed and failed.

The run covers one part of that record: every completed TestMu AI run produces full conversation transcripts for failing and low-confidence scenarios, annotated with which turn caused the failure.

• Evidence for the script element - each metric carries a score, a confidence level and the scenarios that drove it; the annotated transcript names the turn a disclosure went missing.
• Run-side artefact - exportable reports for compliance documentation and audit trails, plus audit logs of who ran what evaluation and when, feed your records system.
• Run-to-call join - keep the run id and the call id in one row, or the transcript and the (a)(2)(vii) entry never match up.

Retention for the period the rule sets is a duty on the seller or telemarketer, so the export is an input to that system. TestMu AI evidences what your agent said and in what order; legal scope is a question for your counsel, the same division of labour described in healthcare agent compliance testing.

The Limits of a Scored Call

A scored call reaches the words spoken in each turn, the order they arrived in, and the party addressed. Every assertion here is built from those signals, and the duties below belong to an owner in legal, payments, or product.

• Retention and retrievability - 310.5(a) runs a five-year clock on the enumerated telemarketing records, from the date each is produced unless the paragraph says otherwise. Whether the script version a call ran under still resolves in year five belongs to the storage owner.
• Allocation of the recordkeeping duty - 310.5(e) lets a seller and the telemarketer calling on its behalf split recordkeeping by written agreement; absent a clear one, both are responsible. Read that clause, then name in the test plan whose system of record holds the call-log fields.
• The payment method exception - 310.3(a)(3) excepts a credit card subject to Regulation Z and a debit card subject to Regulation E from its “express verifiable authorization” requirement. The separate express informed consent obligation at 310.4(a)(7) has no such carve-out, and your payments owner confirms which rail a flow charges.
• Materiality of an offer term - a term nobody wrote into the fixture cannot be graded. The categories come from 310.3(a)(1)(i) to (viii); which of this offer’s terms fall inside them is a business fact your offer owner supplies and counsel confirms.
• Coverage of the flow - it turns on deployment facts outside the transcript: who initiated the call, whether an upsell followed an initial transaction in the same call, and whether the offer carries a negative option or a free-to-pay conversion with preacquired account information. Counsel owns that determination; the scope record carries it into the run.
• The silent direction of a scope error - mark a flow inbound-only while it carries an internal upsell, and the 310.4(d) scenarios your harness selects on that flag never run. Those disclosures are owed only to the extent the upsell information differs from the initial transaction, so the pass covers a delta nobody read. I have twice found that record in an unowned spreadsheet column.
• Duties an inbound mark leaves standing - 310.3(a)(1) attaches before a customer consents to pay with no outbound-call limitation, so those graders run either way.

Each TestMu AI metric returns a pass/fail determination for the scenario, a confidence level of High, Medium or Low based on evaluation volume, and the evidence excerpt that drove it. Every failing or low-confidence scenario comes back as a full transcript annotated with the turn that caused the failure, what the expected behaviour was and what the agent actually did, the artifact a payments owner reads in a dispute. Low confidence means the scenario volume was insufficient for a reliable verdict.

Verdict quality follows the agent prompt, the breadth of scenarios and the personas tested. A Green verdict covers the evaluated scenario set, the tested personas and the configured metrics, with production traffic outside that set untested. The platform reports what an agent said and in what order; legal scope and the discharge of a regulatory duty stay with counsel.

Conclusion

Write the scope record before writing a single assertion. It decides which graders apply, and it is the artifact the rest of the suite rests on.

Then rewrite your disclosure checks to compare turn indices. Most suites already contain the right strings and prove nothing about when they arrived.

If you would rather not build the harness from scratch, TestMu AI generates scenarios from the documents that already describe your offers and runs them against the agent through the same channel your callers use, with outbound call testing having the agent dial a platform-owned number the simulator answers. Failing and low-confidence scenarios come back as annotated transcripts showing which turn drove the verdict. The setup documentation covers the first scenario, and you can create a free account to run one against your own agent.

Whichever way you build it, keep the boundary visible in how you report results. The suite evidences what was said and in what order, and the duties it cannot reach need owners of their own.