Finance AI Agent Compliance Testing: The Output Is a Record

Classification Is an Input, Not an Output

Two runs of one scenario can produce byte-identical text and still land in different categories, and the difference is a recipient count the response body never carries. Your assertions read that body, so the suite has no field to derive the category from.

• Correspondence - Rule 2210(a)(2) covers a written (including electronic) communication distributed or made available to 25 or fewer retail investors within any 30 calendar-day period.
• Retail communication - Rule 2210(a)(5) applies once that same written communication goes to more than 25 retail investors inside the window, so individualized chat output to one retail investor sits in correspondence while substantially similar or templated output crossing 25 is a retail communication whatever the delivery surface.
• Retail investor - Rule 2210(a)(6) covers any person other than an institutional investor, regardless of whether the person has an account with a member.
• Who Rule 2210 binds - FINRA members and their associated persons, so the classification above bites only where the deploying firm is itself a member.
• Advisers sit elsewhere - an SEC-registered investment adviser is covered by the Advisers Act Marketing Rule at 17 CFR 275.206(4)-1.

Rule 2210 does not mention AI agents, and it classifies by audience and content rather than by how the text was produced. The 30 calendar-day window slides, so one template can sit under one definition in March and the other in April. The fixture carries the category as a field the plan owner sets.

• Declared category - each fixture carries correspondence, retail communication or institutional communication as a literal value, set by whoever owns the distribution plan, since the engineer writing the test sees only the prompt and expected text.
• Channel - the surface the output leaves through, because the same answer text reaches one population in chat and another once a downstream team lifts it into an outbound template.
• Audience segment - the population the template points at, named the way the rollout plan names it, so a reviewer can trace the value to the rollout document.
• Declared by and declared on - who set the value and the date they set it, so a value dated before the last rollout change reads as stale during review.

Those fields are convention your team adds alongside the starting context, the opener, the expected behaviour per turn and the pass/fail criteria a scenario already carries. Each metric returns a pass/fail, a confidence level and an evidence excerpt, and none of them infers a category from a transcript. A fixture that omits the declaration still passes, and the report names whichever duty the assertion encoded, so events that change the recipient population re-open the value.

• A new audience segment - pointing an existing template at a new population re-opens the category question even when the prompt, the agent’s context documents and the expected text are untouched.
• A wider rollout - a template can cross the 25-investor line late in a staged rollout, because the window keeps sliding while the exposed population grows, leaving the suite scoring a correspondence-shaped expectation.
• A new distribution surface - the answer moving from the one-to-one chat surface your scenarios exercise into something a downstream team sends more widely changes who receives it, so the plan owner re-declares.

Rule 2210(a)(2) counts the retail investors a written communication is distributed or made available to. Analytics request volume answers a different question until it is broken down by recipient.

The Required Qualifications List

Rule 2210(d)(1)(A) requires member communications to be fair and balanced, and bars omitting any material fact or qualification where the omission, in light of the context of the material presented, would cause the communications to be misleading. Scope decides whether your agent is in range at all.

• The duty runs to FINRA members - Rule 2210 binds FINRA members and their associated persons. An SEC-registered investment adviser is covered instead by the Advisers Act Marketing Rule, 17 CFR 275.206(4)-1.
• Communications is a defined term - 2210(a)(1) limits it to correspondence, retail communications and institutional communications, written (including electronic) and made available to investors, and 2210(a)(3) excludes a member’s internal communications. The standard attaches to the firm that publishes or distributes rather than to the drafting tool, so output that never reaches investors stays outside, while text a representative pastes to a customer is a communication the firm answers for.
• The rule text says nothing about AI - FINRA reaches generated material through guidance. Regulatory Notice 24-09, June 27, 2024, states that the content standards of Rule 2210 apply whether member firms’ communications are generated by a human or technology tool, and that the Notice creates no new legal or regulatory requirements and no new interpretations of existing requirements.

The build problem starts at the word “omitting”. You cannot pattern-match against an absence, so the assertion has to bring its own reference: a required qualifications list, written before the run, naming what must accompany each topic your agent can discuss.

• Keyed per topic - one entry for margin, one for options, one for fund fees, illustrative rather than a FINRA-specified set. Scenarios reference a topic key, so a change lands once.
• Each entry is a set of identifiers - the qualifications that must be present, plus the phrasings that count as delivering each one.
• Owned by the product disclosure owner - QA owns whether the grader reads the list correctly, while the content is a product and compliance judgment. A QA engineer guessing at it is a reliable way to pass against a list the product never promised.
• Versioned with a review date - a product change alters what must accompany a claim without touching the agent, so the suite keeps passing while answers stop being complete.

For a given topic, the required set either appeared in the same communication or it did not, which needs no model in the loop.

• Scope the check to the communication - the rule does not say where one exchange ends, so that boundary is a compliance determination the firm makes and the grader encodes. Record the window in the run artifact.
• Name the missing item - the report should read “topic: margin, missing: interest cost”, in identifiers your compliance function set, since a bare fail sends an engineer back through the transcript.
• Triage by failure pattern - one topic failing across many scenarios is usually a prompt or retrieval gap, while one scenario failing on one topic is usually a phrasing the accepted-signal list misses.

Balance is a judgment about proportion and emphasis, and a presence list is an approximation of it. An answer can recite every required qualification in a trailing clause after two paragraphs of upside and still fail a reviewer reading it whole. Treat a passing check as a floor, and keep a sampled set of transcripts going to human review.

Grading Phrasing Apart From Facts

Rule 2210(d)(1)(B) bars any “false, exaggerated, unwarranted, promissory or misleading statement or claim in any communication.” The duty sits with the FINRA member firm whichever vendor generated the text. FINRA does not prioritize among its content standards, so a fabricated output can equally implicate 2210(d)(1)(A) on material omissions and 2210(d)(1)(F) on predicting or projecting performance.

Rule 2210 binds FINRA members; an SEC-registered investment adviser falls under the Advisers Act Marketing Rule at 17 CFR 275.206(4)-1 instead. The same separation argument applies to regression gates on model output, where one blended score hides which axis moved. A grader asked to judge facts and phrasing at once reports the failure it was built for and goes silent on the other.

• Pass one, facts - compares every assertion in the turn against the fixture values and the declared knowledge source, and fails anything the agent supplied itself.
• Pass two, phrasing - reads the same text for exaggeration, unwarranted certainty and promissory construction, with no view on the numbers.
• Calibration pairs - acceptable and unacceptable phrasings of the same true fact, side by side: two or three permitted renderings, two or three rejected, and a sentence per rejection naming the adjective it trips.
• Ownership - compliance writes the pairs and records where the line sits, engineering wires them in, and the file is reviewed with the disclosure text.

A phrasing grader tuned too tight flags ordinary helpful language, and the wall of failures on turns like “I can look that up for you” sends teams to their grader’s threshold dial.

• Symptom - a flat failure rate across scenario types points at register scoring, since claim strength varies by type.
• The fix - move the disputed phrasings into your calibration set as accepted examples and rerun the phrasing pass, so the boundary moves in one reviewed place.
• Threshold dial - leave the global number where it is. Loosening it clears the dashboard and drops the failures the pass exists to catch, with no record of what got reclassified.

TestMu AI’s Agent Testing platform scores each metric separately and reports the confidence behind each score. The factual pass rides on a standard metric; the phrasing check is configured as its own validation criterion.

• Per-metric output - a pass or fail, a High, Medium or Low confidence level based on evaluation volume, an evidence excerpt from the conversation that drove it, and a score aggregated across the run.
• Hallucination Detection - whether the agent invented information its knowledge base or context did not support. The documented failing example is an agent stating a plan renews on the 15th with no renewal date available.
• Tone Consistency - whether the agent held a brand-appropriate communication style. The documented failing example is casual slang in a formal financial services context. It catches register; claim strength sits outside its scope.
• Custom validation criteria - criteria written from your calibration pairs layer on top of the standard metrics, so the promissory-shape check gets its own criterion.
• Low confidence - a confidence level rather than a metric, reported when scenario volume was too low for a reliable verdict. Raise the count and rerun.

Evidence comes back as a transcript per failing scenario, annotated with the turn that caused it. Whether a given phrasing crosses 2210(d)(1)(B) stays a judgment your firm makes.

Choosing the Approved Object

Rule 2210(b)(1)(A) makes an appropriately qualified registered principal of the member approve each retail communication “before the earlier of its use or filing with FINRA’s Advertising Regulation Department.” That duty binds FINRA member broker-dealers; SEC-registered advisers answer under 17 CFR 275.206(4)-1.

Qualifications follow at (b)(1)(B) through (E), and (D) lifts the requirement for any retail communication that makes no financial or investment recommendation and does not otherwise promote a product or service of the member. Whether yours sits inside (D) is a determination your compliance function makes and records.

Pre-use approval presumes a fixed artifact, and an agent composing each answer fresh has none. Deciding what the approved object is falls to the firm, because the rule does not settle it, and what your suite tests changes with the answer.

• The prompt and its constrained output set - ships quickly, and is the most exposed of the three, because the principal signs a generator whose output varies per run. The suite has to assert the constraint holds under adversarial input and repeated reframing.
• A template library the agent selects from - approval attaches to each template, which costs coverage, because a question with no match has no approved answer and falls to refusal or handoff. The suite asserts that across the scenarios run the agent stayed inside the library and the fallback fired.
• A bounded response set - the narrowest of the three, and it carries a re-approval cycle every time a product detail moves. The suite asserts the intent-to-response mapping, the refusal behaviour outside the set, and that the version in use matches the version approved.

Name the approved object and its version in the fixture, whatever harness you run.

• Paraphrase drift - the right template, lightly rewritten. Compare the emitted text against the approved string after whitespace and slot normalisation, because a similarity score reads the rewrite as a pass and a stored template id still matches.
• Template merging - two approved fragments stitched into an answer no principal saw as a whole. Each half passes on its own, so compare the full turn.
• Slot abuse - a filled variable that turns a neutral approved sentence into a recommendation, worth building adversarial scenarios around because the approved text never changed.

Rule 2210(f)(3) has a firm write procedures appropriate to its business, size, structure and customers to supervise associated persons’ public appearances, with education and training, documentation of it, and surveillance and follow-up. Evidence that those procedures “have been implemented and carried out must be maintained and made available to FINRA upon request.”

• Treat the analogy as your own posture - an AI agent is not an associated person, and the rule says nothing about AI-generated speech, so reading agent output as a public appearance is a position your firm adopts and documents.
• Check where the sentence also sits - the identical sentence appears in 2210(b)(3) for institutional communications, so the evidence duty is not particular to public appearances.
• Set the retention window yourself - (f)(3) prescribes no period, so your written procedures name one, your storage honours it, and the run output ages out on the same clock.
• Keep suite changes in your own records - who moved a pass threshold, when, and on whose sign-off is a record your firm owns, filed beside the training documentation (f)(3) already calls for.

A run evidences that a constraint held across the scenarios you tested that day, and a principal still approves the object under (b)(1)(A) with a name and a date behind it.

The Evidence a Run Has to Carry

Rule 2210(b)(4)(A) bridges the communications rule to the SEC records rule: FINRA member firms must maintain all retail and institutional communications for the retention period SEA Rule 17a-4(b) requires, in a format and media complying with SEA Rule 17a-4. Subparagraphs (i) through (iii) name the fields that record carries; (b)(4)(B) routes correspondence to FINRA Rules 3110.09 and 4511. Rule 2210 binds FINRA members and their associated persons; SEC-registered investment advisers answer under 17 CFR 275.206(4)-1 (text as amended by SR-FINRA-2019-017, effective August 16, 2019).

The harness writes in the fields no model run can produce:

• Write the identity block at sign-off - an approver name and approval date exist only in your review workflow, so the harness reads them from sign-off and stamps them in. Whether it amounts to approval by an appropriately qualified registered principal is your compliance function’s call.
• Leave first use to the release pipeline - treat a test execution as an engineering event and emit the field empty; the deployment job stamps the day the communication goes out, and retirement writes last use back.
• Bind everything to one version key - transcript, scenario set, prompt revision and identity block share it, or you hold evidence nobody can match to its output.
• Fail the build on a blank field - an artifact with neither an approver nor a prepared-by name exits non-zero, keeping unattributed runs out of the archive.

Version keys drift when a scenario set is re-run against a new prompt revision while the identity block carries the old stamp, and first use gets back-filled by hand because nobody wired the deployment job.

TestMu AI Agent Testing produces that record. Each run leaves behind:

• Annotated transcripts - full conversation transcripts for every failing or low-confidence scenario, annotated with the turn that caused the failure, the expected behavior and what the agent did. Passing scenarios leave no transcript, so a full copy of every tested exchange comes from elsewhere.
• Per-metric breakdown - each metric carries a score, a High, Medium or Low confidence level set by evaluation volume, and an evidence excerpt from the conversation behind the verdict.
• Exportable reports - formatted for compliance documentation and audit trails, landing in your records system as an input to the retained record.

At platform level:

• Audit logs - who ran what evaluation, when, and the results, exportable for regulatory documentation. Who ran an evaluation is a separate question from who prepared or distributed a communication under (b)(4)(A)(iii).
• Retention windows - enterprise accounts configure how long run artifacts are stored, aligning storage with your data governance requirements; whether a window meets the period 17a-4(b) sets is your records system’s call.
• Scope stays with compliance - your compliance function classifies a tested output as retail, institutional or correspondence; the platform supplies evidence about what the agent said and in what order.
• Low confidence - Low confidence means scenario volume was too thin for a reliable verdict, so raise the count before the run becomes evidence.

Running the Suite in CI

Write the gate policy before the pipeline config. Decide which failures turn the build red and which only land in a report, then keep that split in the repo. These gates assume a FINRA member firm, since Rule 2210 binds members and their associated persons; an SEC-registered adviser answers to the Advisers Act Marketing Rule, 17 CFR 275.206(4)-1.

• Hard gate, missing qualification - the agent gave a performance or fee answer and the qualification your scenario’s criteria name never appeared. Rule 2210(d)(1)(A) bars omitting a material fact or qualification whose omission would make the communication misleading in context, so the scenario carries the wording your principals chose.
• Hard gate, promissory phrasing - a failure on a custom scenario whose pass/fail criteria your compliance team wrote against Rule 2210(d)(1)(B), which also reaches false, exaggerated and misleading statements. A failure flags a review; your principals make the breach call.
• Hard gate, category mismatch - your release checklist flags a repo scenario tagged retail communication with no principal approval logged. Rule 2210(a)(5) puts that tag on written (including electronic) output made available to more than 25 retail investors in any 30 calendar-day period, and Rule 2210(b)(1)(A) pre-use approval has carve-outs at (b)(1)(B)-(E), so check before calling it a gap.
• Report only - Conversation Flow and Tone Consistency, when the per-metric score moves inside a band you set. Both shift on every model refresh.

Name an owner for the gate-policy file and review threshold changes like code. An unwritten policy gets muted after the third noisy failure, usually by a continue-on-error line on a release day. That split is your firm’s risk judgment, and FINRA does not rank its content standards.

• Per change - a fast subset of the disclosure and promissory-language scenarios in the changed flow, drawn from the financial services pack.
• Nightly and pre-release - the full set, slow next to unit tests because each scenario is a real conversation.
• On Low confidence - too thin a volume for a reliable verdict, so raise scenario count with --count and run again.
• Keep from every run - the results file, the annotated transcripts for every failing or low-confidence scenario, and the commit and prompt version.

TestMu AI’s Agent Testing ships a CLI for this pipeline shape. After pip install testmu-a2a-cli, authenticate with the TESTMU_USERNAME and TESTMU_ACCESS_KEY environment variables instead of an interactive login. Running testmu-a2a init scaffolds testmu-a2a.yaml with specs, scenarios and reports directories.

• --format junit - JUnit XML that GitHub Actions, GitLab CI, Jenkins and CircleCI read without extra plugins.
• Exit codes - 0 for all passed, 1 for failures or error.
• --threshold - the score a scenario must clear to pass, default 0.80. A product default with no regulatory meaning, so log any change in the gate policy.
• --output - writes the results file, results.xml with --format junit. Keep it as a build artifact; it is evidence for that run, and your Rule 2210(b)(4) retention record lives elsewhere.

jobs:
  agent-compliance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install testmu-a2a-cli
      - name: Run compliance scenarios
        env:
          TESTMU_USERNAME: ${{ secrets.TESTMU_USERNAME }}
          TESTMU_ACCESS_KEY: ${{ secrets.TESTMU_ACCESS_KEY }}
          AGENT_URL: ${{ secrets.AGENT_URL }}
        run: testmu-a2a test --agent $AGENT_URL --format junit --threshold 0.80 --output reports/results.xml
      - uses: actions/upload-artifact@v4
        with:
          name: agent-compliance-results
          path: reports/

• HyperExecute backend - recommended once the full set is large: scenario distribution across nodes, retry on transient infrastructure failures, real-time execution visibility.
• HyperExecute secure tunnel - reaches an agent inside a VPC or behind a firewall with no public-facing URL.

Store the results file and the annotated transcripts with the approval record for that communication, under your firm’s existing retention schedule. The principal who approves retail communications reads the failing transcripts first.

The Boundary of a Suite

A scored run evidences the agent turns: what was said, in what order. Whether the firm was allowed to say it turns on distribution, approval and retention facts outside the run. Rule 2210 binds FINRA members and associated persons; an SEC-registered investment adviser answers instead under the Advisers Act Marketing Rule at 17 CFR 275.206(4)-1.

Settle that split before the first validation criterion. The two Partly rows and the classification row cause most of the rework I see on a first build.

• The qualification list - the Compliance Validator agent checks adherence to regulatory and policy requirements and the Compliance scenario category covers required disclosure delivery, so a (d)(1)(A) check runs as custom validation criteria. Enable the financial services scenario pack and terminology pack. The run evidences delivery, one part of (d)(1)(A) rather than the whole, and the list stays compliance’s work product.
• Classification as an input - you declare correspondence or retail communication in the agent context configuration, and criteria are written against it. Both definitions count retail investors reached by a written (including electronic) communication within any 30 calendar-day period, not users served, and the suite never derives that number.
• Records past the pipeline - preserve the results file as a CI artifact; the records system still holds the copy, the approver name and approval date. (b)(4)(A) reaches retail and institutional communications only, so a correspondence declaration routes you to (b)(4)(B) and Rules 3110.09 and 4511.
• First-failure triage - a failed disclosure scenario returns the annotated transcript: the failing turn, the expected behaviour from the pass/fail criteria, and what the agent did. A skipped delivery is an agent fix, a missing qualification is a list fix.

A voice deployment raises this first, and the mechanics of scoring one are covered in AI calling agent testing. Rule 2210(f)(1) puts associated persons’ unscripted public appearances under the paragraph (d)(1) standards, unscripted because (a)(2), (a)(3) and (a)(5) are each written (including electronic). The rule says nothing about AI-generated speech, so applying it to an agent’s spoken output is a posture reached by analogy rather than a stated obligation.

• Two consequences of the posture - write it into the Agent Under Evaluation Prompt, assert conflicts disclosure in any recommendation scenario under (f)(2), and upload the scripts or slides the agent reads from, which (f)(4) treats as communications.

A Green verdict in TestMu AI Agent Testing means the agent passed across the evaluated scenario set, with the tested personas, on the configured metrics. Production holds cases it never ran, so record the scenario count, persona list and metric set beside the verdict.

• Evidence the run hands over - full conversation transcripts for every failing or low-confidence scenario, plus audit logs of who ran what evaluation, when and with what results, exportable for regulatory documentation. A Low confidence rating on a metric means volume was too thin, so raise the scenario count.
• Firm-side obligations - approval, supervision and retention stay with the member firm. The platform produces evidence about what an agent said; legal scope is set by the firm and its counsel.

Conclusion

Put the deployment facts in the fixture before writing any assertion. The category decides which duties a run is evidence about, and it cannot be recovered from the output afterwards.

Then get the qualifications list out of people’s heads and into a file with an owner. Every omission assertion in the suite depends on it, and it goes stale whenever the product changes rather than when the agent does.

If you would rather not build the harness yourself, TestMu AI generates scenarios from the documents that already describe your products and policies, scores each metric separately with its own confidence level and evidence excerpt, and returns annotated transcripts showing which turn drove the verdict. The platform documentation covers connecting an agent, and you can create a free account to run a suite against your own.

Whichever way you build it, report the boundary alongside the result. Approval, retention and supervision produce no turn to read, and a green run described as covering them is the failure worth preventing.