AI Testing Data Security: What SOC 2 Actually Covers

What an AI Testing Tool Actually Sees

Security reviews of testing tools tend to focus on the login and the network path. The larger exposure is the artifacts a run produces, because a test suite deliberately drives an application into its most data-rich states.

A checkout test has to reach a filled cart. An account test has to reach a populated profile. Whatever is on that screen lands in the recording.

Map these five rows against your own pipeline before reading any vendor's report. The exercise tells you which SOC 2 scope questions are worth asking and which are irrelevant to you.

What SOC 2 Actually Covers

SOC 2 is an attestation that an independent CPA firm examined a service organization's controls against the AICPA Trust Services Criteria. The criteria were established by the AICPA Assurance Services Executive Committee, and they define five categories.

Only security appears in every report. The other four categories are scoped in by the service organization, which is why two vendors can both display a SOC 2 badge while attesting to very different things.

The AICPA is direct about this. Following allegations about a compliance vendor's SOC practices, it stated that SOC services should be thoroughly evaluated by service organizations and CPA firms. Treat the badge as the beginning of your review.

Where SOC 2 Stops for AI Testing Tools

The Trust Services Criteria are technology-neutral by design, which is a strength for infrastructure and a limitation for AI. A criterion about restricting logical access applies equally to a database and to a model endpoint, and says nothing specific about either.

Four questions sit outside what a SOC 2 report answers, and each one is decided by vendor policy rather than by the audit:

• Model training use - whether your prompts, application content, and generated tests are used to train or fine-tune any model. No Trust Services Criterion asks this, so a clean report is compatible with either answer.
• Prompt and inference logging - what is written to the model provider's logs, how long it survives there, and whether redaction happens before or after logging.
• Subprocessor model providers - which third-party LLM the feature calls, and in which jurisdiction. This is disclosed in the subservice organization list rather than tested as a control.
• Scope carve-outs - whether the AI features you intend to use were inside the examined system boundary at all, or were added after the observation period closed.

The OWASP Top 10 for LLM Applications 2025 list ranks Sensitive Information Disclosure at LLM02, Supply Chain at LLM03, and Data and Model Poisoning at LLM04.

A testing tool that reads your application and writes to a model touches the top of that list by design. The ranking is a useful agenda for a vendor call, because each entry converts into a question the vendor has to answer concretely.

The same list ranks prompt injection at LLM01, which is a testable property rather than a policy question. Our guide to prompt injection testing covers how to probe for it in your own application.

Standards That Cover the AI Gap

Three frameworks address what SOC 2 leaves open. They are complements, not alternatives, and a vendor that cites all three is describing a genuinely different posture from one that cites a single badge.

ISO/IEC 42001 is the one to lead with, because it is the first AI management system standard and the only one of the three that an independent body can certify against. ISO does not certify organizations itself, so ask which accredited body issued the certificate.

The NIST AI Risk Management Framework was released in January 2023 and is intended for voluntary use.

A vendor describing itself as NIST-aligned is therefore reporting an internal practice, not an external assessment. Draw that distinction out in the call, because the two carry very different weight in a review.

Questions to Ask an AI Testing Vendor

Vendor questionnaires fail when they accept a yes. Each question below has a weak answer that passes a checkbox and a strong answer that is verifiable, and the difference is what a security review should be looking for.

The retention row is the one most teams skip and most regret. A 30-day video retention and an indefinite one produce identical answers to every other question on the list.

How to Reduce Test Data Exposure

Vendor selection is half the control. The other half is what you send, and it is the half you own regardless of which platform you pick.

• Mask before the agent runs, not after - an artifact is created the instant a screen renders, so redacting logs afterwards leaves the video and screenshots untouched. Techniques for this are covered in our guide to data masking.
• Generate seed data rather than copying it - a synthetic account exercises the same code path as a cloned production record without carrying a real person's details into the pipeline.
• Scope secrets to the job - inject short-lived, least-privilege credentials for test runs so an echoed environment variable in a failure dump expires rather than persisting.
• Keep firewalled applications behind a tunnel - staging environments holding real data do not need public exposure to be tested against a cloud grid.
• Set retention deliberately - shorten it where you can and extend it only where debugging genuinely requires the history.
• Separate the regulated suite - the tests touching payment or health data can run under a stricter configuration than the rest, and our notes on GDPR compliance testing cover what that configuration needs to prove.

These six controls also make the vendor conversation shorter. A team that already masks its fixtures has far fewer questions riding on the answer about model training.

For test authoring specifically, KaneAI keeps MFA-protected flows testable by generating time-based one-time passwords from a stored secret key, so a shared authenticator device never enters the pipeline. Tunnel support covers locally hosted and firewalled applications without exposing them to the public internet, proxy configuration controls how test traffic is routed, and network data masking keeps tokens and PII out of the stored request and response logs.

How to Verify What a Vendor Tells You

Every claim in this section was checked against a live page rather than a sales deck, which is the same method your review should use. A vendor that publishes its posture openly can be verified in minutes; one that only answers under NDA cannot.

Three checks separate a documented posture from a marketing claim, and all three can be run before you ever speak to a sales engineer:

• Find the certification scope in writing - the TestMu AI security trust page names SOC 2 Type II, ISO 27001:2022, ISO 27017:2015, and ISO 27701:2019, alongside AES-256 encryption at rest, TLS 1.2 or higher in transit, a per-customer virtual private cloud with a unique tenant ID, and annual external penetration testing by third-party audit firms.
• Find the AI-specific statement separately - certifications and AI policy live in different places. The AI trust page states that inputs and outputs are never used to train any LLM models, and describes an ISO/IEC 42001 aligned AI management system.
• Find the retention schedule as a document - the published data retention policy sets a 30-day default for test builds, video recordings, console and network logs, screenshots, uploaded apps, and audit logs, keeps SmartUI screenshots for 1 year, retains Test Manager and KaneAI entities for as long as the account is active, and states that data is permanently deleted after the retention period expires.

Run those three checks against any shortlisted vendor. Where a page does not exist, that absence is itself the finding, and it belongs in the review notes rather than in a follow-up email.

For teams whose obligations rule out shared infrastructure entirely, a private real device cloud moves execution onto dedicated hardware, which changes the data-residency conversation from a policy question into an architectural one.

Getting Started

Open your current AI testing vendor's trust page and look for three documents: the certification scope, a written statement on model training, and a per-artifact retention schedule. Most reviews stall because the second and third were never requested.

Then map the five artifact types from the first section against your own pipeline and mark which ones carry regulated data today. That map converts a generic questionnaire into a short list of questions that actually decide the purchase.

If you are building the broader programme around this, our guides to security testing and compliance monitoring cover the continuous side, where controls are verified on every build rather than once a year. For finance and healthcare teams specifically, our breakdown of test automation requirements for regulated industries covers the BAA, DPA, and audit-evidence questions this section only touches on.

Note: AI assistance was used in researching and drafting this article. Sawan Garg, Senior Vice President of Engineering at TestMu AI, whose listed expertise includes Platform Engineering and Distributed Systems, verified every statistic, link, and product claim against primary sources before publication. Sources cited are from primary standards bodies (AICPA, ISO, NIST, and OWASP). Read our editorial process and AI use policy for details.