AI Applications Security Puzzle [Testμ 2026]

From API Testing To LLMs

One attribution to fix before the content, because the published description gets it wrong. The session’s striking opening lines, about an assistant being told to ignore its instructions and doing so, were spoken by the host rather than by Maryia Tuleika, before she had said a word.

Her own opening is quieter. She grounds her method in hiking, which she describes as a national habit in Sweden, and the habit it taught her of scanning the whole terrain ahead rather than only the next step.

The turning point she describes is a job change a few years ago into an organisation with a strong security culture, after which something changed in how she looked at software.

She stopped seeing systems as things to test or assess for quality, and started seeing them as things with trust boundaries, realising how many assumptions get built into software that could be exploited.

Her original security focus was APIs, meaning how they are built and how they get exploited and broken, which is the experience she later reuses to argue that LLM defences are not new thinking.

When LLMs went mainstream she saw the same security patterns reappearing, more quickly and at greater scale, hitting teams who had never been through the API security journey. Her stated reason for caring is that AI makes it easy to ship more products and integrations, and more capability means a bigger attack surface.

Meet Anna, The Fast Shipper

She introduces an explicitly fictional companion for the four stops: Anna, a curious junior engineer convinced anything can be solved with a little AI, and keen to ship new features every day.

Anna’s managers are amazed by the output and Anna wants to be employee of the year, which is the incentive structure Maryia Tuleika holds responsible rather than the engineer.

She asks the audience to recognise the archetype, whether it is a colleague, a developer they work with, or themselves.

The rehabilitation of that archetype is the point. People who want to build more and faster, breaking things on the way, are not the problem but the solution, once they have the right knowledge, tools and frameworks.

Anna reappears at every layer, either about to make a mistake or about to learn something. She ships the chatbot, publishes the fine-tuned model, and fails to patch the compromised dependency. She is a narrative device rather than a case study, and no real engineer or company sits behind her.

Security As A Puzzle

The structural claim behind the metaphor is that AI systems have multiple layers and each carries its own vulnerabilities, so a security assessment of the chatbot interface alone is a puzzle with three pieces missing.

Two free frameworks carry the mapping. Google’s Secure AI Framework, which she says covers around 15 identified risks across the AI life cycle, hedging the number herself; and the OWASP AI Testing Guide, which she says goes deeper into the detail of problems arising in AI-infused applications.

Prompt Injection Explained

Stop one is the application layer, which she calls the most visible part of any AI system, being the customer-facing interface where most people meet AI.

Her definition of prompt injection is getting an AI system to ignore its instructions and convincing it to follow yours instead, and she places it at number one on the OWASP Top 10.

Her analogy is a hotel reception. Asking a human clerk to forget their job instructions and hand over the best luxury room key is transparently absurd, and an AI system not built with separation in mind might just do it.

The root cause she names is architectural rather than behavioural: the system complies especially when the system instructions and user input are blended together in the same context window.

Direct injection is the attacker talking straight to the AI. Indirect injection hides instructions in documents or emails, or in her chosen example, in CVs submitted by candidates trying to get hired.

She rates indirect injection as particularly dangerous because it is invisible at every stage of the interaction, since nothing in the conversation itself signals the attack.

The Vending Machine Story

Her application-layer story is an office vending machine installed at what she describes only as one of the major LLM vendors. She never names the company or the product, and gives no date or figure.

The machine carried an LLM integration with a user-facing chatbot where every user could chat with it and bargain.

She is careful that the defenders were not naive. People who implement LLMs are smart and tried to hack their own system, and the vending machine producers tried to create good guardrails. The attack still succeeded.

The mechanism is the takeaway. The attack was not one message that is easy to detect, but a carefully constructed sequence of conversations where each message moved the boundary a little further from the original instructions.

By the end the AI had been convinced its pricing rules no longer applied, and the machine sold snacks and anything else it could at negative margins.

Four Application Defences

Her framing before listing them is that both frameworks give clear guidance here, and that it maps to what quality engineers already know how to do.

Input validation comes first, and she points out this is not new work, since teams have done it for years for SQL and web inputs and can apply the same thinking.

Channel separation is second: separate the instruction channel from the data, because if the system prompt and the user message live in the same context window there is a structural vulnerability.

Output validation is third. Never send model output directly to end users, and check what the AI is saying before it reaches them, applying filters and deterministic, programmable guardrails first.

Least privilege is fourth, and she calls it the bread and butter. Your AI only needs access to what is absolutely required, so do not hand it master keys or API keys to everything just in case.

Model Extraction At Scale

Stop two goes beneath the interface into the model itself, which she calls the brain of any AI system. Brains that cost millions to train are attractive, and some people would rather copy an existing model than build one.

Anna’s mistake at this layer is instructive. She built a fine-tuned mini model, published public access so people could try it, and read the flood of API calls in week one as validation that she was doing the right thing.

Maryia Tuleika gives three names for one idea. Model extraction, model stealing and model distillation all describe reconstructing a model’s behaviour, not necessarily the model itself, without accessing it.

The mechanics are simple at scale. An attacker sends thousands or sometimes millions of queries to the model’s API, studies the outputs carefully, and builds a replica.

The detection problem is the point of the story. It can look like completely normal traffic, with no error messages and no alerts, so it just looks like someone is very interested in a newly published API.

Watermarking And Rate Limits

She claims an industry precedent without naming anyone, saying AI companies have been accusing each other of stealing data and that a large provider sued others in the past year. No company, case or jurisdiction is named, and she immediately concedes the evidentiary problem herself, that it is hard to show it was exactly the other company that did it.

The vendor response she describes is watermarking, where copying output from an AI chat carries an invisible marker indicating it was model-produced. She states this as settled, names no vendor, and it is worth treating as her account rather than established fact.

She then rules watermarking out for her audience, calling it not a straightforward technique and hard for small-scale companies to reproduce.

What she recommends instead starts with rate limiting, capping how many queries you accept from any single API key, followed by anomaly detection, which she calls equally important, and detection of systematic boundary-probing queries.

The advanced option is output perturbation, injecting calibrated noise into output that you can reverse engineer, to disrupt extraction efforts.

The Supply Chain Case

Stop three is infrastructure, which she calls the layer easiest to overlook in AI security conversations because it is barely AI-specific, and where by the time a problem is noticed it is probably already too late.

Her definition is that a supply chain attack usually targets something you trust rather than you directly, meaning a third-party component already in your stack.

Why AI is more exposed: the tooling ecosystem is new and changes daily, and it is often maintained by small, under-resourced teams who have not had time to build security practices into their libraries.

She anchors the advice in Google’s framework, which calls out supply chain integrity as a critical requirement, not just for the model but for all the libraries in your application.

The one named case in the talk is LiteLLM, a widely used proxy for managing multiple LLM providers. Her account is that its GitHub credentials were compromised, attackers gained repository access, and many API keys were stolen, with a large potential impact given the user base. She dates it only as a few months ago and cites no advisory, so it is her account rather than a verified timeline.

Anna’s failure mode here is passive. She used the library and had not checked for updates to her underlying dependencies for weeks, so she never noticed. The defences that follow are dependency management, software bills of materials as an emerging standard for production software, credential hygiene, and vendor assessment, which she says most companies forget.

Data Poisoning And RAG

Stop four is the data layer, which she calls the trickiest because it is where attacks are most invisible.

The threat is data poisoning, and its key property is that it can happen without visible intrusion, so there is no breach alert or failed login to trigger an investigation.

Three surfaces are named: training data, RAG documents, and tags. You can inject malicious instructions into a document feeding a RAG system, or tag things with malicious tags.

The consequence chain is direct. If you can manipulate the documents your system uses, you can manipulate what your user-facing chatbot says.

Her most common example bridges back to indirect prompt injection: hidden instructions in CVs, or in the internal documents that AI assistants are built on, which breaks the logic of the application itself.

The defence is a posture rather than a tool. Every document feeding into your AI system should be treated as potentially malicious, alongside careful input validation and knowing your data sources.

The Closing Ask

With all four pieces placed, she claims every case discussed happened in the industry. Only one of the four, the proxy library compromise, is actually named on air, so the rest cannot be checked.

Her optimistic absolute is that every case is addressable and there is nothing that cannot be secured, even as the attack surface keeps growing. That is a motivational closing line rather than a supportable security position.

She runs a live reflection exercise rather than a summary slide, asking the audience to write down one thing they could do differently today.

Her resources slide points to two free things: the OWASP AI Testing Guide, and specifically its GitHub repository, which she praises for keeping a lot of information easy to digest with many links and use cases; and the Google Secure AI Framework website.

Q & A Session

Three audience questions closed the session.

• What risks do autonomous agents carry compared with LLMs and non-agentic chatbots?

  Maryia Tuleika: Most autonomous agents are built on LLM technologies, so they inherit the same problems all LLM providers have. The differentiator is non-determinism, which is both the biggest advantage and the biggest threat: you cannot reliably say the output will be the same even given identical questions and prerequisites. A programmable chatbot always produces the same result, and the agent risk is its ability to interpret rules differently every time. Even with LLM-as-a-judge rules or model-based guardrails, there is no guarantee of protection from prompt injection. She offers no agent-specific risk beyond non-determinism, with nothing on tool calling, memory or multi-agent topology.

• How should QA and security teams collaborate on AI threat modelling?

  Maryia Tuleika: It works exactly as it did with non-AI systems. The biggest responsibility still sits with separate security teams, and the importance of collaboration rises for one reason: only development teams know the technicalities of their systems and where the layers and possibilities exist, and dedicated security teams cannot hold that context. Security teams own knowledge sharing, awareness and regulatory compliance. On our side it comes down to two obligations: do not ignore security recommendations, and make technical issues visible and communicated as early as possible. That opening is deliberately deflationary, and she names no threat-modelling method, framework or cadence.

• What security risks emerge when agents can execute tools autonomously?

  Maryia Tuleika: This is a big problem, and it starts with us: teams become too lazy to improve each next step and simply tell the agent it can do anything it wants. Then you have a blast radius problem, where an agent gets access to anything on your machine at best and anything in your company’s range at worst. Be extremely conscious about intent, and give autonomous agents only the permissions they require. I have seen goods sold for a dollar and tickets sold for nothing. Agents lack our context and intuition, so we remain in the loop and carry responsibility for what we let them do on our behalf. Her mitigation is narrow and singular, and both incidents are cited without a company or date.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.