System Prompt Design for Voice Agents [Testμ 2026]

Voice Is Less Forgiving Than Chat

Writing a system prompt for a voice agent is not the same job as writing one for a text model. A few words change how the agent handles hesitation, interruptions, tool calling, and even how its personality reads.

The reason it matters more is the medium. People extend far less patience to a voice agent than to a chat window, and a poor experience registers immediately rather than after a paragraph.

Her evidence base for this is volume. Vapi has processed over a million calls, and what she keeps seeing is teams making small changes to an assistant that turn out to have large consequences.

Most people now write these prompts with a coding agent, which she does too. That makes understanding the parts more important rather than less, because you need to know which areas to control and what to look for when the output shifts.

The Conversational Operating System

Her framing for the system prompt is that it functions as the whole conversational operating system. It is not the only part of the conversation, and it coordinates most of what the caller experiences.

Personality lives there, along with state, workflow design, and the rules for tool calling: what the assistant has access to, when it should reach for a tool, and what a call should look like.

Recovery belongs there too, meaning what the assistant does when things are not working. Callers never see any of it, and they experience every one of those decisions at every point in the call.

Anatomy of a Good Prompt

After analysing a large volume of calls, the same sections keep appearing in the prompts that work.

• Personality - how the assistant feels and sounds. Current voice models supply a great deal of empathy and emotionality without being forced to, so the design question is what kind of assistant the use case wants, warm and conversational or dry and to the point.
• Knowledge - what the assistant knows and is there to do. For a restaurant booking agent that means the restaurant details, operating hours, common questions, and when it is allowed to book.
• Workflow and tools - not just which tools are attached, but under what circumstances each should be called, and which keywords should always trigger one.
• Examples - patterns for how to behave in specific scenarios. The best examples are often unknown at launch and emerge from conversation data once real users arrive.

She flagged one dependency running underneath all of it. Tool calling is heavily affected by model choice, because different models interpret the same prompt differently.

Three Agents, One Script

The demo holds everything constant except the prompt. Three restaurant booking assistants, the same scenario, the same two tools for checking availability and making a reservation, and the same test script.

The model stack was identical across all three, running on GPT-4.1 with Vapi voices. Any difference in the calls is therefore attributable to the prompt rather than the setup.

The three prompts represent three instincts most teams have had. One optimises for personality, one for strict procedural control, and one applies the section structure above.

Simulations, Not Manual Calls

Rather than dialling each agent herself, she ran simulations, which is the part of the session that generalises beyond any one platform. Manual calling is what most people do while building, and it is neither structured nor consistent.

A simulation defines the personality and behaviour of the tester, then a scenario holding the caller instructions and the script. The two run an end-to-end call against each other.

What comes back is a transcript, every tool call, evaluations against the criteria you set, and a pass or fail. There are two modes, and the distinction matters.

• Chat simulation - good for confirming that tools are called at the right points in a conversation and that the ordering is logical.
• Voice simulation - a fully simulated call, which is the only way to test interruption, turn-taking and the actual conversational flow.

The recordings she played are both sides simulated, agent talking to agent. Her own name appears in the caller script, which is how you can tell the human was never on the line.

The Personality Prompt

The first prompt spends most of its length on how the assistant should sound, instructing it to use personal anecdotes and to be warm. She was clear there is nothing wrong with wanting that.

The clip she played sounds fine. The agent says it will reserve a table for six at eight, notes the wife’s birthday, the stroller and the shellfish allergy, and asks for a phone number.

When she called it manually it booked the reservation, the conversation was pleasant, and everything worked. What it never does is confirm anything back to the caller.

Across many iterations that becomes a data problem rather than a style problem. A phone number heard wrong, a time slightly off, a party size misread, and none of it caught.

The Rigid Prompt

The second prompt is the obvious correction. Collect and verify everything in order, name first, then number, then reservation, with the requirements laid out in strict numbered lists.

On the page it looks like careful engineering. The model has explicit ordering, nothing can be skipped, and no information can be lost.

In the call it reads the structure out loud. The agent announces that it will confirm each detail step by step, says the word one, asks for the phone number, repeats it back, then moves to the next of eight items.

It produced the longest call of the three and, in her assessment, an unusable one. Writing rules at an LLM the way we tend to reads as reasonable in text and lands badly in speech.

The Best Practices Prompt

The third prompt uses the section structure and still asks for every value to be tracked. What changes is that rules and workflows replace fixed ordering.

She stressed how small the edit is. The instruction set is close to identical, and what differs is how the information is presented to the model, which is why the same models produce a different call.

The result is an assistant that still sounds conversational while confirming what needs confirming. It gets the same information without narrating its own procedure.

The Next Friday Test

The ambiguity was planted deliberately. She wrote next Friday into the caller script on a Wednesday, so the phrase legitimately points at two different dates.

The best-practices agent stopped and asked whether next Friday meant Friday the 21st. The caller said no, and gave the 28th.

It was the only one of the three that asked. The other two resolved the ambiguity themselves and carried on.

Her objection is not to inference in general but to inference at the booking. Guessing a date, an availability window or a party size means a real customer arrives on the wrong night.

The Outcomes

Each assistant ran three simulations. The personality agent failed every criterion she had set, because it performed no confirmation at all.

The reason that matters is what a manual test would have told her instead. Ask for next Friday while privately meaning two days from now, have the model land on the same reading, and the agent looks correct.

The second result is the one that undercuts the usual trade-off. Tuning the prompt to best practices produced not only the best quality but the most efficient call of the three.

She measured that two ways, on median characters generated as a proxy for conversational volume and on call duration, and the best-practices agent won on both. Less chatty, job done, and a defensible claim that the tools were actually called.

The dashboard walkthrough showed how the criteria are written. One of hers checks that the assistant catches a correction mid-sentence, because the script says four and then changes to six.

A Layered Test Loop

Her recommended loop runs in three stages, each catching something the previous one cannot.

• Manual demos - while building, to check that the thing works at all.
• Chat simulations - conversation without the voice layer, confirming that tools are called correctly and the ordering is logical.
• Voice simulations - the full call, where interruption and turn-taking finally get exercised.

Her instruction on criteria is the most reusable line in the talk. Make them atomic units, with pass and fail unmistakable, covering everything that matters to the business case.

Beyond simulations she layers two more checks. Traditional evals sit alongside them, and because the transcript is retrievable programmatically, a script can walk it and assert on specific things deterministically.

How many layers you need is a risk question. The tolerance differs by industry, and she suggested starting with voice simulations and adding the rest according to what going wrong would cost.

Her closing advice on the stack is easy to overlook. Keep it consistent while testing, because changing models changes results, and if you intend to support several then test against several.

Q & A Session

Two audience questions closed the session, and both answers pointed at the same layer.

• How should a voice agent handle interruptions and overlapping speech?

  Amanda Martin: Start with model choice, because some models already handle end-of-speech detection and turn-taking well. She named the Deepgram Flux models as strong here and believed Cartesia handles it well too, while noting she was recalling from memory. Once you know what the model provides, tune the rest at the platform layer. If an agent is interrupting too soon, check the platform controls, and equally check that the model provider’s own turn-taking behaviour is configured correctly rather than assuming it is.

• Are there tricks for handling accents or multiple languages through the prompt alone?

  Amanda Martin: The prompt can carry pronunciation examples when the agent is saying particular things wrong. The accent itself she does not believe is solvable there. This is another decision that starts at the model provider: pick the model that handles the languages you care about best, then work with pronunciation dictionaries and examples for the nuanced parts of the conversation.

Her takeaway pairs the two things the three agents kept separating. Personality is worth spending time on, and it has to be paired with boundaries covering what the agent may do, how it gathers information, and what triggers a tool call.

With clear success criteria and repeatable simulations, a one-line prompt change stops being a gamble. You can make it and know whether it moved the experience for your callers.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.