From Load Testing to Reliability Engineering [Testμ 2026]

One Question

The whole session hangs on one question that sounds simple and rarely gets a confident answer: will this change be safe in production?

Teams run the load tests, collect enormous quantities of data and produce reports many pages long, and still hesitate before going live.

His diagnosis is that running more load tests does not improve the answer, because it is not a testing problem. It is an engineering feedback problem.

A load test can simulate users, execute scripts and generate metrics. What it cannot do on its own is tell you whether the workload was realistic, whether you ran the right test at all, or whether the result clears the bar for production. His conclusion is that the valuable output is not the report but the feedback.

The Limits of Manual Triage

He has been at ING for over twenty years and describes his purpose as giving teams the confidence to go to production.

The scale figure he reads off a corporate slide is 350,000 deployments a week, with the caveat that the slide is a year old and the number is probably higher now. It is a self-reported figure, and he uses it to make a point about cadence rather than to claim a benchmark.

What that cadence rules out is the traditional model. Performance engineering cannot depend on a small number of specialists interpreting every graph by hand.

The numbers he gives sit at different scopes and are worth keeping distinct: distributed tests run across more than a hundred microservices at a time, while knowing which dependencies actually matter spans an estate of around 1,300. Manual triage at that shape consumes the whole day, then starts again tomorrow.

Data Versus Decisions

His central distinction is one line: a load test produces data, and reliability engineering produces decisions.

He draws it as an iceberg. Above the waterline sit the visible things, test scripts, the load generator, a dashboard, response times. Below it sit objectives, workload modelling, observability, dependencies, capacity management, analysis and production feedback.

Weak foundations and the whole thing is a Titanic, in his phrasing: the change will not reach production without incidents.

His reframe is that performance testing has to be treated as a system rather than a script. Reliability emerges when customer journeys, measurable objectives, realistic loads, observability and resilience are evaluated together.

And the hard part, he insists, is not pressing the button or starting the pipeline. It is deciding what to test based on risk, judging whether the environment and test data are credible, and knowing what action the result should trigger.

The Four Golden Signals

You cannot test what you have not defined, and his observation is that functional behaviour usually gets specified while performance behaviour does not, so teams jump straight to a script.

More metrics do not create clarity. Objectives do, which is why he starts from which journey matters and what degradation is tolerable over what period.

• Latency - response time, and his standard for it is refreshingly unscientific: you are quick enough when your customers think you are quick enough.
• Traffic - simply how much demand is on the system.
• Errors - not every error has to reach the consumer, since a request can be retried, but throw enough of them and you are no longer reliable.
• Saturation - how close a constrained resource is to its useful capacity. A CPU-bound system may start queueing somewhere around 80 to 85% CPU.

Useful capacity is itself slippery. You may never reach 100% CPU if you do not have enough threads, and resources have to be balanced before a real limit exists at all. His example is blunt: there is no use putting a hundred CPUs against one megabyte of memory.

The reason they are read together is that they move together. More traffic usually shows up as added latency, a few errors and rising saturation, and any one of them alone means little.

He also cites experiments on large web shops where adding roughly a fifth more latency cost revenue. No study or company is named on air, so that one is best treated as an illustration rather than a citation.

Indicators and Objectives

The chain runs from golden signals to service level indicators to service level objectives. Signals tell you what is happening; objectives tell you what is acceptable.

Not every signal makes a good indicator, and his test for one is that it must relate directly to user experience.

That rules saturation out, in the most quotable form in the session: your user does not experience 100% CPU, they experience high latency. Saturation is a poor indicator and a good alert, because it is frequently what caused the latency, and he says the same of traffic.

Compressed: indicators are what you measure, objectives are what you aim for, and the error budget is whatever fits inside the margin.

The practical payoff is that objective thresholds double as guardrails for both the analysis and the release decision. Visibility alone does not produce a decision, and when you are missing requirements the moment calls for fixing rather than flying.

Choose Tests by Risk

His rule is short enough to put on a wall: do not choose a test by habit, choose it by risk.

Teams that build and run their own software already know what changed and what risk came with it. As he puts it, creating a new endpoint calls for a different load test than fixing some logging.

Running everything available is the failure he warns against, because it produces a lot of false confidence. The point is knowing which test delivers which value.

The questions that drive the choice are about the risk rather than the calendar: is this demand, a surge of users, a capacity limit, whether cloud scaling actually works, long-term degradation, data volume, or recovery?

His framing of the test types matters more than the list. They are not maturity levels you climb; they answer different risk questions. Load tests check latency under normal and peak demand and need not run long. Stress tests ramp until something breaks, usually a saturated resource. Spike, volume, endurance, scalability and resilience tests each answer their own.

The Ticket Drop

His worked example is the one that makes the rule concrete. In the Netherlands concert tickets typically go on sale on a Friday morning, so a big stadium show means everyone arriving at ten and the system flooding.

For that risk he wants two tests. A spike test asks whether the system meets the short peak and recovers from it. A stress test finds where the boundaries actually are.

A regular load test and an endurance test add little, because the latency signal shows up before the spike does anyway. His underlying point applies well beyond ticketing: in banking or e-commerce there is no such thing as people gradually starting to use your system.

He is willing to call endurance testing overrated. With proper observability you do not need to load test for days, because something like a memory leak surfaces well before it breaks anything.

The exception he grants is regulation. Finance answers to many regulators, and one may require proof that a system handles load over a longer period before go-live, which is a compliance requirement rather than an engineering one.

A Disaster Recovery Finding

Resilience testing asks a different question from speed: how does the system behave when it is running suboptimally? That covers chaos testing and more controlled variants such as failover, injected latency, or running a process that claims all the CPU to watch what happens to response times.

As a bank, ING runs disaster recovery tests regularly. It operates across multiple data centres and has to be able to run on one, and he offers an aircraft landing on a data centre as the hypothetical, noting it has never happened.

The finding is the best evidence in the talk, because it landed on his own platform. It runs on four VMs, two in each data centre, in two flavours: one for high-load tests and one for regular tests. They had not been divided properly across the sites.

Lose one data centre and he would have been left with two VMs able to run regular tests and none at all for high-load tests, so he had to double his resources to keep the service running.

His conclusion is the reason to run the test rather than reason about it: sometimes that is not clear on paper, you just need to see it.

Workload Modelling

Even the right experiment misleads if the workload is wrong. A technically perfect script with the wrong user distribution, timing and data tells you very little about production.

His instruction is to model reality rather than assumptions. Systems not yet in production force assumptions, and the obligation is to revisit them the moment you go live.

• Observe production - traffic, timing, latency, errors, and which endpoints actually get hit.
• Capture user behaviour and flow - how people move through the journey rather than which calls exist.
• Build a transaction mix - combining the two into proportions you can drive.
• Create load profiles - the shapes you will actually run.
• Execute - the step most teams treat as the whole job.
• Validate - compare assumptions against reality, investigate deviations, decide what to fix.

Validation is the step he calls most important, because it is what makes the model repeatable and data-driven rather than a one-off guess.

His own addition is worth stealing: score how close each profile is to production. Simplification is unavoidable, since you cannot build a thousand profiles, so there will always be deviation and it is better quantified than ignored.

Analysis and Automation

Metrics do not create confidence, evidence does, and a dashboard can be green while a bottleneck hides behind it.

What makes it evidence is correlation. The test tool shows the user perspective while application and infrastructure logs and traces show what the system was actually doing, and the two have to be read against each other with baselines and objectives alongside.

His position on automation is precise: it should remove toil without replacing engineering judgement. The aim is proper feedback for the engineer, not a machine that decides.

The sequence he automates is collecting and correlating test metrics with infrastructure and application metrics, benchmarking against previous runs and against targets, looking for the patterns performance engineers normally eyeball in graphs such as peaks, slopes and missing load, validating coverage, detecting anomalies, then prioritising improvements.

Coverage validation compares against production: were all the endpoints and components hit, were the volumes right, and are the numbers comparable to production or slower. Slower can point at faulty test data or an environment that was never scaled properly.

He has lived the consequence of missing that. A gradual latency increase too small to notice across a few hours of load still bites in continuous production, so you deploy on Friday and the issues arrive on Sunday.

The Production Loop

Even good analysis only produces a prediction. Only production can say whether the prediction was credible.

A test earns the word predictive only after it has been proven right, which means comparing workload coverage, latencies, errors, saturation patterns and user outcomes between the test and the real thing.

When prediction and observation disagree, his reframe is that it is not a failed test, it is feedback. And the obligation attached to it is that the feedback must not disappear into another report.

The loop closes: production behaviour informs objectives, objectives shape experiments, experiments produce evidence, analysis turns evidence into learning, and learning improves the next workload model. That, he says, is where performance testing becomes reliability engineering.

Five Practical Actions

• Make production your friend - the one he calls most important. Stop treating performance testing as a loose activity and use what you see in production to build the test setup.
• Let objectives drive the decision - during analysis, know which metrics are genuinely crucial for customers and steer on those rather than on everything available.
• Test by risk, not habit - if you cannot explain why you are running a test, do not run it. The smoke test on every build is the exception, because quick feedback is an explanation.
• Learn from everyone - which he pairs with treating reliability as trust, since users expect something and failing to deliver it costs that trust.
• Release so feedback arrives first - do not deploy everything at once just before the stock exchange opens. Canary and candidate releases mean the signal reaches you before you move on.

He is honest that the safety question never gets a 100% answer. Meaningful objectives, intentional test selection, workload models drawn from production, correlated analysis and a loop that learns each round get you close, and that is the offer.

He closes on a line from Charity Majors, that everyone tests in production or lives a lie, and reads it carefully rather than as permission. Her point, as he takes it, is not to script transactions against live systems but that production is where unforeseen user behaviour and untriggered dependencies surface. He mentions having tested against production himself years ago and is explicit that it is not a practice he would encourage.

The session ran over its slot, so the queued questions were noted rather than answered on air.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.