DeepInfer: DeepSeek's Open-Source Inference Framework

What Is DeepInfer?

DeepInfer is DeepSeek's in-house inference framework, the engine behind DeepSeek's large-scale production services, now open source under the Apache 2.0 license. The DeepInfer repository describes this release as offline inference with the core features needed for serving at scale, and credits vLLM as the base it was originally built on, with ideas from SGLang and TensorRT-LLM.

DeepSeek announced the release in a thread on X:

The only supported model is DeepSeek-V4.1-Flash, whose benchmarks and API pricing are covered in DeepSeek V4.1 Flash. The README lists these features in the first release:

• Optimized kernels - FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus custom TileLang kernels, for the model and sampler runtime.
• Parallelism - attention data parallelism or balanced context parallelism, and expert parallelism for the mixture-of-experts layers with expert-parallel load balancing (EPLB).
• DSpark - speculative decoding with draft tokens, where confidence estimates decide which drafts get verified.
• SWA bounded replay - cuts decoder prefill work by replaying only a bounded suffix of the sliding-window attention state.
• Asynchronous scheduling - overlaps CPU batch scheduling with device execution, on by default.
• Batch-invariant inference and deterministic sampling - identical results across batch compositions, and identical outputs for identical input tokens and sampling parameters.
• Structured outputs - generation constrained to a regex, JSON schema or EBNF grammar.

What Do DeepSeek's Production Numbers Show?

DeepSeek's inference deployment Q&A says a single cluster currently serves 155 trillion tokens per day for DeepSeek-V4.1-Flash, with overall revenue of US$38 per GPU-hour at valley pricing. The same document sets the design goals: run on accelerators from several vendors, use ordinary 8-GPU machines for prefill and 64-128-GPU supernodes for decode, and retain KV cache for more than 72 hours.

Other figures from the Q&A that matter when you plan capacity or read throughput claims:

Every figure here is DeepSeek's own measurement on its own fleet, so measure your hardware with the repository's benchmark and sweep scripts before you plan capacity.

The examples README warns that step times inside a profiler trace carry up to nearly 50% inflation at batch size 1, so traces are for inspecting kernel composition, not for deriving throughput.

What Hardware and Software Does DeepInfer Need?

DeepInfer needs NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs, the only two accelerator families the README lists. The build prerequisites are:

• Operating system - Linux.
• Compiler - GCC 13.0 or later.
• Python - 3.12, in a fresh, empty environment before the build.
• Accelerator toolkit - CUDA 13 on NVIDIA, CANN 9.2 on Ascend.

The build is one script, and the platform is chosen by an environment variable that must stay exported wherever DeepInfer runs, evaluation included:

# Inside an empty Python 3.12 environment
export DEEPINFER_ACCELERATOR=nvidia   # use "ascend" on Ascend NPUs
./build.sh

# Offline inference with the default checkpoint
python examples/offline_inference.py

The default checkpoint, DeepSeek-V4.1-Flash-1017, downloads into the Hugging Face cache on the first run. Set DEEPINFER_HF_MODEL_DIR to run from a local copy instead, which keeps test machines off the network.

Can DeepInfer Make LLM Output Deterministic?

Yes, on NVIDIA hardware. Batch-invariant inference keeps model results identical across different batch compositions, and deterministic sampling produces identical outputs for identical input tokens and sampling parameters. Both are NVIDIA-only in this release.

Batch composition matters because of how GPU kernels add numbers. In Thinking Machines Lab's study of nondeterminism in LLM inference, 1,000 temperature-0 completions of one prompt on Qwen3-235B produced 80 unique outputs, and with batch-invariant kernels all 1,000 were identical. The study traces the cause to floating-point non-associativity combined with server load, which changes the batch size and with it the order in which kernels sum values.

DeepInfer's README gives the switches for a reproducible run:

• Set DEEPINFER_BATCH_INVARIANT=1 in the environment.
• Set DEEPINFER_ENABLE_MTP_VARLEN=0, which turns off DSpark's variable-length draft verification.
• Pass a fixed seq_seed to SamplingParams on every request.

In DeepInfer's SamplingParams source, a request without seq_seed gets a fresh random 64-bit seed, and max_tokens defaults to 16.

The seed is the easy step to miss: a suite that omits it samples differently on every run, even with both flags set. Set max_tokens explicitly too, or an eval request stops after 16 tokens.

Determinism costs throughput. Switching off variable-length verification gives up the dynamic draft scheduling in the production table above, and Thinking Machines measured its own batch-invariant kernels running slower than the defaults. Use the deterministic configuration for CI and eval runs, and keep the faster one for production traffic.

vLLM offers the same idea: its batch invariance mode, enabled with VLLM_BATCH_INVARIANT=1, is in beta and lists testing and validation among its uses.

A deterministic engine makes a rerun repeat. It does not make an answer correct, and it does not hold an answer steady across prompt wording or checkpoints, so those checks still need tolerant assertions; testing non-deterministic AI outputs covers assertion types and how many runs a pass rate needs.

Note: TestMu AI Agent Testing reruns a chat or voice agent's scenario suite after a model or inference-engine change and returns a Green, Yellow or Red verdict with a confidence level, so you can tell a real regression from a thin sample. Create a free account.

How Does DeepInfer Regression Test Its Own Engine?

DeepInfer guards its numerics with golden baselines. Perplexity (PPL) baselines live in tests/models/offline_decode_ppl.json, keyed by platform and workload, and validation compares them exactly, sample by sample. A baseline is refreshed with a dedicated script only after a numeric change is confirmed as expected.

The examples README describes the checks around that baseline:

• Structured-output gate - with --check_structured_output, an offline run reuses the same model instance to run regex, JSON and EBNF checks and to verify a normal end-of-sequence, and any failure fails the run.
• Long-context PPL - a separate test feeds a 16,389-token input and generates 4,096 tokens with nextn set to 5.
• Vision-language samples - five fixed inputs (two text-only, one single-image, one three-image and one five-image 4-shot) built from two images stored in the repository.
• Per-platform baselines - when the images or questions change, short and long PPL baselines are re-measured separately on NVIDIA and Ascend.
• Multi-node safety - each node updates or compares only the ranks it owns, and golden baselines are written under a cross-node file lock.

The same pattern carries over to an LLM application. Freeze the inputs, store expected outputs per environment, compare exactly only when the engine runs deterministically, and refresh a baseline as a reviewed change rather than as a fix for a red build. LLM regression testing covers building that suite around prompts and agents.

How Does DeepSeek Score Its Benchmark Evals?

DeepInfer's eval harness scores AIME 2026, HLE and GPQA Diamond with an LLM judge by default, offers a deterministic rule judge for AIME and GPQA, and reports how often the two agree. On AIME 2026, the two modes agreed on all 960 rollouts.

Other choices in the harness worth copying:

• Separate generation and scoring - raw outputs go to disk first, so a run can be re-scored with a different judge without regenerating.
• Coverage check - a Pass@1 figure counts only when coverage_complete in summary.json is true; otherwise a rank or sample is missing.
• Run metadata - meta.json stores the complete configuration of every run for reproducibility.
• Judge limits stated - HLE accepts only the LLM judge, because more than half its answers are free-form text that exact matching would mark wrong.
• Indicative reference scores - the published Pass@1 figures are a sanity check that a deployment is configured correctly, and repeated runs may differ by a small margin.

If your suite grades with a model, run a deterministic checker on the subset where one exists and track its agreement rate with the judge from run to run. When agreement drops, check the judge before you debug the agent. LLM as a judge covers judge design and calibration.

What Does deepseek-recipe Add for Self-Hosted Deployments?

deepseek-recipe gives a self-hosted DeepInfer deployment the same request and response handling as the official DeepSeek API. DeepSeek released it alongside DeepInfer as its production token encoding and decoding component: it converts Chat Completions, Responses and Messages requests into DeepSeek prompts and parses model output back into each protocol.

The security detail is in its tokenizer guide: a special-token spelling typed into a user message is encoded as plain text, so it cannot act as a control token. On output, </think> and the DSML tags are recognized only as tokens, so the same text spelled with ordinary tokens stays content.

That gives you a concrete test for any self-hosted stack. Put strings such as <｜User｜> into user input, tool results and tool-call arguments, and assert that the encoded prompt holds no control token at those positions; a service that tokenizes the rendered prompt itself has to apply the library's special ranges the same way. Prompt injection testing covers the wider attack set.

The tokenizer guide also documents the max_tokens default applied when a request sets none, which decides how long an unbounded test request can run:

• Non-thinking mode - 8,192 tokens.
• Thinking mode - 65,536 tokens.
• Thinking mode with max reasoning effort - 131,072 tokens.

Each default is lowered to the context left after the prompt, and a requested value that overflows the context raises an error instead of being trimmed.

How to Test an Agent Served by DeepInfer

If you serve DeepSeek-V4.1-Flash yourself, with DeepInfer as the engine and deepseek-recipe handling the API protocols, your agent runs on a different stack than the hosted API, so test its behavior before and after the switch. Which tool fits depends on whether the agent talks or acts.

For chat, voice and phone agents, TestMu AI Agent Testing generates scenarios from your documents and grades every conversation on metrics including hallucination, completeness and context awareness. Run one suite against the hosted DeepSeek API and one against your self-hosted service, and compare the per-metric scores rather than the raw text. Start with testing your first AI agent to connect a chat endpoint.

For agents that call tools, write files and hit APIs, Agent Assurance derives a suite from the agent's code or spec, runs the real agent, and checks each criterion against what the run changed rather than what the agent reported. Its run-over-run comparison marks a scenario flaky when it flips with nothing changed, apart from newly failing ones, and on a deterministic engine a remaining flip points at the agent or its environment. Agent Assurance is pre-alpha and installs publicly with npm.

Conclusion

Pin one deterministic DeepInfer configuration for your eval environment first: DEEPINFER_BATCH_INVARIANT=1, variable-length verification off, an explicit max_tokens, and a fixed seq_seed per request, stored with each run's results.

Then gate releases on that environment. The Agent Assurance CI/CD guide runs an explicit suite, preserves evidence, and checks completion and verdict counts instead of treating a successful CLI exit as a passing agent test.