LangSmith vs Braintrust: Evals, Tracing and CI Compared

LangSmith vs Braintrust: Which Fits Your Framework, Team and CI?

LangSmith fits teams whose agents run on LangChain or LangGraph, who want traces routed to datasets and annotation queues by rule, and who may host agents on LangChain's managed runtime. Braintrust fits teams that want an eval-first workflow, pricing that does not count users, and a packaged GitHub Action that reports experiment scores on each pull request. Most Braintrust vs LangSmith decisions come down to four points:

• Framework - LangSmith instruments LangChain and LangGraph automatically; for any other stack, both platforms accept OpenTelemetry traces.
• Who needs access - if many reviewers only read traces, Braintrust lists unlimited users on every plan, while LangSmith's Plus plan charges per seat.
• CI - LangSmith fails a job through pytest or Vitest assertions; Braintrust comments on the pull request, and failing it on a score needs a reporter you write.
• Agent hosting - LangSmith Deployment runs LangGraph agents; Braintrust's own comparison table marks agent deployment as not offered.

Both client libraries keep their package names, langsmith and braintrust, on PyPI and npm, and each shipped a release in the week of this check. If Langfuse is also on your shortlist, LangSmith vs Langfuse covers that pair, and the wider field sits in the round-up of AI agent evaluation tools. Langfuse vs Phoenix compares Langfuse with Arize Phoenix on license, self-hosting and tracing.

LangSmith vs Braintrust Comparison Table

Every cell comes from the product's own docs or pricing page, read on 19 October 2026, except Braintrust's agent-hosting cell, which only Braintrust's own comparison page states. Otherwise, the comparison pages each vendor publishes about the other are not the source for any cell; their claims are checked in the claims section below.

Both trace rows rest on the OpenTelemetry Protocol, which each platform accepts at its own endpoint, so instrumentation written once can feed either.

Do LangSmith's and Braintrust's Claims About Each Other Hold Up?

Braintrust's comparison page and LangChain's comparison page each mark the rival down on several rows. The table puts each claim beside what the named product's own docs or repository said on 19 October 2026.

The action check used the files in Braintrust's eval-action repository, whose README describes a live summary comment on the pull request and lists no score threshold. LangChain's page also says Braintrust has "no native trace-to-dataset routing or trace-to-annotation-queue automation", and it holds only in part: no rule routes traces as they arrive, but Braintrust's dataset pipelines, run with bt datasets pipeline, promote filtered spans or traces from logs into a dataset in bulk.

How Do LangSmith and Braintrust Differ on Tracing and Frameworks?

LangSmith's OpenTelemetry guide says it can "send traces from any OpenTelemetry-compatible application", with automatic instrumentation for LangChain applications and manual instrumentation for other frameworks. Braintrust's OpenTelemetry docs give a trace endpoint where an x-bt-parent header "sets the trace's parent project or experiment", and LLM calls in the incoming spans become Braintrust LLM spans.

• Conversations - LangSmith links each turn's trace by a thread ID and runs multi-turn evaluators after an idle period; Braintrust recommends one trace per conversation with one span per turn, and its group-scope rules score turns logged as separate traces once the group has been idle for a set timeout.
• Model calls - LangSmith's LLM Gateway, still in beta, puts every call through one API key and records each as a trace; autoevals defaults its model calls to the Braintrust Gateway, whose hosted version is in public preview, when no other base URL is set.
• Regions - LangSmith's regions FAQ lists GCP US, GCP EU, GCP APAC and AWS US and does not support migration between regions; a Braintrust organization picks a US or EU data plane when it is created, and Braintrust's docs say that choice cannot be changed later.
• Retention cost - LangSmith's multi-turn evaluators "can extend trace retention by default", which moves traces to the longer and pricier tier unless you opt out per evaluator.

If you instrument with OpenTelemetry rather than either vendor's wrapper, switching later means changing an exporter endpoint and its headers. The spans an agent should emit, and the attributes worth keeping on them, are covered in AI agent tracing. Duration and token use can also be exported as OpenTelemetry LLM metrics, the histograms and counters that OpenTelemetry's GenAI semantic conventions define alongside spans.

How Do Braintrust's autoevals and LangChain's openevals Score the Same Inputs?

Each platform runs model judges, code and human review, and each vendor also publishes deterministic scorers as open source. Braintrust's autoevals docs call its scorers "open-source, deterministic where possible", and openevals is LangChain's equivalent. Neither page compares them, so the scorer run below put both through the same 20 inputs.

from autoevals import ExactMatch
from openevals.exact import exact_match

out = {"b": 2, "a": 1}
ref = {"a": 1, "b": 2}

ae = ExactMatch()(out, ref).score
oe = exact_match(outputs=out, reference_outputs=ref)
print("autoevals ExactMatch:", ae)
print("openevals exact_match:", oe["score"])

autoevals ExactMatch: 0
openevals exact_match: True

run 2026-10-19 22:03 UTC
autoevals 0.4.0, openevals 0.2.0
AE = autoevals, OE = openevals; 1 match, 0 no match

 # item                   AE ex OE ex AE lev OE lev
 1 same text                  1     1      1      1
 2 one typo                   0     0  0.800  0.800
 3 case only                  0     0  0.800  0.800
 4 trailing space             0     0  0.833  0.833
 5 accent                     0     0  0.750  0.750
 6 paraphrase                 0     0  0.238  0.238
 7 wrong ticket id            0     0  0.895  0.895
 8 int vs string              1     0      1      1
 9 bool vs string             1     0      1  0.750
10 float vs int               0     0  0.333  0.333
11 dict same order            1     1      1      1
12 dict keys swapped          0     1  0.750  0.750
13 tool call keys swapped     0     1  0.333  0.333
14 dict vs JSON string        1     0  0.750      1
15 compact JSON string        1     0  0.625  0.875
16 list order                 0     0  0.667  0.667
17 nested value typo          0     0  0.944  0.944
18 empty strings              1     1      1      1
19 missing output             0   err      0    err
20 unicode dash               0     0  0.667  0.667

exact match differs: 7 of 20 [8, 9, 12, 13, 14, 15, 19]
Levenshtein differs: 4 of 20 [9, 14, 15, 19]
errors raised: {19: ['ValueError', 'ValueError']}

How Do LangSmith and Braintrust Gate CI?

Neither platform turns a score drop into a failed pull request until you define the threshold. Braintrust's eval-action manifest lists inputs such as the API key, eval paths, runtime, package manager, error handling and which scores and metrics to report, and none that sets a score threshold.

• On LangSmith - write the threshold as a test assertion, for example a minimum mean score over the dataset, so the runner's own exit code fails the job.
• On Braintrust - Braintrust's CI docs show a custom Reporter whose report_run return value "tells the process to exit", picked up automatically by bt eval from your eval files. The eval-action README says the action runs braintrust eval --jsonl for Node and Python rather than bt eval, and neither page says whether that path honors a reporter, so prove the gate with a commit that should fail.
• Baselines - Braintrust compares an experiment with a baseline; when none is set and experiments carry git metadata, it picks the most recent experiment on the same git branch, which keeps a pull request's scores next to its branch's last run.

How Do the LangSmith and Braintrust Pricing Meters Differ?

The two pricing pages, read on 19 October 2026, meter different things, so the cheaper option depends on headcount against data volume. Amounts change, and both pricing pages list them.

• LangSmith seats - the free Developer plan has one seat; Plus adds seats at a per-seat fee; Enterprise has custom seats and workspaces.
• LangSmith traces - each plan includes a monthly allowance of base traces, then pay-as-you-go; extended traces keep data longer at an extra fee.
• LangSmith add-ons - deployments bill on compute and memory hours, and the LLM Gateway draws on gateway credits.
• Braintrust users - every plan lists users as unlimited, so adding reviewers does not change the bill.
• Braintrust usage - each plan includes processed data in gigabytes and a number of scores, with overage per gigabyte and per thousand scores, plus model credits; Pro adds a flat monthly platform fee, and Enterprise sets a custom one.
• Braintrust retention - each plan has a retention window; on Pro, extra retention is billed per gigabyte per month.

Two settings that move the bill: on LangSmith, how many traces an evaluator upgrades to extended retention; on Braintrust, the sampling rate of online scoring rules, since every scored output counts toward the scores meter.

LangSmith vs Braintrust Self-Hosting, Data Residency and Compliance

Both platforms keep self-hosting on the Enterprise plan. LangSmith offers a fully self-hosted install and a hybrid one, while Braintrust always keeps the control plane and offers two Enterprise ways to put the data plane in your cloud: BYOC, which Braintrust operates inside your account, and self-hosting, which you run from published code such as the Braintrust Terraform module for AWS. That module creates the VPC, databases and associated resources for a self-hosted data plane.

• LangSmith self-hosted - "an add-on to the Enterprise plan", installed on Kubernetes with Helm, using ClickHouse for traces and feedback, PostgreSQL for operational data, Redis or Valkey for queues, and optional blob storage.
• LangSmith hybrid - the pricing page describes a SaaS control plane with a self-hosted data plane on Enterprise.
• Braintrust self-hosted - "Self-hosting is only available on the Enterprise plan"; you run the data plane, which stores experiment records, logs, traces and datasets, and the docs say "The control plane does not store or process your sensitive data."
• Braintrust clouds - official Terraform modules for AWS, Google Cloud and Azure; on GCP and Azure the data plane runs on Kubernetes with Helm.
• Stated compliance - LangSmith's regions FAQ says "We are also SOC 2 Type 2 certified and are HIPAA compliant" and states GDPR compliance; Braintrust's security docs say "Braintrust is SOC 2 Type II compliant" and that it can sign BAAs for HIPAA, which its pricing page lists on Enterprise plans.

Braintrust's security page names no hosted region, but its organization docs offer a US or EU data plane, and for full GDPR compliance the security page points to a BYOC or self-hosted deployment with data residency in the EU. Neither vendor moves an existing organization to another region, so if residency decides your choice, ask both in writing which region will hold traces, prompts and evaluation results. If self-hosting is the deciding factor, Braintrust vs Langfuse walks through each one's self-hosted setup step by step.

LangSmith and Braintrust SDK Versions, Licenses and Repository Activity

A second script, run on 19 October 2026, read the PyPI and npm registries and the GitHub API for both SDKs and counted the packages pip resolves for each. The archived langsmith-sdk repository says the SDKs moved and that "Packages are unchanged, nothing to do for installs." Everything in the console block below is what the script printed.

run 2026-10-19 22:01 UTC
90-day window starts 2026-07-21

PyPI langsmith
  latest 0.14.11 (2026-10-16), first 2023-06-26
  versions in 90 days: 33 (1 pre-release)
  license field: MIT
  direct runtime deps: 14
  pip resolves (Windows, Py 3.14): 25
PyPI braintrust
  latest 0.47.0 (2026-10-19), first 2023-05-24
  versions in 90 days: 21 (0 pre-release)
  license field: MIT
  direct runtime deps: 10
  pip resolves (Windows, Py 3.14): 21
npm langsmith
  latest 0.11.0 (2026-10-15), first 2023-06-26
  versions in 90 days: 25 (1 pre-release)
  license field: MIT
  direct runtime deps: 1
npm braintrust
  latest 3.38.0 (2026-10-19), first 2023-07-09
  versions in 90 days: 16 (0 pre-release)
  license field: MIT
  direct runtime deps: 28

pip resolves, scorer libraries (same setup):
  autoevals 9, openevals 49

GitHub repositories:
  langchain-ai/langsmith-sdk
    archived true, GitHub license MIT
    last push 2026-10-16
  langchain-ai/langsmith-python
    archived false, GitHub license MIT
    last push 2026-10-19
  langchain-ai/langsmith-javascript
    archived false, GitHub license MIT
    last push 2026-10-19
  braintrustdata/braintrust-sdk
    now braintrustdata/braintrust-sdk-javascript
    archived false, GitHub license Apache-2.0
    last push 2026-10-19
  braintrustdata/braintrust-sdk-python
    archived false, GitHub license Apache-2.0
    last push 2026-10-19
  braintrustdata/braintrust-sdk-javascript
    archived false, GitHub license Apache-2.0
    last push 2026-10-19

• Current repositories - LangSmith's SDKs now live in langsmith-python and langsmith-javascript; Braintrust's old braintrust-sdk address redirects to braintrust-sdk-javascript, beside braintrust-sdk-python. File issues against the new repositories.
• Release pace - in the 90 days to the run, the script counted 33 versions of langsmith on PyPI against 21 of braintrust, and 25 against 16 on npm, so pin versions in CI on either platform.
• License mismatch - Braintrust's package metadata declares MIT on PyPI and npm, while both SDK repositories carry an Apache License 2.0 file; the langsmith package and repositories all say MIT.
• Dependency count - the script's pip dry runs resolved 25 packages for langsmith and 21 for braintrust; for the scorer libraries, 49 for openevals, which pulls in LangChain, LangGraph and the OpenAI client, against 9 for autoevals.

How to Move Between LangSmith and Braintrust

• Move tracing first - point an OpenTelemetry exporter at the new platform's endpoint with its auth header, and run both side by side for a release so you can compare trace counts.
• Carry the scorers over as code - both libraries ran in the test above with no account and no API key, so a deterministic scorer can move as a plain function.
• Re-baseline before trusting deltas - run the old and new scorer on the same examples; in the test, exact match flipped on 7 of the 20 inputs, all of them typed, structured or missing values, with nothing about the outputs changed.
• Rebuild the CI gate - turn LangSmith test assertions into a Braintrust reporter, or the reverse, and check that a known-bad commit still fails the job.
• Move the reference set last - export the examples your release decisions rest on and re-run both platforms on them; a golden dataset kept in your own repository makes this a file copy.

Step 3 matters most for structured outputs such as tool-call arguments: in the scorer run, the same tool call with its keys in a different order passed openevals and failed autoevals.

What Do LangSmith and Braintrust Leave Unchecked Before Release?

Both platforms grade what an agent returned and what its trace recorded. In the scorer run, a reply naming the wrong ticket still scored close to a perfect match in both libraries, and even a perfect match would say nothing about whether ticket 4417 was closed in the ticketing system. On either platform, checking that means writing a code evaluator that queries the system yourself.

TestMu AI's Agent Assurance takes on that check for agents that act, before they ship. It starts from the agent's source code, or from a PRD, a spec or an API description if you cannot share the code, turns that into functional and adversarial scenarios that each carry acceptance criteria, then invokes the agent itself on staging. For the ticket example, this is the evidence each criterion is scored on:

• Ticket record - a read-only query, run through an MCP tool you provide and sign off on or through a status endpoint, shows whether the ticket really ended up closed.
• Ticket tool call - the close call your profile reports is held against the tool list the agent declares, and a scenario can name tools that must stay unused, such as a delete call.
• Saved files - notes or exports the agent writes under the output paths your profile lists, along with any artifacts the run produces, are read as evidence.
• The agent's own word - model judges score some criteria and read checks or pattern rules score others, but a reply saying the ticket is closed never counts as proof that it is.
• Outcome - a criterion ends Pass or Fail when the evidence decides it and Unable to Verify when it cannot; the scenario pass rate is computed over scenarios whose verdict was decided, and Unable to Verify results get a separate count beside it.

The Rook CLI is how you run Agent Assurance, which remains pre-alpha and publicly installable; one npm command sets it up:

npm install -g @testmuai/rook

Installing through npm calls for Node.js 20 or newer, and the CLI covers macOS and Linux, and 64-bit Windows through npm or WSL. Homebrew and a checksum-checking shell installer, listed in the Rook CLI installation guide, serve macOS and Linux without a separate Node.js install. Keep every run on staging, since a ticket the agent closes during a run stays closed and nothing rolls it back.

In CI, a zero exit status only says the run finished, so a pipeline gates on the verdict totals in the run's JSON report, as run Agent Assurance in CI/CD describes. Agent Assurance does not trace production traffic; LangSmith or Braintrust keeps that job, while the staging run checks what a release candidate changed.

LangSmith vs Braintrust Decision Checklist

Start this week by running your own scorers on 20 of your real outputs in both autoevals and openevals, as in the test above, before you compare any numbers across the two platforms. Then work through the rest:

• List your frameworks. LangChain or LangGraph points to LangSmith's automatic tracing; for anything else, instrument with OpenTelemetry and keep both options open.
• Count who needs read access to traces; a large reviewer group favors a meter without seats, a small team with heavy logging favors a trace allowance.
• Write the CI threshold yourself on either platform, and test it by pushing a commit you know should fail.
• Decide whether conversations are one trace or many, because the two platforms score multi-turn sessions in different ways.
• If data must stay in your cloud, budget for an Enterprise contract on either platform and confirm which parts stay with the vendor.
• For agents that change files, records or tickets, add a staging run that checks those changes before release; the Agent Assurance test scenarios docs show how its acceptance criteria are written.