Braintrust and LangSmith are full observability-plus-eval platforms with deep tracing and agent monitoring, but self-hosting is typically an enterprise-tier feature in both. Langfuse is MIT-licensed and self-hostable at the free tier, with a similarly broad observability feature set. All three compare eval runs using score-based drift rather than an explicit, pinned baseline as a core primitive. EvalCI is narrower and self-hosted from day one (Apache-2.0 core engine, evalci-core), built specifically around an explicit baseline and severity-aware CI exit codes for merge-gating — full observability and agent tracing are still on its roadmap, not shipped today. The right tool depends on whether the priority is a comprehensive observability platform or a focused, self-hosted CI regression gate.
Every team building LLM products eventually needs a real answer to "did this change make things worse," and the tooling landscape for that question has consolidated around a few names. Here's an honest look at where each one actually fits, including the one gap that's still surprisingly uncommon across all of them.
Braintrust
Braintrust is a comprehensive eval and observability platform — tracing, agent monitoring, and eval scoring in one product, with a polished UI and broad provider support. It's a strong pick for teams that want a single platform covering both evaluation and full production observability. Self-hosting is typically gated behind an enterprise plan rather than available from the free or standard tier, which matters if eval traffic can't leave your own infrastructure.
LangSmith
LangSmith, from the LangChain team, covers similar ground — tracing, evaluation, and monitoring, with particularly strong integration if the stack is already built on LangChain or LangGraph. Like Braintrust, self-hosting sits behind an enterprise tier rather than being available by default, and eval comparisons are generally score-based rather than built around an explicit pinned baseline.
Langfuse
Langfuse stands out on licensing — MIT, and self-hostable with the core feature set at no cost, which is a meaningfully different starting point from Braintrust or LangSmith. It covers tracing, session tracking, prompt management, and LLM-as-judge evals in a broad observability-first package, making it a strong choice for a team that wants an open-source, self-hosted observability platform with evals built in rather than a narrower CI-specific gate.
The shared gap: score drift vs. an explicit baseline
All three platforms above are built primarily as observability tools that also do evaluation, and their regression comparison is generally score-based — did the number move, relative to a recent average or a prior run. That's useful, but it's a softer signal than pinning a specific, human-approved run as the fixed baseline for every future comparison, and none of the three treats an explicit baseline as the central primitive the way a CI-focused regression gate needs.
Where EvalCI fits
EvalCI is deliberately narrower than all three: it's not trying to be a full observability platform (that's explicitly on the roadmap, not shipped today) — it's built specifically to gate a merge on an explicit, pinned baseline with severity-aware exit codes, self-hosted from day one via Docker Compose or Kubernetes rather than self-hosting being an enterprise feature. The open-core detection engine, evalci-core, is Apache-2.0 and pip-installable.
See the concrete setup in how to add LLM regression testing to a CI/CD pipeline, or the full EvalCI page for the complete feature comparison table.
How to actually decide
- Want one platform for full observability, tracing, and evals together, and self-hosting isn't a hard requirement → Braintrust or LangSmith.
- Want a fully open-source, self-hostable observability-plus-evals platform at no cost → Langfuse.
- Specifically need a self-hosted, explicit-baseline CI merge gate, and can add observability separately if needed → EvalCI.
Frequently asked questions
What's the difference between LLM observability and LLM evaluation?
Observability covers tracing and monitoring what a system actually does in production — requests, latency, traces through multi-step agent flows. Evaluation covers scoring output quality against a dataset, typically before a change ships. Braintrust, LangSmith, and Langfuse cover both; EvalCI is currently focused specifically on the evaluation and CI-gating side.
Can I self-host Braintrust or LangSmith?
Generally only on an enterprise-tier plan for both. Langfuse and EvalCI are both self-hostable without an enterprise tier — Langfuse under MIT, EvalCI's core detection engine under Apache-2.0.
On the observability side specifically — since that's explicitly still on EvalCI's roadmap rather than shipped — see the ForgeObserver LLM observability platform case study for a look at a full production deployment of that piece.
The takeaway
Braintrust, LangSmith, and Langfuse are strong, broad tools if the goal is a full observability platform with evaluation built in. If the specific need is a self-hosted CI gate built around an explicit, pinned baseline rather than score drift, that's a narrower job none of the three is purpose-built for — which is the gap EvalCI is built to close.
Have a similar challenge?
Book a free 30-minute architecture call and we'll tell you honestly whether and how we can help.
Book Free Discovery Call →