Promptfoo vs evalci-core: Open-Source LLM Eval Comparison
Promptfoo is a broad, mature open-source tool (MIT-licensed, 350k+ developers) for prompt testing, model comparison, and red-teaming, run via YAML config from the command line — as of March 2026 it was acquired by OpenAI, which raises a fair neutrality question for teams using it to compare outputs across multiple model providers. evalci-core (Apache-2.0) is narrower and more specific by design: an explicit-baseline regression detection engine built to be a CI merge-gate rather than a general-purpose testing and red-teaming framework. The honest framing is scope, not superiority — Promptfoo for broad prompt testing and security scanning, evalci-core for teams that specifically want baseline-based regression gating and provider-neutral tooling.
Promptfoo and evalci-core are both open-source, both runnable from a CI pipeline, and both used to catch problems in LLM outputs before they ship. Beyond that, they're built for meaningfully different jobs, and the honest comparison is about scope and fit rather than one being simply better than the other.
Promptfoo: broad, mature, widely adopted
Promptfoo is a CLI-first, open-source framework for testing and evaluating LLM prompts — YAML-defined test cases, a wide range of assertion types (exact match, regex, JSON schema, cost and latency thresholds, LLM-graded checks), support for essentially every major model provider, and a red-teaming module that scans for prompt injection, jailbreaks, and dozens of other vulnerability categories. It's MIT-licensed, fully self-hostable with all features free, and has real scale behind it — hundreds of thousands of developers and a large share of Fortune 500 usage.
That breadth is Promptfoo's real strength: one tool for prompt testing, model comparison, and security scanning, with a mature ecosystem and CI integration that's been battle-tested widely. It's explicitly a pre-deployment tool — it doesn't monitor live production traffic or alert on real-time degradation, which is a reasonable scope boundary, not a flaw, but worth knowing going in.
What changed in March 2026
Worth naming directly since it's recent and relevant: OpenAI acquired Promptfoo in March 2026. The core tool remains MIT-licensed and self-hostable today, and the acquisition doesn't change what it does technically. What it does introduce is a fair question for any team using Promptfoo specifically to compare outputs across multiple model providers, including OpenAI's competitors — a testing tool now owned by one of the providers it evaluates is a legitimate neutrality consideration, even without assuming any deliberate bias. For teams where that's not a concern, this changes little day to day.
evalci-core: narrower, purpose-built for CI regression gating
evalci-core is Apache-2.0, pip-installable, and deliberately scoped narrower than Promptfoo: it's built around an explicit baseline as a core primitive — pin a specific approved run, and every future comparison is measured against exactly that run, with severity-aware exit codes that let a pipeline decide which level of regression actually blocks a merge. It doesn't do red-teaming or broad multi-provider prompt experimentation the way Promptfoo does; the entire design is oriented around one job — catching regressions in CI, reliably and with an unambiguous pass/fail signal — done well, rather than a wide feature surface.
How to actually choose
- Need broad prompt testing across many providers plus security/red-team scanning, with a large mature ecosystem → Promptfoo.
- Multi-provider neutrality is a real concern for your evaluation tooling specifically → worth factoring the March 2026 acquisition into that decision.
- Want an explicit-baseline regression gate as the core CI signal, self-hosted from day one without an enterprise tier → evalci-core.
- Nothing stops using both — Promptfoo for broader prompt iteration and red-teaming, evalci-core specifically as the merge-blocking regression gate.
For the concrete mechanics of setting evalci-core up as a CI gate specifically, see how to add LLM regression testing to a CI/CD pipeline.
Frequently asked questions
Is Promptfoo still open source after the OpenAI acquisition?
Yes, the core tool remains MIT-licensed and self-hostable as of this writing. The long-term roadmap and neutrality implications for a tool now owned by a model provider are a separate, more open question worth tracking if that matters for your use case.
Can Promptfoo and evalci-core be used together?
Yes — they solve different problems. Promptfoo works well for broad prompt iteration, multi-model comparison, and red-teaming; evalci-core is purpose-built as an explicit-baseline CI regression gate. Using Promptfoo earlier in development and evalci-core as the merge gate is a reasonable combination.
The takeaway
These aren't competing for the same job. Promptfoo is a broad, mature testing and red-teaming framework; evalci-core is a narrower, purpose-built regression gate for CI. The right pick — or combination — depends on whether the priority is breadth of testing capability or an unambiguous, baseline-driven merge gate, with vendor neutrality now a genuine additional factor since March 2026.
Have a similar challenge?
Book a free 30-minute architecture call and we'll tell you honestly whether and how we can help.
Book Free Discovery Call →