Deepak Gupta markTools
AI Tools · LLM Evaluation and Prompt Management

Top 7 LLM Evaluation and Prompt Management Platforms 2026: How to Stop Shipping Prompt Changes Blind

Braintrust, Langfuse, LangSmith, Promptfoo, Comet Opik, Arize Phoenix and W&B Weave compared on offline evals, online evals against production traffic, CI regression gates, human review queues and prompt versioning. Every price taken from the vendor's own pricing page on 2026-09-18.

By ·Sep 18, 2026·14 min·7 tools compared
LLM EvaluationPrompt ManagementBraintrustLangfuseLangSmithPromptfooOpikArize PhoenixAI Tools

The answer, before the table

If you need a merge gate this afternoon and you have no budget, use Promptfoo: a YAML file, twenty test cases, one CI step, no account. If you want one system covering both production traces and offline experiments, run Langfuse self-hosted for free or take Cloud Core at $29 a month. If the eval is the point and you want a clean A-versus-B diff that blocks a bad prompt from merging, buy Braintrust and model its score meter first. If your stack is LangChain or LangGraph, LangSmith is the path of least resistance and the per-seat cost of your reviewers is the number to check. If you want annotation queues and thirty built-in metrics for under twenty dollars a month, Comet Opik is the value pick.

This is not the LLM observability comparison

The two categories get sold together and bought together, and teams regularly buy the wrong one.

Observability answers the backward-looking question. A user got a bad answer at 14:02. Which model handled it, what was in the context window, which tool call failed, how many tokens did it burn, and what did it cost. That is tracing, and it is covered in our Top 5 LLM Observability Platforms 2026 comparison.

Evaluation and prompt management answer the forward-looking question. Someone wants to change the system prompt, swap the model, or adjust the retrieval step. Is the new version better, worse, or differently broken, measured against a fixed set of examples, before any user sees it. That is datasets, experiments, scorers, CI regression gates, online scoring of production traffic, human review queues and a versioned prompt registry with a rollback path. That is this page.

The overlap is real and it is one direction: online evaluation scores live production traffic using the scorers from your offline suite, then feeds the low-scoring cases back into the dataset. That loop is why several vendors sell both, and why you should be precise about which product line you are pricing:

Vendor Observability product line Evaluation and prompt product line
Langfuse Tracing, sessions, cost tracking Datasets, experiments, scores, prompt management (same app, same meter)
LangChain LangSmith traces, runs, monitoring LangSmith datasets, evaluators, annotation queues, Prompt Hub (same trace meter)
Arize Arize AX, the commercial monitoring platform Phoenix, the open-source local-first eval and experimentation tool
Weights & Biases Weave tracing and monitors Weave evaluations and scorers (W&B Models is training runs, not this)
Braintrust Logs, attached to the eval loop Experiments, scorers, playground, human review (the core product)
Comet Opik spans and tracing Opik test suites, 30+ metrics, prompt library, annotation UI
Promptfoo None, by design Declarative eval files, assertions, matrix runs, red-team probes

If you are shortlisting for both jobs at once, read both pages and decide whether one tool doing both at a lower price beats two tools each doing one well. At small scale Langfuse or Opik usually wins that argument. At large scale the meters diverge sharply and the arithmetic changes.

One platform on this list no longer exists

Humanloop built one of the first LLM evaluation and prompt management platforms. Anthropic acqui-hired the team in 2025, billing stopped on 30 July 2025, and the platform went offline on 8 September 2025, taking prompt workflows, evaluations, logs and API integrations with it. Anthropic stated it did not acquire the assets or IP.

That is not a footnote, it is a buying criterion. Several vendors on this list are venture-funded startups in a consolidating category. Before you standardise, confirm you can export your datasets, your scorer definitions and your prompt history, and confirm that the instrumentation you add is portable. Arize Phoenix is OpenTelemetry-native and Langfuse, Opik and Promptfoo are open source, which is a meaningful hedge.

Quick Comparison

PlatformBest ForOffline Evals (Datasets + Experiments)Online Evals on Production TrafficHuman Review QueuesPrompt VersioningPublished Entry Price (checked 2026-09-18)
BraintrustEval-first teams that gate merges on a scoreCore product: experiments, scorers, side-by-side diffsYes, logs and online scoringYes, 1 per project on Starter, unlimited on ProYes, via playground and prompt objectsStarter $0; Pro $249/month; Enterprise custom
LangfuseOpen-source teams that want evals and tracing in one self-hostable stackDatasets, experiments, custom scores, LLM-as-judgeYes, scores attached to production tracesYes, 1 queue on Hobby, 3 on Core, unlimited aboveYes, versioning, caching, playgroundSelf-host free (OSS); Cloud Hobby $0; Core $29/month; Pro $199/month; Enterprise $2,499/month
LangSmithLangChain and LangGraph shops already tracing thereDatasets and evaluationsYes, on traced runsYes, annotation queues for human feedbackYes, Prompt Hub and PlaygroundDeveloper $0/seat (5k base traces/month); Plus $39/seat/month (10k base traces/month); Enterprise custom
PromptfooPutting an eval in CI this afternoon with no accountCore product: declarative test files, assertions, matrix runsNo, it is a pre-deployment harnessNo queue, review happens in your own processConfig-file versioned in your repo, not a hosted registryCommunity free forever (includes 10k red-team probes/month); Enterprise and On-Premise custom, no published price
Comet OpikSmall teams that want annotation UI and 30+ built-in metrics cheaplyTest suites, assertions, 30+ built-in metricsYes, span-level scoringYes, dedicated annotation UI and queuesYes, prompt library with versioningOpen source free; Free Cloud $0 (25k spans/month); Pro $19/month (100k spans/month); Enterprise custom
Arize Phoenix / AXTeams that want a local-first OSS eval loop with an enterprise upgrade pathExperiments and datasets, unlimited on every AX tierYes, via AX production monitoringYes, labeling queues, unlimited on every AX tierPrompt iteration in Phoenix, dashboards in AXPhoenix open source free; AX Free $0 (25k spans/month); AX Pro $50/month; AX Enterprise custom
W&B WeaveML teams already living in Weights & Biases for training runsEvaluations with LLM-as-judge scorersYes, tracing plus monitorsLimited, no dedicated queue product publishedWeave-managed objects rather than a standalone prompt registryFree $0 (1 GB/month Weave ingestion); Pro from $60/month (1.5 GB/month); extra ingestion $0.10/MB

Braintrust

Best For
Eval-first teams that gate merges on a score
Offline Evals (Datasets + Experiments)
Core product: experiments, scorers, side-by-side diffs
Online Evals on Production Traffic
Yes, logs and online scoring
Human Review Queues
Yes, 1 per project on Starter, unlimited on Pro
Prompt Versioning
Yes, via playground and prompt objects
Published Entry Price (checked 2026-09-18)
Starter $0; Pro $249/month; Enterprise custom

Langfuse

Best For
Open-source teams that want evals and tracing in one self-hostable stack
Offline Evals (Datasets + Experiments)
Datasets, experiments, custom scores, LLM-as-judge
Online Evals on Production Traffic
Yes, scores attached to production traces
Human Review Queues
Yes, 1 queue on Hobby, 3 on Core, unlimited above
Prompt Versioning
Yes, versioning, caching, playground
Published Entry Price (checked 2026-09-18)
Self-host free (OSS); Cloud Hobby $0; Core $29/month; Pro $199/month; Enterprise $2,499/month

LangSmith

Best For
LangChain and LangGraph shops already tracing there
Offline Evals (Datasets + Experiments)
Datasets and evaluations
Online Evals on Production Traffic
Yes, on traced runs
Human Review Queues
Yes, annotation queues for human feedback
Prompt Versioning
Yes, Prompt Hub and Playground
Published Entry Price (checked 2026-09-18)
Developer $0/seat (5k base traces/month); Plus $39/seat/month (10k base traces/month); Enterprise custom

Promptfoo

Best For
Putting an eval in CI this afternoon with no account
Offline Evals (Datasets + Experiments)
Core product: declarative test files, assertions, matrix runs
Online Evals on Production Traffic
No, it is a pre-deployment harness
Human Review Queues
No queue, review happens in your own process
Prompt Versioning
Config-file versioned in your repo, not a hosted registry
Published Entry Price (checked 2026-09-18)
Community free forever (includes 10k red-team probes/month); Enterprise and On-Premise custom, no published price

Comet Opik

Best For
Small teams that want annotation UI and 30+ built-in metrics cheaply
Offline Evals (Datasets + Experiments)
Test suites, assertions, 30+ built-in metrics
Online Evals on Production Traffic
Yes, span-level scoring
Human Review Queues
Yes, dedicated annotation UI and queues
Prompt Versioning
Yes, prompt library with versioning
Published Entry Price (checked 2026-09-18)
Open source free; Free Cloud $0 (25k spans/month); Pro $19/month (100k spans/month); Enterprise custom

Arize Phoenix / AX

Best For
Teams that want a local-first OSS eval loop with an enterprise upgrade path
Offline Evals (Datasets + Experiments)
Experiments and datasets, unlimited on every AX tier
Online Evals on Production Traffic
Yes, via AX production monitoring
Human Review Queues
Yes, labeling queues, unlimited on every AX tier
Prompt Versioning
Prompt iteration in Phoenix, dashboards in AX
Published Entry Price (checked 2026-09-18)
Phoenix open source free; AX Free $0 (25k spans/month); AX Pro $50/month; AX Enterprise custom

W&B Weave

Best For
ML teams already living in Weights & Biases for training runs
Offline Evals (Datasets + Experiments)
Evaluations with LLM-as-judge scorers
Online Evals on Production Traffic
Yes, tracing plus monitors
Human Review Queues
Limited, no dedicated queue product published
Prompt Versioning
Weave-managed objects rather than a standalone prompt registry
Published Entry Price (checked 2026-09-18)
Free $0 (1 GB/month Weave ingestion); Pro from $60/month (1.5 GB/month); extra ingestion $0.10/MB
1

Braintrust

Best Overall

Best for: Teams that want the eval to be the gate, not a dashboard they check after a bad release

“Braintrust is built around the experiment rather than the trace. You define a dataset, a task and a set of scorers, run it against a candidate prompt or model, and get a side-by-side diff against the last run before anything merges. Logs and online scoring exist, but they hang off the eval loop instead of the other way round. If your actual problem is that nobody can say whether the new prompt is better, this is the shortest path to an answer.”

Pros

  • Evals and experiments are unlimited on every tier including the free Starter plan, so the cost model does not punish running more tests
  • Unlimited users on all tiers, which removes the per-seat tax that stops domain experts from reviewing outputs
  • Human review is a first-class product, with unlimited review on Pro and Enterprise
  • Published overage rates let you model cost before you commit: processed data at $4/GB on Starter and $3/GB on Pro, scores at $2.50/1k on Starter and $1.50/1k on Pro
  • Enterprise tier offers on-prem or hosted deployment for teams that cannot send prompts to a vendor cloud

Cons

  • Pro jumps to $249/month, an order of magnitude above Langfuse Core at $29/month and Opik Pro at $19/month
  • Starter retention is 14 days and Pro is 30 days, with longer retention billed at $0.50/GB/month
  • Self-hosting is gated behind the Enterprise tier, unlike Langfuse, Phoenix and Opik
  • Scores are a metered unit, so a chatty multi-scorer eval suite can move the bill faster than a trace-metered platform would
Honest Weakness: Braintrust prices the thing you want to do more of. Scores are metered at $2.50 per 1,000 on Starter and $1.50 per 1,000 on Pro, and a regression suite with six scorers across 500 examples burns 3,000 scores per run. Run that on every pull request and the meter, not the subscription, becomes the real number. Model your scorer count against the included 10k (Starter) or 50k (Pro) scores before you standardise on it.

What it does that observability does not

An observability platform answers what happened on request 41,902. Braintrust answers whether version B of your prompt is better than version A across 500 recorded examples, before either reaches a user. The unit of work is an experiment: a dataset, a task function, and scorers that return numbers. The product is the diff between two experiment runs, which is what a reviewer needs to approve or reject a prompt change.

Regression suites and CI

Experiments run from CI against a pinned dataset, so a prompt change that drops a scorer below threshold fails the build in the same place a unit test would. This is the behaviour that separates an eval platform from a dashboard. Teams that never wire the suite into CI end up with a very expensive scoreboard nobody reads.

Human review

Starter allows one human review configuration per project and Pro allows unlimited. Combined with unlimited users on every tier, the practical effect is that a support lead or a lawyer can label outputs without a seat purchase, which is usually the blocker on getting non-engineers into the loop.

Starter $0/month ($10 model credits, 1 GB processed data, 10k scores, 14-day retention). Pro $249/month ($100 model credits, 5 GB processed data, 50k scores, 30-day retention). Enterprise custom with on-prem or hosted deployment. Overages: processed data +$4/GB (Starter) or +$3/GB (Pro), scores +$2.50/1k (Starter) or +$1.50/1k (Pro), extra retention +$0.50/GB/month. Unlimited users on every tier. Checked on braintrust.dev/pricing, 2026-09-18.

Visit Braintrust
2

Langfuse

Best Open Source

Best for: Teams that want evals, prompt management and tracing in one stack they can self-host for free

“Langfuse is the only platform here that gives you the full loop, tracing plus datasets plus experiments plus a versioned prompt registry, in an open-source package you can run yourself at zero licence cost. Cloud Core at $29/month is the cheapest credible hosted tier in this comparison. Buy it when you want one system for both questions, what happened in production and whether the change is better, and you would rather not pay per seat to find out.”

Pros

  • Open source and self-hostable for free via Docker or Kubernetes, with no feature paywall on the core loop
  • Unlimited users on every paid cloud tier, starting at $29/month for Core
  • Prompt management includes version control, caching, a playground and experiments on all plans
  • Evaluation covers datasets, experiments, custom scores, LLM-as-judge evaluators and human annotation
  • Single metered unit across tracing and evals at $8 per 100k units above the included allowance, which is easier to forecast than separate trace and score meters

Cons

  • The free Hobby tier caps at 2 users and 1 annotation queue, which is too tight for a team review workflow
  • Core at $29/month includes 100k units, the same allowance as Pro at $199/month, so the step up buys features and support rather than volume
  • Enterprise cloud is listed at $2,499/month, a steep jump from Pro
  • Self-hosting means you own the Postgres and ClickHouse dependencies, the upgrades and the retention policy
Honest Weakness: Langfuse is a strong generalist and a weaker specialist. Because tracing came first and evals were layered on, the experiment-comparison workflow is less opinionated than Braintrust's and the CI story takes more assembly than Promptfoo's. If the single thing you need is a merge gate with a clean A-versus-B diff, you will build more of it yourself here. The compensation is that you get the production side in the same tool at a fraction of the price.

Which product line does which job

Langfuse sells one product, not two. Tracing, datasets, experiments, scores and the prompt registry are all in the same open-source application and the same metered unit. That is why it appears both here and in our LLM observability comparison. The distinction to hold in your head is functional rather than commercial: traces answer the production question, datasets and experiments answer the pre-release question.

Prompt management

Prompts are versioned objects with caching and a playground, available on every plan including Hobby. The SDK fetches a named prompt by label, so a rollback is a label change rather than a deploy. This is the feature that most teams underestimate until the first time a prompt edit reaches production without a code review.

Self-hosting economics

At high volume the arithmetic favours self-hosting sharply. Cloud overage is $8 per 100k units, so an application generating 20M units a month costs roughly $1,600 in overage alone. Running the open-source build on your own infrastructure removes that line item and replaces it with an operations cost you control.

Self-hosted open source free. Cloud: Hobby $0/month (50k units/month, 2 users, 1 annotation queue); Core $29/month (100k units, unlimited users, 3 annotation queues); Pro $199/month (Teams add-on $300/month); Enterprise $2,499/month. Overage $8 per 100k units on all paid tiers, lower with volume. Checked on langfuse.com/pricing, 2026-09-18.

Visit Langfuse
3

LangSmith

Runner Up

Best for: Teams already building on LangChain or LangGraph who want datasets, annotation queues and Prompt Hub next to their traces

“LangSmith is the default if your application is already LangChain or LangGraph, because the traces arrive with no instrumentation work and the eval and prompt tooling sits on top of them. Datasets, evaluations, annotation queues for human feedback, and the Prompt Hub and Playground are available from the free Developer tier upward. Outside the LangChain ecosystem the case is weaker, and the per-seat model is the thing to check first.”

Pros

  • Zero-instrumentation capture for LangChain and LangGraph applications
  • Annotation queues for human feedback are included rather than sold as an add-on
  • Prompt Hub and Playground give a hosted, shareable prompt registry
  • Developer tier is genuinely usable at $0 per seat with up to 5k base traces per month
  • Enterprise tier offers self-hosting and custom workspace structures

Cons

  • Plus is $39 per seat per month, so adding five non-engineer reviewers costs $195/month before any usage
  • The pricing page states pay-as-you-go beyond the included traces but does not publish a per-trace overage rate in the plan table
  • Trace volume, not eval volume, is the meter, which is an awkward fit if you mainly run offline experiments
  • Strongest when you are inside the LangChain ecosystem, which is a real coupling decision
Honest Weakness: LangSmith is the only platform in this comparison whose pricing page did not publish an overage rate we could quote. The plan table lists $0 and $39 per seat with trace allowances of 5k and 10k, then says pay-as-you-go thereafter. For a product that meters the thing your application does on every request, that is the number you most need before you commit, and you have to ask for it. The per-seat model compounds the issue, because human review is exactly the workflow you want to hand to people who will never write code.

Which product line does which job

LangChain sells LangSmith as a single platform with an observability surface (traces, runs, monitoring) and an evaluation surface (datasets, evaluators, annotation queues, Prompt Hub). They share a billing meter based on traces. When you compare it against Braintrust or Promptfoo, compare the evaluation surface; when you compare it against Langfuse or Helicone, compare the observability surface.

Human feedback loop

Annotation queues route selected runs to a reviewer who scores them against a rubric, and those scores become dataset labels. That closes the loop between production traffic and the offline suite, which is the workflow most teams are missing. The constraint is commercial rather than technical: each reviewer needs a $39/month seat on Plus.

Prompt Hub

Prompts are versioned and shareable, with a playground for iteration. Because the hub is hosted, a prompt change is a platform action rather than a code deploy, which is convenient and also the reason you want review and rollback configured before you hand access to anyone.

Developer $0/seat/month, up to 5k base traces/month then pay-as-you-go, 1 seat. Plus $39/seat/month, up to 10k base traces/month then pay-as-you-go, unlimited seats at the same rate. Enterprise custom pricing, custom trace allowance, customisable seats and workspaces. Per-trace overage rate is not published in the plan table. Checked on langchain.com/pricing-langsmith, 2026-09-18.

Visit LangSmith
4

Promptfoo

Best Free Option

Best for: Getting a regression suite running in CI today, in a repo, with no account and no data leaving your infrastructure

“Promptfoo is a free, open-source command-line eval harness. You write a YAML file describing prompts, providers, test cases and assertions, and it runs the matrix and reports pass or fail. It is the fastest way to go from no evals to a merge gate, and it is the only option here that requires no vendor account at all. It is not a production monitoring tool and does not pretend to be.”

Pros

  • Community tier is free forever and includes all LLM evaluation features
  • Runs locally or self-hosted, so prompts and test data never leave your infrastructure
  • Config lives in your repo, so prompt versions are reviewed by the same process as your code
  • Includes red teaming and vulnerability scanning with 10k probes per month on the free tier
  • No seat licences, so every engineer and reviewer can run the suite

Cons

  • No online evaluation against production traffic, by design
  • No hosted prompt registry, so a prompt change still requires a deploy
  • No human review queue, so labelling workflows must be built around it
  • Enterprise and On-Premise tiers publish no price at all, so budget planning requires a sales call
  • Results live where you put them, which means reporting and history are your problem until you buy Enterprise
Honest Weakness: Promptfoo solves half the problem stated at the top of this page. It stops you shipping a prompt change blind, because the suite runs before the merge. It does nothing about the change that looked fine offline and degrades on real traffic three days later, because there is no online evaluation and no production trace store. Teams that adopt it usually end up pairing it with Langfuse or Opik, and should budget for that second system rather than discovering the gap in an incident.

Why a config file is the point

Because the eval definition is a file in your repository, a prompt change and the test that proves it is better arrive in the same pull request, reviewed by the same people. Hosted platforms split those into two systems and two audit trails. For teams with strong engineering discipline and weak tolerance for vendor sprawl, the file-based model is an advantage rather than a limitation.

Red teaming overlap

The free tier includes 10k red-team probes per month, which puts Promptfoo in the adversarial testing category as well. If prompt-injection and jailbreak coverage is the actual requirement, read our LLM red teaming comparison alongside this page, because the dedicated tools go deeper.

Where it stops

There is no trace store, no online scorer, no annotation queue and no hosted history. Everything after the merge is outside its scope.

Community free forever: all LLM evaluation features, red teaming at 10k probes/month, vulnerability scanning, run locally or self-hosted. Enterprise custom pricing. On-Premise custom pricing. Additional red-team probes purchasable on Enterprise. No dollar figures are published for the paid tiers. Checked on promptfoo.dev/pricing, 2026-09-18.

Visit Promptfoo
5

Comet Opik

Best Value

Best for: Small teams that want built-in metrics, an annotation UI and a prompt library for under $20 a month

“Opik is the value pick. The Pro cloud tier is $19 a month for 100k spans and up to 50 team members, and it ships 30-plus built-in evaluation metrics, a prompt library with versioning, a dedicated annotation UI and annotation queues. The open-source build is free and self-hostable with the full feature set. For a team of five that needs the whole loop and cannot justify $249 a month, this is the sensible answer.”

Pros

  • Pro at $19/month for 100k spans/month and up to 50 team members is the cheapest hosted tier in this comparison
  • Open-source self-hosted build is described as the full AI observability and agent testing feature set, not a crippled edition
  • 30+ built-in evaluation metrics covering LLM-as-judge and heuristic approaches, plus custom metrics
  • Dedicated annotation UI and annotation queues for expert review
  • Free Cloud tier gives 25k spans/month, 60-day retention and up to 10 team members
  • Verified academic researchers get the Pro plan free

Cons

  • Spans are the meter, and an agent run produces many spans per user request, so 100k goes faster than it reads
  • Smaller ecosystem and community than Langfuse or LangSmith, which matters when you hit an integration edge
  • AI-powered debugging and guardrails are Enterprise-only
  • Part of Comet, a company whose centre of gravity is classic ML experiment tracking rather than LLM application development
Honest Weakness: The low price is real but the span meter is the catch. Opik counts spans, not requests, and a single agent turn with tool calls and retrieval can emit ten or more. A team reading 100k spans as 100k conversations will overrun Pro in the first week and land on Enterprise pricing that is not published. Instrument one representative request, count the spans it emits, and multiply before you plan around the $19 tier.

Built-in metrics versus your own

Opik ships more than thirty evaluation metrics out of the box, split between LLM-as-judge scorers and deterministic heuristics. That shortens the first week considerably, because most teams stall on writing their first scorer. Treat the built-ins as a starting point: a generic hallucination metric will not encode what correct means for your domain, and the metric that actually gates your release usually has to be written.

Human review

The annotation UI and queues are included on every tier, including free. Combined with 10 members on Free and 50 on Pro, Opik is the cheapest way to get non-engineers labelling outputs, which is the bottleneck in most eval programmes.

Open Source free, self-hosted from GitHub, full feature set. Free Cloud $0/month: 25k spans/month, 60-day retention, up to 10 team members. Pro Cloud $19/month: 100k spans/month, 60-day retention, up to 50 team members, customisable span limits and retention. Enterprise custom: unlimited spans and members, custom deployment, SSO, SOC 2, ISO 27001, HIPAA and GDPR coverage. Checked on comet.com/site/pricing, 2026-09-18.

Visit Comet Opik
6

Arize Phoenix and Arize AX

Honorable Mention

Best for: Teams that want a local-first open-source eval loop with a documented upgrade path to an enterprise platform

“Arize splits its offering cleanly. Phoenix is the open-source, local-first platform for tracing, evaluation, experimentation and prompt iteration. Arize AX is the commercial platform, and every AX tier including the $0 Free tier lists evals, experiments, datasets and human annotations as unlimited. That combination suits a team that wants to start on a laptop and move to a hosted platform without changing tools.”

Pros

  • Phoenix is open source, local-first and free, covering tracing, evaluation, experimentation and prompt iteration
  • AX Free at $0 includes 25k spans/month, 1 GB ingestion/month and 15-day retention
  • AX Pro is $50/month, between Opik and Braintrust on price
  • Evals, experiments, datasets and human annotations are unlimited on all three AX tiers, so the eval workload is not the meter
  • Labelling queues and custom dashboards are available across tiers
  • OpenTelemetry-native, so instrumentation is portable if you leave

Cons

  • Two products with overlapping names creates real confusion when comparing quotes
  • AX Free retention is 15 days, the shortest free retention in this comparison
  • The free tier caps signal detection at 10 issues per month
  • Arize's heritage is ML observability, so parts of the platform assume an ML-monitoring mental model rather than an application-development one
Honest Weakness: You have to decide which Arize you are buying, and the marketing does not make that easy. Phoenix is a genuinely good free local eval harness. AX is a production observability platform with evals attached. A team that trials Phoenix, likes it, and then prices AX is pricing a different product with a different centre of gravity. Decide up front whether the requirement is a laptop-scale eval loop or an enterprise monitoring platform, because the answer changes which one you should even trial.

Which product line does which job

Phoenix is the open-source eval and experimentation tool. AX is the commercial observability platform. Phoenix is what belongs on this page; AX is what belongs in the observability comparison. Arize positions it as staying local and open, then moving to AX when you need it.

Unlimited evals as a pricing stance

Arize meters spans and ingestion, not evaluations. That is the opposite of Braintrust's score meter, and it changes behaviour: there is no cost reason to run fewer scorers. If your eval suite is scorer-heavy and your traffic is modest, this pricing shape is materially cheaper.

Phoenix open source, free, self-hosted or local. Arize AX Free $0: 25k trace spans/month, 1 GB ingestion/month, 15-day retention, 10 signal issues/month. AX Pro $50/month. AX Enterprise custom pricing. Evals, experiments, datasets and human annotations listed as unlimited on all three tiers. Checked on arize.com/pricing, 2026-09-18.

Visit Arize Phoenix and Arize AX
7

Weights & Biases Weave

Honorable Mention

Best for: ML teams already running W&B for training runs who want LLM evals in the same account

“Weave is the LLM application layer of Weights & Biases, sitting beside W&B Models, which handles training and fine-tuning runs. Evaluations, tracing and LLM-as-judge metrics are available on the Free and Pro tiers. Buy it because your team already lives in W&B and consolidation is worth more than best-of-breed, not because Weave wins the eval comparison on its own merits.”

Pros

  • Natural fit for teams already using W&B Models for training and fine-tuning runs
  • Free tier includes 1 GB/month Weave data ingestion and 5 GB storage
  • Pro starts at $60/month with 1.5 GB/month ingestion and 100 GB storage
  • Self-hosted Personal tier at $0 and a self-hosted Advanced Enterprise tier for teams with data residency requirements
  • AI application evaluations, tracing and LLM-as-judge metrics are listed across tiers

Cons

  • Ingestion overage is $0.10/MB, which is $100 per GB and by far the steepest metered rate in this comparison
  • Pro includes only 1.5 GB/month of Weave ingestion before that meter starts
  • The pricing page details no human annotation or prompt management pricing, so the review workflow is harder to scope than on Langfuse, Opik or LangSmith
  • Seat structure is split between Model seats and Weave seats, which complicates a simple headcount calculation
  • Now owned by CoreWeave, a GPU cloud, which is a strategic dependency worth naming if you also buy compute
Honest Weakness: The $0.10 per MB ingestion overage is the number that decides this. LLM traces are verbose: full prompts, retrieved documents and completions. An application ingesting 20 GB a month past the Pro allowance is looking at roughly $2,000 in overage, against $8 per 100k units on Langfuse or unlimited evals on Arize AX Pro at $50. Weave is the right answer when W&B consolidation is worth that premium, and the wrong one when it is not.

Which product line does which job

W&B Models covers experiment tracking for training and fine-tuning. W&B Weave covers LLM application development: tracing, evaluations and monitoring. If your comparison is against Braintrust or Promptfoo, you are comparing Weave. The Models product belongs next to the fine-tuning platforms, not here.

Ownership

CoreWeave completed its acquisition of Weights & Biases on 5 May 2025. Weave continues to ship as a product, and the acquisition has not removed the free tier or self-hosting. It does mean that a team buying both GPU capacity and eval tooling may be consolidating two dependencies into one vendor without noticing.

Free $0/month: 1 GB/month Weave data ingestion, 5 GB storage, up to 5 Model seats. Pro from $60/month billed monthly: 1.5 GB/month ingestion, 100 GB storage, up to 10 Model seats. Additional ingestion $0.10/MB, additional storage $0.03/GB. Enterprise custom. Self-hosted Personal $0/month; self-hosted Advanced Enterprise custom. Checked on wandb.ai/site/pricing, 2026-09-18.

Visit Weights & Biases Weave

Which One Should You Pick?

Use CaseOur Recommendation
You have never written an eval and want a merge gate this weekPromptfoo. A YAML file, twenty test cases and a CI step, with no account, no seat licence and no data leaving your infrastructure. Add a hosted platform later once you know which scorers matter.
You need both production traces and offline experiments, on a small budgetLangfuse Cloud Core at $29/month, or the free self-hosted open-source build. It is the only option here where one system and one meter covers tracing, datasets, experiments and a versioned prompt registry.
Prompt changes must be approved by a domain expert who is not an engineerBraintrust or Comet Opik, because both allow reviewers without a per-seat charge (Braintrust has unlimited users on every tier, Opik Pro allows 50 members at $19/month). LangSmith's annotation queues are good but each reviewer needs a $39/month Plus seat.
The application is built on LangChain or LangGraphLangSmith, for zero-instrumentation capture plus datasets, annotation queues and Prompt Hub in the place your traces already land. Price the per-seat cost of your reviewer headcount before committing, and ask sales for the per-trace overage rate.
Your eval suite has many scorers and modest trafficArize AX, which meters spans and ingestion rather than evaluations and lists evals, experiments, datasets and human annotations as unlimited on every tier including Free. Avoid Braintrust's score meter in this shape.
Prompts and test data cannot leave your infrastructureSelf-hosted Langfuse, self-hosted Opik or Arize Phoenix, all free and open source. Promptfoo also runs entirely locally. Braintrust self-hosting exists but requires the Enterprise tier.
Your team already tracks training runs in Weights & BiasesW&B Weave, for consolidation. Model the $0.10/MB ingestion overage against your trace volume first, because it is roughly $100 per GB and traces are verbose.

How we evaluated

Last verified: 18 September 2026. Every price, allowance, overage rate and tier limit on this page was read off the vendor's own pricing page or documentation on that date, and each entry states where. Where a vendor does not publish a number, the page says so rather than filling the gap from a third-party aggregator. Two vendors fall into that category: LangSmith publishes plan prices and included trace allowances but no per-trace overage rate in its plan table, and Promptfoo publishes no dollar figure for its Enterprise or On-Premise tiers.

The criteria, in the order they decide real outcomes:

  • Offline evaluation. Can you define a dataset, run a candidate prompt or model against it, and get a comparison against the previous run. This is the feature that turns a prompt change from an opinion into a measurement.
  • CI integration. Can that comparison fail a build. An eval that runs only when someone remembers is a scoreboard, not a gate.
  • Online evaluation. Can the same scorers run against a sample of production traffic, and do low scores route anywhere useful.
  • Human review. Is there an annotation queue, and what does it cost to put a non-engineer in front of it. Per-seat pricing is the usual blocker, so seat cost is treated as a feature constraint rather than a line item.
  • Prompt management. Versioning, labels, rollback and whether a prompt change requires a deploy. A hosted registry is a speed gain and a governance risk at the same time.
  • The meter. Scores, traces, spans, units or ingested megabytes. These are not comparable to each other, and the shape of the meter decides whether running more evals is cheap or expensive. This is where most cost surprises come from.
  • Exit cost. Open source, self-hosting, OpenTelemetry instrumentation and dataset export, weighed as a hedge against a category that has already lost one significant vendor.

Vendor selection

The seven platforms here were chosen because each is available to buy or run today and each covers at least the offline evaluation loop. Humanloop was excluded despite its history in the category because the platform was shut down on 8 September 2025 following Anthropic's acqui-hire, and there is nothing to purchase. Weights & Biases is included with a note that CoreWeave completed its acquisition of the company on 5 May 2025, because a team buying both GPU capacity and eval tooling should know it may be consolidating onto one vendor.

Pure observability platforms that do not sell an evaluation loop, such as Helicone, are not listed here; they are covered in the LLM observability comparison.

What we did not do

No product on this page was benchmarked, load-tested or run against a private dataset. There are no hands-on testing claims anywhere in this comparison. What the page offers instead is an accurate, dated reading of what each vendor publishes. It also states each product's weakest point plainly, and draws a clear split between this category and the observability category it is most often confused with.

Note

Editorial independence: this is a vendor-neutral comparison with no paid placements, sponsorships or affiliate links. Rankings reflect fit for the stated use cases, not commercial relationships.

Sources

Frequently Asked Questions

What is the difference between LLM evaluation and LLM observability?
Observability answers what happened in production: which model was called, what it returned, how long it took, what it cost, and where a specific bad response came from. Evaluation answers whether a change is an improvement: run a candidate prompt or model against a fixed dataset, score the outputs, and compare against the previous version before shipping. You need both, and the two overlap because online evaluation scores live production traffic using the same scorers as the offline suite. If your question is why yesterday's request failed, read our LLM observability comparison. If your question is whether tomorrow's prompt change is safe to merge, this page is the right one.
Several vendors appear on both comparisons. Which product line does which job?
Langfuse sells one product covering both, metered as one unit; treat its datasets and experiments features as the eval product. LangSmith is one platform with an observability surface and an evaluation surface (datasets, annotation queues, Prompt Hub) sharing a trace meter. Arize splits explicitly: Phoenix is the open-source eval and experimentation tool, Arize AX is the commercial observability platform. Weights & Biases splits by product: W&B Models is training-run tracking, W&B Weave is the LLM application layer with evals and tracing. Braintrust and Promptfoo are eval-first and do not compete as production observability platforms; Promptfoo does not do production monitoring at all.
Who should not buy an LLM evaluation platform at all?
Three groups. First, teams with no labelled examples. Every platform here is a harness for a dataset you do not have yet. The first useful step is writing thirty to fifty examples with known-correct answers, which costs a day and no money. Second, teams shipping fewer than a handful of prompt changes a month, where a spreadsheet and a careful reviewer genuinely is proportionate. Third, teams with no human willing to define what correct means, because an LLM-as-judge scorer with no ground truth just launders one model's opinion into a number that looks like a measurement. Start free with Promptfoo or self-hosted Langfuse and buy a platform when the free tooling is what is slowing you down.
What happened to Humanloop?
Humanloop, one of the first LLM evaluation and prompt management platforms, was acquired by Anthropic in an acqui-hire during 2025. Billing stopped on 30 July 2025 and the platform went offline on 8 September 2025, after which prompt workflows, evaluations, observability logs and API integrations were no longer accessible. Anthropic stated it did not acquire Humanloop's assets or IP, and the team joined Anthropic. It is not a purchasable option in 2026 and is excluded from this comparison for that reason. It is worth remembering when you evaluate the smaller vendors here: keep your datasets and prompts exportable.
What does an eval actually cost to run?
Three separate line items. First, the model calls: a 500-example suite against a mid-tier model is usually a few dollars per run, and it is charged by your model provider, not the eval platform. Second, LLM-as-judge scoring, which multiplies those calls by the number of judged scorers. Third, the platform meter, which differs by vendor: Braintrust meters scores at $1.50 to $2.50 per 1,000, Langfuse meters units at $8 per 100k, Opik and Arize meter spans, and Weave meters ingested data at $0.10/MB. The platform fee is often the smallest of the three at low volume and the largest at high volume. Count the scorers in one run and multiply by your intended CI frequency before choosing.
Do I need online evaluation, or is an offline regression suite enough?
An offline suite catches regressions against cases you already know about. Online evaluation catches the cases you did not think of, by scoring a sample of real production traffic with the same scorers and surfacing low scorers for human review. The practical workflow is that online scoring feeds new failure cases back into the offline dataset, which is how the suite stops going stale. Promptfoo is offline only. Braintrust, Langfuse, LangSmith, Opik, Arize AX and Weave all support scoring production traffic. If you have real users, budget for both.
Is hosted prompt management worth it, or should prompts live in the repo?
It depends who edits them. Prompts in the repository get code review, version history and deploys for free, which is why Promptfoo's file-based model suits disciplined engineering teams. A hosted registry, as in Langfuse, LangSmith Prompt Hub or Opik's prompt library, lets a product manager or domain expert change a prompt and roll it back by label without a deploy. That is a real speed gain and a real governance risk in the same feature. If you adopt a hosted registry, insist on versioning, an audit trail and a rollback path, and gate publishes behind the eval suite.

About the author

is the founder and creator of LoginRadius, a customer identity platform he built and scaled to over a billion users. He is now the founder of GrackerAI, a GEO platform for B2B SaaS and cybersecurity teams, and has spent more than 15 years building identity and security products.

Related Comparisons

GPU Cloud and AI Compute

Top 7 GPU Cloud and AI Compute Providers 2026: Price Per GPU-Hour by Chip Class, Verified From Each Provider's Own Pricing Page

7 tools compared

Fine-Tuning and Model Customization

Top 8 Fine-Tuning and Model Customization Platforms 2026: What It Costs, When It Wins, and Why OpenAI Is Shutting Its Own Down

8 tools compared

AI Legal / Contract

Top 5 AI Legal and Contract Tools 2026: Harvey vs Spellbook vs Ironclad vs LegalOn vs Luminance

5 tools compared

AI Sales / SDR

Top 5 AI Sales / SDR Tools in 2026

5 tools compared