How to Monitor LLM Quality in Production

A practical guide to monitoring LLM quality after deployment with traces, metrics, human review, alerts, regression testing, and production feedback.

Monitoring LLM quality after deployment means evaluating real interactions, not just watching latency and error rates. Capture the input, output, context, tool activity, model and prompt version, then score a sample against the quality dimensions that matter to the product. Route uncertain or high-risk cases to human reviewers, investigate clusters of failures, and turn confirmed failures into tests for the next release.

The goal is not one quality score. It is a feedback loop that tells you when the system is getting worse, which users or workflows are affected, and what changed.

Production monitoring starts with traces, not dashboards

A request and its final answer are not enough to explain most LLM failures. A production trace should preserve the events that led to the answer:

the user input and relevant conversation history;
the retrieved documents, chunks, scores, and source identifiers;
the prompt and model configuration used;
tool calls, arguments, results, retries, and errors;
the generated output and any user-visible citations;
latency, token usage, cost, and status;
the application, model, prompt, retrieval, and policy versions.

Tracing turns a quality alert into an investigation. If groundedness falls, you can check whether retrieval missed the evidence, the model ignored evidence that was present, or a tool returned an unusable result. If task completion falls, you can inspect the individual steps instead of guessing from the final response.

Galtea’s tracing documentation describes spans for tool calls, retrieval operations, and LLM invocations, including inputs, outputs, timing, errors, and parent-child relationships. Its production monitoring workflow can attach real interactions to your application’s own conversation IDs, so the monitoring record can remain connected to the session in your product.

Do not send more data than the investigation needs. Redact or tokenize personal data before it enters the monitoring stream, restrict access to raw traces, and define retention by risk and operational value. A trace that cannot be accessed safely is not a useful observability strategy.

Measure the dimensions that describe a working system

“Quality” is too broad to operate as a single metric. Choose separate measures for the ways your application can succeed or fail.

Dimension	What to measure	Useful evidence
Task success	Whether the user’s request was completed correctly	Expected outcome, structured fields, workflow state, or human verdict
Factuality and groundedness	Whether material claims are accurate and supported by the available context	Reference answer, retrieval context, citations, or expert review
Relevance and completeness	Whether the response answers the request without omitting a necessary step	User input, conversation context, expected answer, or rubric
Safety and policy adherence	Whether the system follows refusal, disclosure, privacy, and domain rules	Policy checks, adversarial cases, and escalations
Tool and workflow behavior	Whether the system chose the right tools, arguments, order, and stopping condition	Trace spans, expected tool calls, tool results, and state transitions
Reliability	Whether requests complete without timeouts, malformed output, or failed dependencies	Request status, retry count, schema validation, and error traces
Latency and cost	Whether the experience stays within its operational budget	End-to-end latency, time to first token, tokens, model cost, and tool time
User experience	Whether people accept, correct, abandon, or escalate the result	Explicit ratings, corrections, re-prompts, handoffs, and support cases
Drift	Whether inputs, retrieved evidence, or outcomes are changing	Input segments, embedding or topic distributions, failure rates, and version trends

Not every application needs every measure. A document extraction workflow may weight field accuracy and schema validity. A support assistant may weight answer relevance, groundedness, escalation behavior, and user corrections. A multi-step agent may need stronger monitoring of tool correctness and task completion than a single-turn classifier.

Galtea’s metric model separates deterministic checks from LLM-as-a-judge and human evaluation. That distinction matters in production. Use deterministic checks where the expected behavior is exact, use an evaluator for open-ended criteria, and reserve human review for cases where domain judgment or calibration is required. The Galtea metric documentation describes these evaluation types and supports custom criteria for a product’s own quality standard.

Sample real traffic, then score it deliberately

Scoring every interaction is not automatically better. It can increase cost, expose more sensitive data, and create a large queue of low-value results. A better starting point is a sampling policy that reflects risk and coverage.

a representative slice of ordinary traffic;
every interaction that triggers a safety, policy, or reliability signal;
new or changed user segments;
rare workflows and high-value tasks;
cases with negative feedback, correction, escalation, or repeated prompting;
traffic from each active model, prompt, retrieval, and application version.

Use stratified sampling rather than one random pool when failure risk differs by workflow. A random sample may contain mostly easy requests and hide a smaller, high-risk segment. Track the sample composition so a change in the monitored population is not mistaken for a change in quality.

For low-risk interactions, asynchronous evaluation is usually enough. For a high-risk workflow, a synchronous guardrail may be needed to block or route a response before it reaches the user. Those are different controls: monitoring explains what happened after the interaction, while a guardrail acts during it.

Build a quality scorecard that leads to a decision

Every metric on the scorecard should answer a question:

What failure does this metric detect?
Which users, workflows, or versions does it cover?
What evidence does the evaluator receive?
What action follows a breach?

A score without an owner or response path is decoration. For each metric, record:

the definition and pass condition;
the evaluation method and judge version;
the sampling rule;
the minimum sample size for an alert;
the comparison baseline;
the segment and version breakdowns;
the owner who investigates;
the action expected after confirmation.

Start with a small set of metrics tied to the most expensive failure modes. Add a metric when it catches a failure the current scorecard misses, not because the dashboard has room for another chart. Galtea supports custom metrics, thresholds, and evaluation criteria, so the scorecard can follow the product’s requirements instead of a generic benchmark.

Calibrate automated evaluators against human judgment

An LLM judge is a measurement instrument, not ground truth. Before using its result to alert or block a release, compare it with labeled examples from the real task distribution. Have a domain expert review clear passes, clear failures, and borderline cases. Record written reasons for disagreements. Then revise the rubric, test it on held-out examples, and keep watching for drift in judge behavior.

Do not collapse distinct dimensions into one vague rubric. A response can be relevant but unsupported, factually correct but unsafe, or safe but incomplete. Separate metrics make those cases actionable.

Human review should remain part of the loop. Send reviewers cases that are high risk, borderline, representative of a new failure cluster, or useful for recalibrating an automated evaluator. Galtea’s human evaluation workflow routes evaluations to reviewers and records their score and reasoning.

Use thresholds that account for uncertainty

An alert threshold is a decision rule, not a universal industry number. Set it from the product’s risk tolerance, the cost of a failure, the metric’s calibration, and the amount of traffic available.

Use three kinds of thresholds:

The absolute threshold is the minimum acceptable quality for a critical behavior.
The relative threshold is the allowed change from a stable baseline.
The volume threshold is the minimum number of examples before the signal can page someone.

Pair the threshold with a persistence rule. One failed evaluation should usually create a review item, not wake an on-call engineer. A sustained drop across enough examples, or a severe policy failure, should trigger escalation. Monitor both the aggregate score and the worst affected segment. A healthy average can hide a serious regression in one language, customer tier, workflow, or model version.

Treat alert fatigue as a quality problem. If an alert does not lead to a diagnosis or action, change its threshold, sampling rule, routing, or severity. Group related failures into an incident and show the representative traces. Do not page on every individual evaluator verdict when the useful unit is a cluster.

Run the operating loop from signal to fix

Production monitoring becomes valuable when every alert follows the same path.

1. Capture and classify the interaction

Store the trace with the versions, segment, outcome, and evaluation status. Keep the raw evidence needed for review, but apply redaction and access controls before storage.

2. Score the right sample

Run deterministic checks first where they are cheap and exact. Apply automated evaluators to open-ended dimensions. Route high-risk, uncertain, or disagreement cases to a qualified reviewer.

3. Detect a breach or a new pattern

Compare the current period with a stable baseline. Break the result down by model, prompt, retrieval configuration, workflow, geography, language, and customer segment where those dimensions are meaningful. A change in traffic mix can move an aggregate score without any code change.

4. Cluster failures before changing the prompt

Group failures by symptom and likely cause. Useful clusters include unsupported claims, missing retrieved evidence, incorrect tool selection, refusal errors, malformed structured output, policy violations, and latency or cost spikes.

Read the traces behind each cluster. A low groundedness score can come from retrieval failure, context truncation, or generation behavior. Those causes need different fixes. Rewriting the prompt before locating the cause creates churn and can hide the original signal.

5. Reproduce the failure

Preserve the input, context, configuration, and version that produced the failure. Re-run it with the current and previous version. For stochastic systems, use a small repeated run when variance could explain the difference. If the failure cannot be reproduced, keep the original trace and investigate distribution, dependency, or provider changes rather than discarding it.

6. Turn the confirmed failure into a regression case

Add a redacted example to the evaluation dataset with the expected behavior and the metric that should catch it. A production incident that never becomes a test is likely to return.

7. Feed the fix back into release evaluation

Run the new case with the broader regression set before deployment. Compare the candidate against the previous version, and check that the fix did not trade one failure for another. Galtea’s version model is designed to compare product iterations and record the facts that distinguish one version from another.

A cadence that balances speed and coverage

The right cadence depends on traffic, risk, and how quickly the system changes. The following is a practical starting point:

Cadence	What to do	Output
Per interaction or near real time	Capture traces, reliability events, policy signals, and user feedback	Searchable evidence and immediate safeguards
Each evaluation batch	Score the selected sample and route uncertain cases	Metric results and review queue
Daily	Check volume, quality, latency, cost, and failure clusters by segment and version	Triage list and incidents
Weekly	Review human disagreements, new failure modes, drift, and evaluator calibration	Updated rubric, dataset, or sampling policy
Before each release	Run the regression set and compare the candidate with the deployed version	Release decision and version record
Monthly or after a material change	Reassess the scorecard, retention, access, thresholds, and metric usefulness	Monitoring policy update

This cadence is not a substitute for risk analysis. A system handling sensitive decisions may require more frequent review and stronger escalation than a low-risk internal assistant. A low-traffic system may need a longer window before drawing conclusions, while a high-volume system can detect smaller changes sooner.

Protect privacy without losing the evidence

Production traces can contain personal, confidential, or regulated information. Treat them as sensitive operational data.

Redact direct identifiers and secrets before evaluation.
Prefer tokenized or reversible references only when the reviewer genuinely needs re-identification.
Keep raw inputs separate from aggregate metrics where possible.
Limit trace access by role and log reviewer access.
Set retention periods for raw traces, evaluator results, and derived regression cases.
Review third-party model and evaluator data-processing terms.
Exclude data from training or debugging workflows unless the organization has approved that use.
Document what is sampled, why it is sampled, and how deletion requests propagate.

Privacy controls should be designed with the monitoring schema, not added after the first incident. The Galtea monitoring page describes controls for sensitive AI systems, including retention controls, private deployment options, and access protections. Confirm the configuration that applies to your own deployment before making a compliance claim.

Common mistakes that make monitoring noisy

Tracking only infrastructure metrics

Latency, error rate, and token cost matter, but they cannot tell you whether the answer was correct or useful. Pair system health with outcome and behavior metrics.

Reporting one blended quality number

A blended score hides trade-offs. Keep factuality, relevance, safety, task success, and tool behavior visible separately, then use the traces to explain the movement.

Sampling only easy traffic

If the sample excludes escalations, rare workflows, and new segments, the dashboard will report a clean system that users do not experience. Keep targeted coverage alongside representative sampling.

Alerting on scores without enough evidence

Small samples and noisy judges produce unstable alerts. Require a meaningful volume, compare with a baseline, and use a persistence rule.

Treating an LLM judge as the truth

Calibrate it against human judgment, track disagreements, and change the rubric when it misses a meaningful failure. A judge that is not checked can turn a measurement problem into a product incident.

Fixing the prompt before reading the trace

The prompt is only one part of the system. Retrieval, tools, model configuration, context assembly, and application code can all cause the same visible symptom.

Letting the regression set go stale

Production failure modes change. Add representative cases from real incidents, remove cases that no longer represent the product, and version the dataset with the application and evaluator configuration.

Frequently asked questions

What is the difference between LLM evaluation and LLM monitoring?

Evaluation measures an output against a criterion. Monitoring applies evaluation and operational checks to real behavior over time, then routes the results into investigation and action. Evaluation can happen before release or on demand. Monitoring is the continuing production loop.

How much production traffic should I evaluate?

There is no universal percentage. Start with enough representative traffic to detect the failures that matter, then add targeted sampling for high-risk workflows, new segments, negative feedback, and changed versions. Increase coverage when the cost of a missed failure is higher than the cost of evaluation.

Should every production response be scored by an LLM judge?

No. Use deterministic checks where possible, sample open-ended responses, and reserve more expensive or human evaluation for cases that need it. The right design depends on risk, volume, latency, privacy, and evaluator calibration.

How do I monitor a multi-turn agent?

Capture the full session and the spans inside each turn. Record retrieval, tool calls, tool results, state transitions, and the final outcome. Evaluate both individual behaviors, such as tool correctness, and the end-to-end task. Galtea’s tracing guide covers hierarchical spans for agent operations.

How do I monitor RAG quality without duplicating a RAG guide?

Keep production monitoring focused on the loop: sample real sessions, measure groundedness and answer quality, inspect retrieval evidence when a score falls, and turn confirmed failures into regression cases. For deeper offline and production evaluation concepts, see Galtea’s complete guide to LLM evaluations.

How do I know whether a quality drop came from the model or the application?

Compare the affected version with the previous version using the same evaluation set, then inspect traces for prompt, retrieval, tool, dependency, and model changes. Record model and application facts with each version. If the provider changed behavior without a visible version change, the trace and time series are the evidence you have to localize it.

What should happen after a monitoring alert?

Create an incident or review item, identify the affected segment and version, inspect representative traces, classify the cause, and reproduce it. If the failure is confirmed, add a redacted regression case, fix the responsible component, re-run the evaluation set, and continue monitoring after release.