Why Evaluation Matters

Unlike traditional software, LLM applications have non-deterministic outputs. You can't just write unit tests — you need systematic evaluation.

Key Metrics

  • Accuracy: Does the output match the expected answer?
  • Relevance: Is the response on-topic and useful?
  • Faithfulness: Does the answer stick to the provided context?
  • Toxicity: Does the output contain harmful content?
  • Latency: How long does generation take?
  • Cost: What's the per-request cost?

Evaluation Approaches

  • Human evaluation: Gold standard but slow and expensive.
  • LLM-as-judge: Use a strong model (for example GPT-6 Astra or Claude Sonnet 5.5) to grade outputs.
  • Automated metrics: BLEU, ROUGE for summarization; pass@k for code.
  • A/B testing: Compare two versions with real users.

Tools

  • RAGAS: Evaluates RAG pipelines (faithfulness, relevance, recall).
  • DeepEval: Open-source LLM evaluation framework with 14+ metrics.
  • LangSmith: Tracing and evaluation platform by LangChain.
  • Braintrust: Evaluation and observability for AI products.

Best Practices

  • Build an eval dataset of 100-500 examples covering edge cases.
  • Run evaluations on every prompt change.
  • Track metrics over time in a dashboard.
  • Set quality gates: block deploys if scores drop below threshold.

Building an Evaluation Dataset

The foundation of any LLM evaluation is a high-quality test set. Aim for 100–500 examples that cover your main use cases, edge cases, and failure modes. Include both “easy” cases (should always pass) and “hard” cases (known failure modes). Label each example with the expected output or at least the expected behavior.

Key Metrics by Task Type

  • Q&A / RAG: Exact match, F1 score, and LLM-as-judge (use a current frontier model to grade responses on a 1–5 scale for relevance, accuracy, and completeness).
  • Summarization: ROUGE-L for overlap, plus LLM-as-judge for coherence and faithfulness (does the summary contain information not in the source?).
  • Classification: Precision, recall, F1 per class. Also track the confusion matrix to find systematic misclassifications.
  • Code generation: Pass@k (does the code pass unit tests?), plus human review for style and maintainability.

Automation and Regression Testing

Treat LLM evaluation like unit tests. Run your eval suite on every prompt change, model update, or configuration tweak. Tools like LangSmith, Braintrust, or custom scripts with pytest can automate this. Set a quality gate: if any metric drops by more than 5% compared to the baseline, block the change and investigate.

Start With Twenty Examples, Not Two Thousand

The most common evaluation mistake is waiting for a large labelled dataset. Twenty examples you have actually read, each with the answer you would accept, will change your decisions more than two thousand scraped ones you have not. Start there, ship against it, and grow it only when a specific failure you care about is not represented in the set.

The set should reflect the distribution of real use, including the boring cases. A set of twenty interesting questions measures how well you handle interesting questions and tells you nothing about the ninety percent of traffic that is routine.

Score the Component, Not the Vibe

End-to-end scores on open-ended output are noisy and hard to act on. Decompose the pipeline: was the right context retrieved, was the answer faithful to that context, was the format valid, was the latency acceptable. Each of these has a cheap check, and each points at a different fix. A system that fails faithfulness needs a better prompt; one that fails retrieval needs a different index; one that fails format needs a schema constraint. A single number cannot distinguish them.

Judge Models Need Their Own Calibration

Using a model as a grader is reasonable and often necessary for open-ended prose, but the judge must be calibrated against human labels on a sample you actually read. Measure agreement, not just the judge's scores. Pin the judge's version, and re-check the calibration when it changes, because a silently updated judge can move your whole dashboard.

Keep the Failures

Store every failed example with the input that produced it. Over months, that collection becomes the most valuable asset in the project, because it is the only artefact that encodes what your system is actually bad at.

Model names were checked against the official documentation of OpenAI and Anthropic on 2026-11-10.