Blog

Practical writing for AI engineers shipping AI agents. From calibrating LLM judges to running evaluation pipelines in production.

LLM evaluations

Featured

the complete guide for LLM evaluations in 2026

How to measure whether an LLM application actually works, from traces and golden datasets through judge calibration, CI regression gating, and production monitoring.

All posts

LLM as a Judge prompts: templates, rubrics, and best practices

Production-grade prompt templates for the four criteria RAG and agent pipelines need first, with the rubric patterns and anti-patterns that calibration runs consistently surface.

Why AI coding agents are bringing the CLI back

APIs and SDKs made the GUI optional for power users. Coding agents are making the CLI strategic again, and most AI-product companies haven't shipped what those agents actually want first.

LLM as a Judge: The Complete Guide

LLM-as-a-judge is the practice of using one language model to evaluate another model’s outputs against a rubric, making scalable AI evaluation practical for chatbots, RAG systems, and agents. The article explains the three core judging modes, where LLM judges work well, where they fail, how to write reliable rubrics, and why calibration against a labelled gold set is mandatory before production use.

Golden datasets for regulated AI: six Q&A frameworks tested

We benchmarked six Q&A generation frameworks (DeepEval, Giskard, LangChain, LlamaIndex, RAGAS, Galtea) on the same gpt-4.1 against the same calibrated judges. Confidence scores spread 17.8 points across an otherwise controlled comparison. The framework with the highest diversity score, RAGAS at 0.723, came last on confidence at 0.760, because its "diversity" was inflated by malformed and hypothetical questions with no source-document ground truth. Giskard wrote English questions from Spanish sources 46% of the time. For regulated or multilingual deployments, validity and language fidelity decide more than phrasing variety. The simplest pre-ship test stays the same: read 30 of the generated questions before you ship.

How to optimize your LLM Judge for AI evaluations (And why most teams get it wrong)

Most teams building LLM evaluation pipelines spend a lot of time on the judge itself, which model to use, how to write the rubric, and which dimensions to score. Almost none of that effort goes into evaluating whether the judge is actually right.

How are your LLM Products Used?

We analysed 10,000 real interactions with LLM-powered products to map how users actually behave with them — appropriate use, intentional misuse, ambiguous inputs — and what that means for testing before deployment

Galtea: Pioneering Responsible GenAI Adoption

Why we built Galtea, the gap we saw between AI capability and enterprise readiness, and how a Barcelona Supercomputing Center spin-off is approaching AI governance, accuracy, and EU AI Act compliance.

How Galtea's Conversation Simulator Prevents Real-World AI Failure

Single-turn tests miss the failures that matter — agents losing context, choosing the wrong tool, drifting from the user's goal. How our conversation simulator runs multi-turn scenarios with personas and goals, and what to measure when "the response was correct" isn't enough.

Red Teaming LLM-Powered Systems: Breaking Beyond the Model

Most red-teaming targets the model. The bigger gap is testing the system — its purpose, capabilities, boundaries. A walkthrough of how we generate product-aware adversarial prompts, with a healthcare-assistant example showing how a roleplay strategy bypasses guardrails.

How to create a solid set of test cases to evaluate your GenAI system

Three approaches to test-case generation for GenAI systems — red teaming with curated attack databases, gold-standard generation for RAG and tool calling, and synthetic-user simulation for multi-turn conversations. With concrete examples for each.

Inside Galtea’s Red Teaming Pipeline for LLM Security

How we built our red-teaming dataset: collecting adversarial prompts from public sources, cleaning and standardising them, and using K-means clustering on sentence embeddings to surface six categories of attack. Includes a curated open-source subset on Hugging Face.

Exploring state-of-the-art LLMs as Judges

Benchmarks for Glider, FlowJudge, Phi-3.5-mini, Selene, GPT-4o, and Claude 3.5 Sonnet as judges across general rubrics and red-team safety tasks. Where small fine-tuned models hold up, where they don't, and what the latency and memory tradeoffs look like.

Cybersecurity Concerns Delay Widespread MCP Adoption

Model Context Protocol expands what LLM apps can do — and the attack surface that comes with them. A walkthrough of the five biggest security risks MCP introduces, mapped to OWASP categories.

Why Evaluation is the Key to Scaling Generative AI

Most enterprise GenAI projects stall when they try to scale past the MVP. The reason is evaluation strategy. Here's the four-stage view of where teams get stuck, and what it takes to move past Stage 3.

Galtea Team

February 24, 2025

4 minutes