Production Agent Engineering: DSPy, Deterministic State Graphs, and Calibrated LLM Judges

1. Executive Summary#

Most prototype AI agents fail in production for three reasons: brittle prompt templates break when switching models, unconstrained autonomous loops get stuck in infinite tool calls, and teams have no statistically rigorous way to evaluate changes. Production agent engineering solves these with DSPy, guarded state graphs, and calibrated LLM-as-a-Judge pipelines.

2. Why Manual Prompt Engineering is Technical Debt#

Hand-tuning 200-line strings filled with "IMPORTANT: DO NOT HALLUCinate" and hardcoded few-shot examples mixes program specification (what the step should accomplish) with model-specific hyperparameter tuning (how to coax a specific checkpoint into doing it). When you migrate from gpt-4o to Llama-3.3-70B or Claude-3.5-Sonnet, hardcoded prompts often regress.

The DSPy Paradigm:#

• Signatures: Declarative input/output type contracts (e.g., context: list[str], question: str -> answer: str, citations: list[int]).
• Modules: Composable building blocks (dspy.Predict, dspy.ChainOfThought, dspy.ReAct) analogous to PyTorch nn.Module layers.
• Optimizers (Teleprompters like MIPROv2 & BootstrapFewShot): Given a training set of inputs and a programmatic validation metric, DSPy automatically synthesizes instructions, mines high-performing reasoning traces, and compiles the optimal prompt or fine-tuned weights for your target model.

3. Taming Multi-Agent Loops: State Machines Over Autonomy#

Unconstrained multi-agent chats (where Agent A talks freely to Agent B until they "agree") suffer from compounding error cascades: if each step in a 10-step autonomous loop has 92% reliability, end-to-end reliability drops to 0.92^10 = 43.4%.

Production Best Practices:#

1. Explicit Directed State Graphs: Model workflows as deterministic finite state machines with typed state schemas. Only use an LLM to make bounded routing decisions or transform state fields.
2. Grammar-Constrained Decoding: Never rely on prompt instructions alone to output valid JSON or tool calls. Enforce schemas at the logit level using Context-Free Grammars (CFGs) or JSON-schema FSMs (via vLLM/SGLang guided_decoding), masking out invalid token IDs before sampling.
3. Idempotent Tool Contracts & Circuit Breakers: Cap cyclic retry edges (e.g., max_retries = 2) and return structured compiler/validation errors back to the agent state rather than raw stack traces.

4. Building an LLM-as-a-Judge Pipeline That Doesn't Lie#

Using an LLM to grade outputs on a 1-to-5 scale without calibration produces noisy, unreliable metrics due to four well-documented biases:

1. Position Bias: Judges systematically favor the first option in pairwise comparisons. Fix: Run every pair twice ((A, B) and (B, A)) and require agreement.
2. Verbosity Bias: Longer, nicely formatted markdown answers receive higher scores even when factually wrong. Fix: Grade atomic factual claims against retrieved ground truth separately from formatting.
3. Score Compression (1–5 Scale Drift): LLM judges rarely output 1 or 2, clustering 90% of scores between 3 and 4. Fix: Replace Likert scales (1–5) with binary pass/fail (True/False) checklists decomposed by specific criteria.
4. Unaligned Rubrics: A judge's raw pass rate might be 80% when the true human-aligned quality is 55%. Fix: Curate a 100-example gold dataset labeled by human domain experts, measure the Judge's True Positive Rate (TPR) and True Negative Rate (TNR), and apply the Rogan-Gladen statistical estimator to compute confidence intervals on every CI/CD evaluation run.