Evaluation harness
An automated system that runs a curated set of inputs through an AI system, scores the outputs, and catches regressions before they reach customers.
An evaluation harness is the scaffolding that makes an AI system improvable. It runs a curated set of representative inputs - the golden set - through the system on every code change, every vendor model update and every release. It scores the outputs, compares them against known-good behaviour, and blocks deploys that regress beyond a threshold.
Without a harness, every change to an AI system is a coin flip. With one, the team can swap models, tighten prompts and argue from numbers rather than intuition.
What it contains
A production-grade evaluation harness has four moving parts:
- A golden set of 100–500 representative inputs with known expected behaviour, version-controlled and reviewed quarterly.
- Automated scoring - exact-match for structured tasks, rubric-based LLM-as-judge for open-ended outputs, human review for the high-stakes tail.
- Production sampling - a small percentage of live traffic flows through the same scorer, with low-confidence outputs queued for human review.
- A red-team catalogue of adversarial probes run continuously. See red-team harness.
Why it's under-invested in
Harnesses are boring. Demos are exciting. The teams that invest in a harness on week one ship production systems; the teams that defer it ship pilots that quietly regress and get pulled a year later.
How Safemode ships one
Every Safemode engagement includes an evaluation harness as a week-one deliverable, not a phase-two upgrade. It's wired into CI, runs on every merge, and is handed to the client's team with a written operating guide.