I Built an Agent Eval Harness. Real Agents Broke the Clean Version of the Story
6.1 relevance
Score Breakdown
technical depth 7
novelty 6
actionability 6
community 4
strategic 4
personal 8
Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.
Agent eval harness is directly relevant to agent orchestration and evaluation.
Summary
AgentEval Forge, now on PyPI, is an open-source evaluation harness for AI agents that uncovered how real-world agents break clean evals during field testing with 19 LangGraph and PydanticAI agents from 150+ repos. It provides adapters for five agent surfaces (subprocess, HTTP, LangGraph, PydanticAI, etc.), 20 launch scenarios, 17 deterministic scorers, and 11 LLM-as-judge metrics, with a security model and CI integration. The project's key insight: agent evaluation must judge the run, not just the answer, and safety failures trump all other metrics.