LLMs suck at generating large, structured data. Tips on how to get your AI agent to do it reliably
Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.
Actionable tips for making LLMs output reliable structured data, directly useful for AI agent development.
LLMs fail at generating large structured JSON due to schema drift, all-or-nothing failure, and hallucination, even with structured output modes like OpenAI's response_format. The article proposes a builder pattern where the model calls tools to incrementally accumulate structured data (e.g., for insurance claims), storing it outside the token window to compress conversation mid-flight and avoid coupling research with output. This approach, used by tools like Kiro CLI, solves the semantic problem by shifting the model's role from producing a final blob to orchestrating function calls.
- Implement tool-based builder methods that let your agent incrementally accumulate structured data, keeping the final output outside the token window to avoid all-or-nothing failures and context overflow.
For engineers building production AI agents that process large documents or require reliable structured outputs, this pattern directly addresses context window limits and hallucination, enabling robust multi-step data extraction without brittle fixup code.