LLMs suck at generating large, structured data. Tips on how to get your AI agent to do it reliably
Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.
Actionable tips for making LLMs output reliable structured data, directly useful for AI agent development.
LLMs fail at generating large structured JSON due to schema drift, all-or-nothing failure, and hallucination, even with structured output modes like OpenAI's response_format. The article proposes a builder pattern where the model calls tools to incrementally accumulate structured data (e.g., for insurance claims), storing it outside the token window to compress conversation mid-flight and avoid coupling research with output. This approach, used by tools like Kiro CLI, solves the semantic problem by shifting the model's role from producing a final blob to orchestrating function calls.