A builder solved the structured output problem. Not with prompt engineering - with a control layer that sits above the LLM and enforces reliability at the system level. The result: 100% structured output success rate in production, handling JSON failures, silent errors, and API outages.
The implementation matters because it addresses the gap between LLM demos and LLM products. Demos tolerate failures. Products don't. This control layer bridges that gap through system design rather than model tuning.
Why Prompt Engineering Isn't Enough
Prompt engineering optimises for average case performance. You craft better instructions, provide examples, tune temperature settings. The model's output improves - from 80% success to 95%, maybe 98% if you're very good.
But production systems need 100%. A 2% failure rate on financial data means wrong transactions. On medical records, it means data loss. On automated workflows, it means manual intervention - which eliminates the automation value entirely.
The builder's insight: treat the LLM as an unreliable component and build a reliable system around it. Don't fight the model's statistical nature - design for it.
The Control Layer Architecture
The control layer intercepts every LLM response before it reaches application logic. It validates structure, catches malformed JSON, detects silent errors, and handles API failures. When something breaks - and in production, something always breaks - the control layer manages recovery without surfacing errors to users.
The validation happens in stages. First pass: is the response valid JSON? Second pass: does it match the expected schema? Third pass: do the values make semantic sense? Each stage has specific recovery strategies.
For malformed JSON, the control layer extracts partial data and requests completion. For schema mismatches, it identifies missing fields and prompts specifically for those fields. For semantic errors - like negative quantities or future dates where past dates are required - it flags the issue and requests correction.
This isn't one big retry loop. It's targeted recovery based on failure type. That distinction matters for both cost and latency.
Handling Silent Errors
Silent errors are worse than obvious failures. The LLM returns valid JSON that matches your schema, but the content is wrong. A date in the wrong format. A category that doesn't exist in your system. A quantity that's plausible but incorrect.
The control layer implements domain-specific validation rules. For each field, it knows what valid looks like - not just type, but allowed values, realistic ranges, consistency with other fields. It catches errors the LLM can't self-detect because the LLM doesn't know your business rules.
The builder's implementation includes confidence scoring. When the control layer detects something questionable but not definitively wrong, it flags it for human review rather than blocking the workflow. This handles edge cases without building brittleness into the system.
API Outage Resilience
LLM APIs go down. OpenAI has outages. Anthropic has outages. Every provider has outages. The control layer treats this as normal and routes around it.
The implementation maintains a priority list of LLM providers. Primary provider fails? Switch to secondary. Secondary fails? Tertiary. The switching happens automatically, preserving the same prompt and validation logic across providers.
This requires provider-agnostic prompt design - no provider-specific features, no reliance on unique capabilities. That's a constraint, but it's the price of resilience. The builder argues it's worth it: a system that works 100% of the time with slightly less optimal prompts beats a system that works 99% of the time with perfect prompts.
Production Results
The builder reports 100% structured output reliability over thousands of production requests. Not 99.9% - actually 100%. The system hasn't shipped a malformed response to application logic since deployment.
Cost increased by roughly 15% due to validation overhead and occasional retry requests. Latency increased by an average of 200ms per request. Both are acceptable trade-offs for eliminating failures entirely.
The most interesting result: developer velocity improved. Engineers stopped writing defensive code around LLM responses. They stopped handling edge cases in application logic. The control layer became the single point where reliability is enforced, and everything downstream could assume clean data.
What This Means for Builders
If you're building production systems with LLMs, this architecture is worth studying. The core lesson isn't the specific implementation - it's the approach. Treat the LLM as unreliable by design. Build reliability into the system, not the prompts.
Prompt engineering still matters for quality. But reliability comes from architecture. Validation, recovery strategies, provider failover, domain-specific rules - these are system design problems, not prompt design problems.
For business owners evaluating LLM implementations: ask about the control layer. If the answer is "we have really good prompts," that's not enough. You need system-level reliability guarantees, not model-level optimisations.
The 100% reliability claim sounds ambitious, but the architecture justifies it. When you validate every response, implement targeted recovery for every failure mode, and maintain provider redundancy, you can actually achieve it. That's the difference between a demo and a product.