Falling Token Prices Hide Rising AI Bills

Today's Overview

Here's a curious paradox in AI economics right now: token prices have dropped dramatically-we're talking 10 times cheaper per unit in some cases-yet companies are seeing their total AI bills go up, not down. The culprit isn't the price. It's the complexity.

When you run a simple question through an LLM, it uses maybe 50 to 500 tokens. But when you ask an AI agent to reason through something, refine its answer, call external tools, and try again if it fails, the token count explodes. A complex coding task might burn through 50,000 to 100,000 tokens. Legal document analysis? 250,000 plus. Multi-agent workflows can hit a million tokens per task. Each step in the reasoning chain adds cost-and the complexity keeps growing as we push AI into harder problems.

The Real Cost Bottleneck

This matters because companies like Notion have seen measurable margin compression from AI inference costs-a 10 percentage point hit to profit margins is not a rounding error. Some users are consuming tens of thousands of dollars in compute while paying flat-rate subscription fees. The economics are brutal when you can't see the cost breakdown. Teams need granular visibility: cost per workflow, cost per feature, cost per customer. Without it, you're scaling usage blind and margins evaporate. AI FinOps-financial operations for AI-is emerging as the discipline that might actually make this manageable, treating token consumption as a constrained resource the way early cloud teams learned to manage compute.

Meanwhile, at GTC this week, Jensen Huang was clear: energy is the constraint. Not chips, not fab capacity-energy. And energy efficiency means the work you do with it has to matter. Vera, Nvidia's new CPU, is designed specifically for agents waiting on tools, because you can't afford to idle a GPU while waiting for a CPU to fetch data. The entire stack is getting rearchitected around inference, and the implication is clear: if you're building AI features, you need to be obsessive about the tokens you're burning.