A developer wakes up to find their AI agent has sent 300 emails overnight. Every single one is factually accurate. Every single one went to the wrong people.
This is the new problem space. We've spent two years obsessing over whether AI systems hallucinate - whether the facts they generate are true. But as AI moves from answering questions to taking actions, fact-checking becomes almost irrelevant. The real question isn't "is this information correct?" It's "should this thing be doing this at all?"
When AI Systems Act, Not Just Speak
An AI that writes code, sends emails, or deploys infrastructure isn't just generating text you can verify. It's making decisions with consequences. And those consequences don't care whether the underlying facts were correct. They care whether the action was appropriate.
A new framework from developers working with autonomous AI systems breaks this down into four verification layers that have nothing to do with factual accuracy: direction, scope, reversibility, and responsibility.
Direction means: is this action aligned with what the user actually wants? An AI might correctly identify that a codebase has technical debt and accurately refactor it - but if you needed a quick bug fix before a client demo, perfect code that arrives tomorrow is worse than messy code that works today.
Scope means: how far can this thing go? An AI agent with permission to "clean up the database" needs boundaries. Can it delete records? Archive them? Merge duplicates? The facts it uses to identify duplicates might be flawless. The devastation of deleting 10,000 customer records is the same either way.
The Undo Button Problem
Here's where it gets uncomfortable. Some AI actions are reversible. Some aren't. And the ones that aren't reversible need a completely different level of scrutiny before they happen - not after.
If an AI agent generates a pull request, you can review it, reject it, modify it. Low stakes. But if it sends an email to your entire client list, you can't un-send it. If it drops a production database table, "oops, the facts were right but the action was wrong" doesn't bring the data back.
This is why the old verification model - generate output, check facts, approve or reject - breaks down. By the time you're checking, the action might already be irreversible. The verification has to happen before the system acts, not after.
Who Owns the Mistake?
The responsibility question is the one nobody wants to answer. When an AI agent makes a decision, who is accountable for the outcome?
It's easy when a human uses AI as a tool - writes a prompt, gets an answer, decides what to do with it. The human made the decision. But when you give an AI agent autonomy - the ability to act without asking permission each time - that accountability chain gets murky fast.
If your AI agent deploys code that breaks production, is it the developer's fault for not setting better guardrails? The AI company's fault for building an unreliable system? The manager's fault for allowing autonomous deployment? All of the above?
The answer matters, because it determines who fixes the process. And right now, most teams haven't even asked the question.
Building Governance, Not Just Validation
The shift here is from output validation to behaviour governance. You're not checking whether the AI's facts are correct. You're defining what the AI is allowed to do, under what conditions, with what oversight, and with what rollback plan.
That means:
Action permissions, not just data access. An AI agent might need read access to your codebase to understand it. But write access? Deployment access? Those are different permission levels entirely, and they need explicit boundaries.
Approval gates for irreversible actions. Anything that can't be undone needs human sign-off. No exceptions. If your AI agent wants to delete something, send something external, or change production config, it asks first.
Audit trails for everything. When an AI takes an action, you need a log of what it did, why it thought that was the right move, and what data it used to decide. Not for blame - for learning. The next time it makes that decision, you want to know whether the reasoning was sound.
The Real Test
Here's the practical test for whether your AI governance is working: if your AI agent does something catastrophically wrong, can you explain exactly why it happened and what rule would have prevented it?
If the answer is "well, the facts were right, it just... shouldn't have done that" - you don't have governance. You have hope. And hope is not a strategy when you're giving AI systems the ability to act autonomously.
The developers building these systems now are figuring this out the hard way. The rest of us get to learn from their mistakes - if we pay attention. The question isn't whether AI will take more actions on our behalf. It will. The question is whether we'll build the guardrails before we need them, or after.