By Rami Heera, VP and Practice Lead, Advisory at phData
Only 13% of business and IT leaders believe their organization has adequate governance in place for AI agents, despite 75% of IT leaders currently piloting or deploying them.
Developing an AI governance framework starts with treating agents like new employees: scoped access, a tracked identity, and a named owner accountable for what they do.
Human-in-the-loop review breaks down under volume and turns into a rubber stamp unless reviewers understand what they’re approving and clear escalation thresholds are in place.
Every new employee who touches enterprise systems goes through the same ritual. IT provisions an account. Someone decides what that account can see and touch. A manager signs off. There’s a name attached to every action that person takes, and a clear line back to who approved their access in the first place.
AI agents get none of that. They’re moving into the same systems right now, making the same kinds of decisions, and in most organizations, nobody ran the ritual. This is what enterprise AI governance actually looks like: the same rules already applied to people, extended to the systems now acting for them.
I’ve started thinking about agents almost the way I think about new hires. The way a human comes into an organization, gets access to certain systems, and gets access to data isn’t all that different for an agent.
The gap is what an agent doesn’t have: years of judgment about what looks normal and what doesn’t.
According to Gartner, only 13% of business and IT leaders believe their organization has adequate governance in place for AI agents, despite 75% of IT leaders currently piloting or deploying them. They’re missing the basics: clear boundaries between what an agent can decide on its own and what needs human approval, real-time monitoring of agent behavior, and audit trails that show what an agent actually did.
Most organizations have no one person responsible for closing that gap. When an agent gets it wrong, there often isn’t a clear answer for who owns the outcome.
The dangerous governance gap
Why AI agents carry a different risk than AI that only advises
For most of the last decade, enterprise AI sat in an advisory role. A model would surface a recommendation, flag an anomaly, or rank a set of options, and a person decided what to do with that information. If the model was wrong, the damage was bounded by the human step in between.
That step absorbed the risk.
Agentic AI removes that step. An agent can execute a price change instead of only recommending it. It can approve, route, and close out an invoice instead of only flagging it for review.
Once AI moves from advising to acting, the risk profile changes completely, and so does the question of who is responsible when something goes wrong. This is why I push clients to treat governed multi-agent architecture as a high priority design decision, supported by organizational outcomes that drive real revenue lift or risk/cost mitigation. If you try to retrofit once an agent is already live, the risk to your organization has already been exposed, and you will spend several times the expected ROI in mitigation activities.
A bad recommendation is a data quality problem. A bad autonomous action is an incident, and increasingly, a compliance one that will hurt your brand and your bottom line.
The EU AI Act’s transparency requirements and similar state-level rules like Colorado’s are turning agent accountability from a best practice into a legal expectation, which means the organizations still treating governance as optional are running out of room to do that.
Treat AI agents like new employees, not software features
The instinct in most organizations is to treat an AI agent like a software deployment: build it, test it, deliver it, monitor its uptime. That instinct misses what actually makes agents risky.
A new hire with system access and an AI agent with system access create the exact same category of exposure. The difference is that a new hire spent years building the judgment to know when something looks wrong. An agent has none of that built in.
The questions I keep coming back to with clients: What does it have access to? What doesn’t it have access to? How does it treat, manage, and handle data? Those are the guardrails that determine whether an agent produces not just high-quality outputs, but outputs that hold up once security, risk management, and compliance get involved.
Building an AI governance framework around that question means treating access as a design decision, not an afterthought. Every agent should get scoped, least-privilege access to only the systems and data it needs for its specific task, the same way a new analyst doesn’t get admin rights to the finance system on day one.
The exception here would be if there are subsets of data and system access that pose low risk, and continue to encourage experimentation and innovation without risking organizational security. The aim is to create holistic human agent operating models that do not contravene the security measures we have spent decades building, while not stifling growth through AI. Ultimately, every agent needs an identity that can be tracked, an owner who is accountable for its behavior, and an audit trail that captures what it touched and why.
None of this is new thinking. It’s the same access control and data lineage discipline enterprises already apply to human employees, and it’s the same discipline our Intelligence Platform builds into every agent before it reaches production.
What’s new is that most organizations haven’t extended it to agents yet. An agent doesn’t show up in an HR system asking to be onboarded. It shows up in a sprint backlog asking to be deployed.
What happens when human-in-the-loop becomes a rubber stamp?
The default answer I hear most often to “how do we control this” is to put a person in the approval path and let them catch what the agent gets wrong, also known as human-in-the-loop. In my experience, that safeguard breaks down faster than most organizations expect.
As Dr. Fern Halper, founder of AI Foundations Group, put it in a recent industry analysis, the question is: “Is the human actively reasoning and participating in the work or are they simply reviewing outputs and then clicking approved, because those are two very different governance and cognitive models.”
Is the human actively reasoning and participating in the work or are they simply reviewing outputs and then clicking approved, because those are two very different governance and cognitive models.
When approval volume climbs, reviewers stop reasoning through each decision and start clicking through them. The control still exists on paper, but it’s no longer serving the purpose it was designed for.
This is where fixing one process step gets confused with building oversight that actually works. Adding a review gate to a single workflow doesn’t teach the organization what to look for the next time a different agent touches a different system.
Institutional learning means the person approving an agent’s action actually understands what that action does, what could go wrong, and what the acceptable range of outcomes looks like, not just that a button exists for them to click.
Without that understanding, a human-in-the-loop is a compliance checkbox disguised as oversight. It fails at exactly the moment it matters most: when an agent does something unusual that a distracted reviewer waves through, out of habit.
Who owns enterprise AI governance today?
Ask most enterprises “who owns enterprise AI governance?”, and the answer sprawls across the org chart.
The CIO owns the infrastructure the agents run on. The CDO owns the data they touch. The CFO increasingly owns the budget conversation about where agentic AI is allowed to spend money autonomously, and the value being generated against cost (primarily unfettered token usage). An AI council, if one exists, owns a set of principles that rarely translates into day-to-day approval decisions.
There isn’t one singular body that really manages this today, and that’s part of the challenge in front of us.
That diffusion has a measurable cost. McKinsey’s 2026 AI Trust Maturity Survey found that organizations with explicit, named ownership for responsible AI, through a dedicated governance role, audit function, or ethics team, score an average of 2.6 on its maturity model, compared to 1.8 for organizations without a clearly accountable owner.
I’ve previously made this same argument in phData’s approach to responsible AI governance: named ownership isn’t a formality. It’s the difference between a governance framework that exists on paper and one that actually changes what happens the next time an agent is proposed for a new use case.
A governed structure that works doesn’t route every decision through one overloaded committee. It assigns clear accountability at three levels:
who owns the overall agent portfolio and its risk posture
who owns the platforms and guardrails agents run inside of
who owns each individual use case and is answerable for its performance and compliance.
Portfolio-level ownership sets the risk appetite and reports it to the board. Platform-level ownership makes sure the guardrails, identity controls, and monitoring exist for every agent to plug into, instead of each team building its own from scratch. Use-case-level ownership is the person who answers for a specific agent’s decisions the way a manager answers for a direct report’s work.
AI councils are a reasonable first step toward this, and many organizations are standing them up right now, but a council that meets quarterly to set policy is not the same thing as an accountability structure that functions day to day. Most councils are good at writing principles and bad at owning the approval decision for the agent request that lands on someone’s desk next week.
Closing that gap means giving the council, or whatever body replaces it, actual decision rights over specific use cases, not just a seat at the table when something goes wrong.
The organizations closing this gap are the ones building governance around named owners and defined access, the same architecture they’d use to onboard a new employee, before they scale the next wave of agents into production.
Governance built in from the start, not bolted on after the first incident, is what makes it possible to deliver AI in production at scale without carrying that exposure.
Give your board a governance structure they can actually see
The enterprise Intelligence Platform blueprint breaks down the four governance capabilities that turn enterprise AI governance into a system.
FAQs
What is an AI governance framework?
Enterprise AI governance runs on an AI governance framework, the policies, roles, and controls that define how an organization approves, monitors, and holds AI systems accountable throughout their use, from the data they can access to who is responsible when they act. For agentic AI, that framework has to cover autonomous actions, not just model outputs, which means defining access permissions, audit trails, and named ownership before an agent goes into production.
Why do AI agents need to be governed differently than traditional AI models?
Enterprise AI governance has to treat AI agents differently than traditional AI models, which only advise while a person acts on the recommendation. AI agents act directly, executing decisions, calling systems, and changing data without a human step in between. That shift means the risk of a mistake is no longer absorbed by a person reviewing the output, which is why agents need the same access controls, identity, and accountability structure enterprises already require of employees.
How is accountability assigned when an AI agent makes a mistake?
Enterprise AI governance is missing a single accountable owner at most companies today, with responsibility split across the CIO, CDO, CFO, and any AI council that exists. Deloitte’s 2026 State of AI in the Enterprise report found that only 21% of organizations have a mature governance model for agentic AI, and closing that gap requires assigning a named owner at the portfolio, platform, and individual use case level.
Is human-in-the-loop enough to govern AI agents safely?
Enterprise AI governance can’t rely on human-in-the-loop review alone, even though it’s necessary. When approval volume grows, reviewers can start rubber-stamping agent decisions instead of genuinely evaluating them, which turns the safeguard into a compliance formality. Effective oversight requires reviewers who understand what they’re approving, combined with defined escalation thresholds and audit trails, not just a person in the approval chain.
How should enterprises start building governance for agentic AI?
Enterprise AI governance starts by treating each agent like a new employee: define what systems and data it can access, assign a named owner accountable for its performance and compliance, and build an audit trail before the agent goes into production. Organizations with explicit, named ownership for AI governance score meaningfully higher on maturity models than those with diffused or unclear accountability, according to McKinsey’s 2026 AI Trust Maturity Survey.
