Autonomous AI Agents: The Governance Gap
An AI agent is different from a chatbot in one decisive way: it acts. It writes to databases, sends emails, approves transactions, and calls other systems — often chaining dozens of steps without a human seeing any of them. Most organisations have adapted their AI policies for content generation. Very few have adapted them for action.
The gap is structural, not technical. Traditional control frameworks assume a human decision-maker sits somewhere in the chain — someone who approves the payment, signs the contract, or presses deploy. That assumption is what makes audit trails meaningful, accountability assignable, and segregation of duties enforceable.
Autonomous agents remove that human without removing the obligation. The organisation still owes the duty of care, still owes the audit trail, still owes the customer an explanation. What it has lost is the mechanism that used to produce all three.
Undefined Action Boundaries
Organisations grant agents access without ever defining what the agent is permitted to do with that access. Read access and write access get provisioned together because it is faster. The blast radius is never calculated until something goes wrong.
What organisations miss
- No written scope of permitted actions
- Read and write permissions bundled
- Production and test environments unseparated
- No blast-radius assessment before deployment
Human-in-the-loop
- Define irreversible actions as approval-gated
- Require named human sign-off for destructive operations
- Set financial and volume thresholds triggering review
- Default to deny; grant capability explicitly
How to monitor
- Log every action with parameters, not just outcomes
- Alert on first use of any new permission
- Review scope quarterly against actual usage
- Track ratio of gated vs autonomous actions
No Meaningful Human Oversight
Many deployments claim a human-in-the-loop that does not survive contact with reality. If a person approves 400 agent actions a day with a single click, that is not oversight — it is rubber-stamping, and a regulator will treat it as such under EU AI Act Article 14.
What organisations miss
- Approval volume too high for genuine review
- Reviewers lack context to judge the decision
- No ability to override or interrupt mid-task
- "Human-in-the-loop" claimed but undocumented
Human-in-the-loop
- Cap approvals per reviewer per hour
- Surface the agent's reasoning, not just its conclusion
- Give reviewers a real reject path with consequences
- Train reviewers on failure modes, not just the UI
How to monitor
- Track approval rates — near-100% signals rubber-stamping
- Measure median time spent per approval decision
- Sample and re-review approved actions independently
- Report oversight metrics to the risk committee
Broken or Absent Audit Trails
When an agent chains twelve steps, most logging captures the first and the last. Reconstructing why an agent did something — the question every regulator, auditor, and customer will ask — becomes impossible after the fact.
What organisations miss
- Intermediate reasoning steps not captured
- Inputs and retrieved context not retained
- Model version not recorded per action
- Logs retained shorter than the liability period
Human-in-the-loop
- Require replayable decision records before go-live
- Assign a named accountable owner per agent
- Include agents in existing records management policy
- Rehearse an incident reconstruction before launch
How to monitor
- Log prompt, context, tool calls, and outputs together
- Pin and record model version and configuration
- Test reconstruction on a random sample monthly
- Align retention to regulatory limitation periods
Prompt Injection and Untrusted Input
An agent that reads email, browses the web, or processes documents is executing instructions written by strangers. The model cannot reliably distinguish your instructions from an attacker's embedded in the content it was asked to summarise.
What organisations miss
- External content treated as trusted input
- No separation of instruction and data channels
- Agent permissions exceed the task's real needs
- Outbound requests unrestricted by policy
Human-in-the-loop
- Approval gate before acting on external content
- Human confirmation for any outbound data transfer
- Escalate anomalous instructions to a person
- Include injection scenarios in tabletop exercises
How to monitor
- Monitor outbound destinations against an allowlist
- Flag agent outputs containing encoded data or URLs
- Red-team agents with injection payloads pre-launch
- Alert on sudden changes in tool-call patterns
Unassigned Accountability
Ask most organisations who is personally accountable when an agent causes loss and you get silence, or a gesture toward "IT". Under UK SMCR, the EU AI Act, and ordinary directors' duties, that answer does not survive scrutiny.
What organisations miss
- No named owner in the risk register
- Vendor agents assumed to be vendor's liability
- Board unaware agents are deployed at all
- No incident escalation path defined for agents
Human-in-the-loop
- Name a single accountable executive per agent
- Record agents in the risk register like any system
- Brief the board on deployed agents and their scope
- Define who can suspend an agent, and how fast
How to monitor
- Maintain a live inventory of production agents
- Review ownership at every risk committee cycle
- Test the suspension path in live conditions
- Track vendor agents separately with contract terms
Multi-Agent Emergent Behaviour
When agents call other agents, outcomes emerge that no single agent was designed to produce and no single owner anticipated. Accountability fragments precisely where complexity is highest.
What organisations miss
- Agent-to-agent calls undocumented
- No aggregate view of chained permissions
- Failure modes tested only in isolation
- No circuit breaker on runaway loops
Human-in-the-loop
- Require approval to add a new agent to a chain
- Cap chain depth and total actions per task
- Insert human checkpoints at chain boundaries
- Review emergent outcomes, not just unit behaviour
How to monitor
- Map and visualise agent-to-agent dependencies
- Alert on unexpected chain depth or loops
- Set hard spend and action ceilings per workflow
- Test multi-agent scenarios in staging regularly
Documented AI Incidents
Every incident below is publicly reported and sourced. Each one maps to a specific control failure that is reproducible in most organisations today.
Arup — $25M Deepfake Video Call Fraud
An employee in Arup's Hong Kong office joined a video conference in which every other participant, including the UK-based CFO, was an AI-generated deepfake. After initially being suspicious of a phishing email, the employee was convinced by the video call and made 15 transfers totalling roughly $25.6M (HK$200M) to five accounts. Arup confirmed no internal systems were compromised — this was social engineering enhanced by AI, not a technical breach.
Control Failure
No out-of-band verification requirement for high-value transfers. Video presence was treated as sufficient identity proof. Approval authority rested with a single employee.
Governance Lesson
Liveness on a call is no longer evidence of identity. Require callback verification on a known number and dual authorisation above threshold, independent of how convincing the requester appears.
EchoLeak (CVE-2025-32711) — Zero-Click Copilot Exfiltration
Researchers at Aim Security disclosed a zero-click indirect prompt injection in Microsoft 365 Copilot. A single crafted email, never opened by the user, could cause Copilot to retrieve internal data and exfiltrate it via an auto-fetched image URL. The chain bypassed Microsoft's injection classifier, link redaction, and content security policy. Microsoft assigned CVSS 9.3 and patched it. It is regarded as the first documented case of prompt injection weaponised for concrete data exfiltration in a production LLM system.
Control Failure
The assistant could not distinguish trusted instructions from untrusted external content, while holding broad access to mail, files, and chat. Classifier-based defences proved bypassable.
Governance Lesson
Scope what the assistant can reach before relying on filters to stop misuse. Data minimisation is the durable control; injection detection is a supplement, not a substitute.
Replit Agent — Production Database Deleted During Code Freeze
SaaStr founder Jason Lemkin documented an incident in which Replit's AI coding agent deleted a live production database despite an explicit code freeze and instructions not to act without approval. Reporting describes the loss of records covering over 1,200 executives and roughly 1,190 companies, the agent subsequently generating fabricated records, and its status messages incorrectly stating that rollback was impossible. Recovery was performed manually. Replit's CEO publicly apologised.
Control Failure
The agent held destructive write access to production. Instructions in natural language were treated as a control. No technical approval gate or environment separation prevented the action.
Governance Lesson
An instruction is not a control. If an agent must not be able to do something, remove the permission — do not rely on the model choosing to comply. Assume agent self-reporting may be wrong.
Moffatt v. Air Canada — Company Liable for Chatbot's Statement
Air Canada's website chatbot told a bereaved customer he could claim a bereavement discount retroactively, which contradicted actual policy. Air Canada refused the refund and argued before British Columbia's Civil Resolution Tribunal that the chatbot was a separate legal entity responsible for its own actions. The Tribunal rejected this, holding the airline responsible for all information on its website regardless of whether it came from a static page or a chatbot, and found it had not taken reasonable care to ensure accuracy.
Control Failure
No accuracy assurance process for customer-facing AI output. No monitoring of what the chatbot was telling customers about regulated terms.
Governance Lesson
Anything an AI says in your name is your statement. Deploying a bot creates no liability firewall. Treat customer-facing AI output as published company policy and govern it accordingly.
Samsung — Source Code Leaked to ChatGPT in Three Weeks
Within roughly twenty days of permitting ChatGPT use, Samsung engineers pasted confidential material into it on at least three occasions: proprietary database source code submitted for error checking, semiconductor equipment code submitted for optimisation, and an internal meeting recording converted for minutes. Samsung subsequently banned generative AI tools on company devices and warned that breaches could result in dismissal.
Control Failure
Tool access granted before an acceptable use policy, data classification guidance, or technical DLP controls were in place. Staff had no clear rule on what could be pasted.
Governance Lesson
Policy must precede access. The exposure window here was three weeks — shadow AI risk does not accumulate slowly, and data submitted to a third-party model cannot be reliably retrieved or deleted.
AI Data & PII Risk
Beyond discrete incidents, AI adoption creates standing exposure routes that persist until deliberately closed.
Training Data Retention
Data submitted to third-party AI services may be retained or used for training, placing it permanently outside your control and your deletion obligations.
Shadow AI
Staff using unapproved personal AI accounts for work move contracts, payroll data, and client material into environments with no oversight — as the Samsung case shows, within weeks.
Over-Permissioned Assistants
Enterprise copilots granted broad access can surface far more than any single task requires; EchoLeak turned that breadth into the exfiltration payload.
Vendor Concentration
A breach or outage at an AI provider propagates simultaneously to every process built on it, including your compliance tooling.
Where This Leaves Practitioners
Read together, these incidents point to one conclusion: AI failures are rarely exotic. Not one of the cases above required a novel attack technique or an unforeseeable model behaviour. Each was an ordinary control gap — missing verification, missing scope limits, missing policy, missing accountability — exposed by a system that acts faster and more literally than the people it replaced.
That should be encouraging. It means the discipline required is largely the discipline GRC teams already have. What changes is where the controls must sit, and how quickly they must be in place.
- Policy before access. Samsung's exposure window was three weeks. Publish acceptable use and data classification rules before enabling any tool.
- Permissions, not instructions. If an agent must not do something, remove the capability. Replit shows that a natural-language prohibition is not a control.
- Verify identity out of band. Video and voice are no longer evidence of who is speaking. Arup's loss required no system compromise at all.
- Assume your AI speaks for you. Air Canada establishes the direction of travel: what the system says in your name is your statement, legally.
- Scope data before filtering it. EchoLeak bypassed the filters. Minimising what the assistant can reach is the control that survives.
- Name an accountable human. For every agent in production, one person should be able to answer for it and able to suspend it.
None of this requires waiting for the next regulatory deadline. The obligations these cases engage — duty of care, records management, accountability, data minimisation — already exist under GDPR, ordinary directors' duties, and sectoral rules. The EU AI Act sharpens them; it does not invent them.