Part three: Managing the machine mind: Who is responsible for AI’s behavior?

An AI agent can answer a customer or act on an invoice before anyone can examine its reasoning. If the outcome is wrong, saying that a human was “in the loop” offers little comfort. Who gave the agent authority? And who could have stopped it?

The first article in this series asked why AI appears to have a personality, even though it does not think or feel like a person. The second explored how psychology could help identify unsafe behavior and improve testing. This final part turns to the workplace: once a business allows AI to influence people and take action, who is responsible for those actions if they negatively impact the business that deployed them?

The question is becoming more pressing as AI moves beyond producing document drafts and recommendations. In PwC’s 2025 survey of 300 senior executives, 79% said their companies were adopting AI agents, but only 17% said agents were fully adopted across almost all workflows and functions. An agent with access to business systems can complete a sequence of tasks, each of which may look reasonable in isolation. Its apparent competence can encourage people to grant it more freedom before they understand how it behaves in everyday work.

Accountability follows authority

When an AI agent causes harm, several parties may have contributed. The model provider built the underlying technology. A business selected it, connected it to data, and decided what it could do. An employee may have reviewed its output. Yet these roles do not carry equal control over a particular decision.

“The organization deploying the agent must own accountability for how it is used,” says Jerry Caviston, CEO of Archive360. A provider may share responsibility for a defect, he explained, but a company cannot give an agent access to its systems and then point to the vendor when something goes wrong.

That distinction matters because deployment choices shape the risk. An agent permitted to suggest a reply presents a different problem from one allowed to send it. The same system might safely identify an invoice discrepancy while being unsuitable to change a supplier’s payment details. Businesses, therefore, need to define an agent’s authority for each task, including the data it may use, but more critically, the conditions under which it must stop and refer to a human.

Donna Wilczek, chief product and technology officer at Basware, places responsibility with the people who design that authority. “Accountability follows ownership of the process, not proximity to the mistake,” she tells ITPro. An executive who approves an agent’s role is responsible for that design; an employee reviewing one exception is responsible for the decision they are genuinely able to make.

This also changes how businesses should describe an AI agent. Calling it a colleague can imply that it has judgment and responsibility of its own. Calling it an ordinary tool can obscure its ability to act across several systems. Luciana Rousseau, head of client innovation at Morson Group, describes agents as “delegated computational actors.” Systems may occupy a role but cannot own the consequences attached to it.

A useful starting point is a named human owner for every important AI workflow. “If nobody’s name is on it, it shouldn’t be running,” says Daria Hvizdalova, head of AI Adoption at Make. That owner must have the authority to change the workflow or suspend it when evidence suggests that its behavior is causing problems.

When human oversight becomes a rubber stamp

A common answer to AI risk is to require human approval. But an approval button alone does not establish meaningful oversight. If an agent produces hundreds of recommendations while a reviewer has seconds to consider each one, the human may become a routine step in a process whose decisions have effectively already been made.

The Morson Group’s Rousseau explained that supervision at machine speed must focus on “exceptions, consequences and behavioural signals.” People should spend their attention on high-impact decisions, unfamiliar circumstances, conflicting evidence and actions that are difficult to reverse. Asking them to inspect every ordinary transaction invites fatigue and habitual approval.s carefully as the system itself.

Hvizdalova offers a way to test for this: “If humans approve 99% of what the AI proposes, you probably have a rubber stamp rather than a meaningful control.” Bastiman agrees from the regulatory side, noting that “an override rate near zero is a red flag, not a success metric,” since it usually means a reviewer is confirming rather than evaluating. Wilczek describes the same failure from the employee’s perspective: handed a long queue with little explanation, “they’ll just click through,” left to absorb the blame for a decision the system had effectively already made.

The reviewer also needs permission to disagree. Microsoft’s 2026 Work Trend Index found that 86% of surveyed AI users said they treated AI output as a starting point and retained responsibility for their thinking. Wilczek emphasizes that an employee faced with a long queue of unexplained recommendations may only be absorbing blame for choices they cannot realistically challenge. A case requiring approval should show what information the agent used and what action it proposes. The employee must be able to investigate and say no without having to fight the workflow.

Consider an agent handling invoices. It could resolve a small variance within a finance team’s agreed tolerance, then send an unusual case to a person with the supporting records. A change to supplier bank details might require a separate verification process outside the agent’s control. These are different decisions within the same workflow, so a single setting labeled “autonomous” or “human approved” would conceal the distinctions that matter.

Oversight should also account for human behavior. A system that usually gets the answer right can make checking feel unnecessary, particularly when it presents its conclusions confidently. Part one of this series examined why people read human qualities into AI’s manner, with part two considering how to test troubling behavior. In production, those same traits affect whether employees question an answer. Leaders must assess the people and pressures around the system as carefully as the system itself.

Watch what happens after deployment

A successful demonstration is evidence that an agent worked under test conditions. It does not establish that the agent will respond consistently to unfamiliar customers or interactions with other systems. Monitoring must, therefore, begin when an agent goes live.

The measurements also need to extend beyond whether tasks were completed. Wilczek suggests watching for downstream signals such as rising invoice recording requests or supplier queries. An agent could appear productive while creating work or risk elsewhere. A process owner needs a way to narrow its permissions while investigating, and have enough information to correct decisions that can be reversed.

This is where a specialist in AI behavior might help. Such a person would look across many decisions for patterns that an individual reviewer cannot see: growing correction rates, persistent confidence despite falling accuracy, unusual action sequences or repeated approaches to a rule’s boundary. The role would require knowledge of the business process as well as the technology.

Dr Janet Bastiman, chief data scientist at Napier AI, cautions against treating terms such as “deception” or “goal drift” as evidence of human motives. What matters is observable behavior, she emphasizes. Whether outputs diverge from stated reasoning and whether the system is pursuing a convenient measure at the expense of the real objective. A specialist who understands model validation but not the work being automated could miss the most consequential change.

This expertise need not always sit in a newly titled job. Risk and operational teams can share the work, provided someone finds patterns and a process owner can act on them. Monitoring has little value if nobody can reduce access or stop the agent.

Give competence a boundary

The hardest systems to govern may be the ones that rarely fail. When an agent performs well most of the time, leaders may be tempted to expand its reach. Employees may stop checking, and customers may assume that a polished answer reflects a sound decision. None of those reactions proves the agent can be trusted with a more consequential task.

If a business cannot establish what information influenced an important outcome or which rule authorized the action, the agent’s independent authority should be limited. It may still recommend an action or handle work that is inexpensive and easy to reverse. Material financial decisions and other consequential actions need stronger evidence and a clear route to human judgment.

The practical task for leaders is to write down those boundaries before deployment, then revisit them as performance changes. For each workflow, they should identify its owner, define the agent’s permitted actions, set escalation points, preserve decision records, and test how quickly authority can be withdrawn. Reviews should consider real outcomes, including complaints and corrections, alongside speed and productivity.

It’s the gap between the appearance of human control and its substance that has revived the question running through this entire series: does the modern workplace need dedicated AI behavior specialists? .

Rousseau believes something like the role is necessary, though she resists calling it “psychologists for machines,” describing it instead as expertise in “the behaviour of the whole human-AI system.” Bastiman is more sceptical of the need for a new title, pointing out that “second-line model validation and model risk management already exist to do this” within regulated industries. Both she and Wilczek caution against borrowing language built for human motive, with Wilczek warning that words like “deception” risk being applied “somewhere it doesn’t quite fit yet.”

That brings the series back to its opening question. AI does not need a psychologist in the human sense. But businesses do need to understand the behavior their systems produce and the habits people develop around them. When an agent appears capable of acting like a colleague, the organization must be especially clear about the one thing it cannot do: take responsibility for its decisions, as an AI is fundamentally a tool for human users, no matter how much autonomy it is granted