Which Work Can You Let an AI Agent Run Unattended? A Test for Leaders

Let an AI agent run unattended only where three things hold: a wrong result would be visible quickly, somebody would notice it without being told to look, and the work could be put back. Everything else either waits for a named person to release it, or stays with a human.

That decision is being made for you this term. Somewhere in your organization a piece of software will pick up a job, work through six or seven steps and finish it, and nobody will watch it happen. Not because a leadership team decided that. Because the feature arrived switched on inside a platform you already use.

An AI agent, once you strip the vendor language off it, is not a smarter chatbot. It is software that carries out a sequence of actions toward a goal instead of handing you a draft and stopping. The change is not capability. It is attendance.

So the question in front of a leadership team is not which agent to buy. It is which work may run without a person watching, and what has to be true before you say yes. Most teams answer in four words: a human checks it. Research published in August 2026 argues that those four words, on their own, are not a safeguard. They are a hope.

What does an AI agent actually change?

An agent changes who is in the room while the work happens.

A chatbot produces something and waits. You read it, you fix it, you send it. An agent takes the goal and the permissions, and then reads, writes, decides and acts across several steps before anyone sees the result. Two examples, both reported by Government Technology on July 22, 2026. Carrie Pitts-Densmore of Johnston Community College describes agents working inside the enrollment pipeline: "We do use agents to look at student records, so that we can replicate what our employees do." And Joe Siedlecki of Amira Learning, which builds AI reading coaches and literacy screening tools for schools, gives the plainest working definition of an agent I have read: "It's making decisions pinned back to a specific goal, and the intent is to observe and change action."

This is arriving faster than most governance is. Microsoft's 2026 Work Trend Index, published on May 5, 2026, reported fifteen times year-on-year growth in active agents across Microsoft 365, and eighteen times in large enterprises. The same report found that only 26 percent of the AI users it surveyed said their leadership was clearly and consistently aligned on AI. Agents are multiplying inside organizations whose leaders have not yet agreed on a position.

Why is "a human in the loop" not automatically a safeguard?

Because the job of watching an agent wears down the person doing the watching.

That is the argument in AI Agents Push Humans Out of the Loop, published on August 24, 2026 (revised September 6, 2026) by Margaret Mitchell, Avijit Ghosh and Samir Passi of Hugging Face and Data & Society. It is a position paper, an argument rather than a trial. Their case is uncomfortable and, I think, correct. Systems that require human oversight are currently built in ways that erode the capacities oversight depends on: "Overseers must process voluminous outputs generated faster than they can meaningfully review them." What follows is what the authors call approval fatigue, "in which users stop paying close attention to what they are approving." Worse, the systems learn from those approvals. If approvals are read as evidence of success, the paper warns, a system "can begin to optimize for disincentivizing thoughtful oversight: producing confident summaries, reducing friction, and surfacing oversimplified plans that are easy to skim." The loop optimizes against the scrutiny it exists to provide. Oversight, in their phrase, degrades the overseer.

Sitting alongside that is a finding from Princeton about the machines themselves. In Towards a Science of AI Agent Reliability, published on February 18, 2026, Stephan Rabanser, Sayash Kapoor, Arvind Narayanan and colleagues separate an agent's accuracy from its reliability, and report that "reliability gains lag noticeably behind capability progress." Agents have become markedly more capable. They have not become correspondingly consistent, which is why the authors measure whether an agent behaves the same way across repeated runs of the same task. The paper's own illustration, a hypothetical in its appendix, is the one to carry into a meeting: an airline refund agent that might approve the refund on three of five attempts and deny the identical request on the other two.

Put those two findings beside each other and you get the practical problem. An agent that is wrong in the same way every time is annoying but supervisable, because a person learns where to look. An agent that is right most of the time and differently wrong the rest of it is what a tired reviewer is least equipped to catch. That is the combination arriving in your organization this term.

What is the Loop Test?

The Loop Test is three questions I ask before accepting "a human checks it" as an answer: Would that person notice a wrong result without being told to look for it? Would they notice in time to stop it mattering? And if they missed it, could the work be put back? Oversight that survives all three is a safeguard. Oversight that fails any of them is a signature.

1. Would they notice? Not could they, if they read every line. Would they, at four o'clock on a Thursday, with the rest of the job still to do. If the answer depends on someone reading output at volume, you have built approval fatigue into your own controls. A reviewer who checks exceptions and samples will notice. A reviewer asked to read everything will notice nothing.

2. Would they notice in time? A wrong predicted grade caught in the same afternoon is an error. The same error caught after it has gone to a parent is an incident. Ask where the work lands and how long it sits there before it becomes real to somebody outside the building. That interval, not the accuracy rate, is your actual margin.

3. Could it be put back? A draft can be rewritten. An email cannot be unsent, a record written into a student information system is a job to unpick, and a message to a family cannot be recalled at all. Reversibility is the simplest safety feature you own, and the one most often given away to save a step.

This is my suggested way of thinking about agent oversight, not an established standard. It is deliberately narrow. It says nothing about whether a decision is yours to make, which is a separate question I have written about in which decisions AI should never make. The Loop Test is about work, and about whether anyone would notice if the work went wrong.

How do you design review so the reviewer still notices?

Design the review around the reviewer's attention, not the agent's output: show people fewer things, make them commit before they see the answer, and protect the skill they are being asked to use.

Mitchell, Ghosh and Passi do not stop at the diagnosis. Drawing on automation and human-computer interaction research, they propose design features for the people who build agents and working protocols for the organizations that deploy them. Several translate straight into a school office. Five are worth a leadership team's time, and I have put them in the order I would try them:

1. Review batches, not events. Look at a completed unit of work, such as a morning's absence letters, rather than approving each action as it scrolls past. The paper pairs batch review with bounded autonomy, so the agent can only do so much before it stops.
2. Commit before you look. For the few high-stakes items, ask the reviewer to write down their own view before they see the agent's recommendation. The paper calls this pre-commitment. A reviewer who has already decided cannot simply agree.
3. Plant a known error. The authors suggest canary tasks and audits that compare an agent's summaries against its action logs. A hypothetical example: once a month the attendance lead slips two letters carrying a deliberate fault, a wrong date or the wrong sibling's name, into the batch an agent has drafted, and notes whether anyone caught them. Tell the team the practice exists. The point is to measure the review, not the reviewer, and to find out before a parent does whether the review is real.
4. Keep the skill alive. The paper calls for regular unassisted practice of the core task and for rotation between assisted and unassisted work. A person who has not written an absence letter by hand in a year is a poor judge of whether the agent's letter is right.
5. Separate the reviewer from whoever benefits from speed, and drop the speed metrics. The paper recommends that oversight sit apart from the person who gains when approvals come fast, and that productivity measures that discourage scrutiny be removed. If you grade the office on how quickly the queue clears, you have told them not to look.

The authors also list review time and override rates among the signals worth watching. My reading, and it is only that, is that a reviewer who has overridden the agent twice in a term is either supervising an unusually good agent or not supervising. You will want to know which.

The paper presents these as proposals, not as tested practice, so treat them as things to try on one agent and observe, not as a standard.

Which work should run unattended, and which should not?

Every task an agent could do belongs in one of three places, and naming the place is the decision.

Placement	What it means	When it fits
Hand it over	The agent runs, unattended, and the output is used	Wrong results are visible, quickly, and can be undone
Hold the send	The agent does the work, a named person releases it	Wrong results are recoverable only before they leave the building
Keep it human	The agent does not touch it	The judgment is the work, or the harm is not reversible

Most organizations run into trouble because they have only two settings, on and off, and then find that "on" quietly means "hand it over." Hold the send is the setting that is missing, and it is the one that does most of the work. It takes a few seconds an item and keeps a name attached to everything that leaves the building.

The Unattended Work Check: six lines to settle before the next leadership meeting

Six lines, filled in for one agent, turn a discussion into a decision. Copy them.

1. The agent. Name it, and name the platform it arrived inside. If you cannot name the platform, you are not describing an agent you chose.
2. The work. One task, in a sentence a parent would understand. Not "communications." The letter that goes home when a child is absent.
3. The placement. Hand it over, hold the send, or keep it human. One of the three, written down.
4. The person. A name, not a team. "The office" is not a person and has never checked anything.
5. What that person looks at. Exceptions and flagged uncertainty, with a number against it: twelve items, not four hundred.
6. The review date. Agents change when the vendor ships an update, so a decision made in October is a decision about October.

Read the six lines back, and the pattern of what is blank tells you where you actually are. All six filled in is a decision. Lines one to three filled in and four to six blank is the commonest result, and it is a placement with nobody behind it. Line three blank is the position most organizations are in without knowing it: the agent is running, and nobody has said that it may.

This is a suggested way of ordering the decision rather than a standard anyone has validated. What I can say for it is that the shape holds at every size. The advisory work I have done with the UK Department for Education, KHDA in Dubai and the Ministry of Education in Kazakhstan comes down to the same sentence a single school has to write: who may let software act without being watched. Systems and single schools get stuck in the same place, at line four.

What I Tell Leadership Teams

The mistake I see most often is that leadership teams answer "who checks it?" with a role rather than a person.

Across the leadership teams I work with, in schools and in organizations outside education, the conversation usually runs the same way. Someone says the office team will review it. I ask which member of the office team, on which day, looking at what. The room goes quiet, because the honest answer is that everyone assumed someone else was reading it. That is not carelessness. It is what happens when oversight is assigned rather than designed.

The teams that get this right do one small, specific thing: they stop asking a person to read the output and start asking the agent to surface its own uncertainty. Flag what you were unsure about. List what you changed. Show me the three you would not bet on. A reviewer given twelve exceptions does a proper job. The same reviewer given four hundred approvals does not, and the research above explains why.

I write for Forbes about AI and work, and I spend most weeks in rooms with leaders whose software is further along than their policies are. That gap, between what an organization's tools can now do on their own and what its leaders have agreed they may do, is the widest I have seen it.

What do agents do to jobs, and what should you tell staff?

They take tasks, not jobs, and your staff are less opposed to that than you may fear.

The most useful evidence here is the Stanford audit Future of Work with AI Agents, by Yijia Shao, Humishka Zope, Yucheng Jiang, Jiaxin Pei, David Nguyen, Erik Brynjolfsson and Diyi Yang, which surveyed 1,500 workers across 104 occupations and 844 tasks and was last revised on February 1, 2026. Workers were positive about agent automation for 46.1 percent of the tasks they were asked about, and the most cited motivation, freeing up time for high value work, was selected in 69.4 percent of cases. Where they resisted, the leading reason was not fear of job replacement at 23.0 percent. It was lack of trust in the system's accuracy, capability or reliability, at 45.0 percent. And on the researchers' Human Agency Scale, equal partnership rather than full automation was the dominant worker-desired level in 47 of the 104 occupations.

Read that as a leader and the staff conversation changes shape. The question your team is really asking is not "will this replace me." It is "will this be trusted with something I will be blamed for." Trust is a leadership answer, not a training answer, and it is built the same way AI confidence is built: permission, practice and protection. It is also why the response when an agent's output does go out wrong belongs to a separate conversation about the person involved, not to the agent's settings.

There is a second consequence, and it is the one I would put in front of a governing board. When software does the doing, the scarce skill becomes noticing, which is not the skill most of your people were hired or trained for, and it belongs to the same family as the other skills that get scarcer as AI improves. Nobody gets better at catching a subtle error by sitting through a launch webinar. They get better by being given a small number of things to check and being told their catch rate matters.

What should you do this term?

Start by finding out what is already running, then give every piece of it a placement and a name.

1. List the agents already running. Not the ones you chose. The ones that arrived switched on in tools you already use.
2. Run the Loop Test on each one, out loud, in a leadership meeting. Write the three answers down.
3. Fill in the Unattended Work Check for the two that matter most, and accept that line four is where you will stall.
4. Change what reviewers are asked to look at. Exceptions and flagged uncertainty, never the full output. Then try the five design choices above on the one agent that matters most.
5. Set a date to look again. Agents change under you when the vendor ships an update, so a decision made in September is a decision about September.

One agent with a placement and a name is a decision. Forty of them, arriving from the edges while nobody is counting, is a transfer problem rather than a technology one, and it is a different piece of work.

None of this requires you to know how the technology works. It requires you to know what your organization would notice, and how fast. Outsource the doing. Do not outsource the noticing.

If your leadership team is deciding which work may now run without a person watching it, that is the decision I work through with senior teams in AI strategy sessions. I write each week about what agents are quietly taking over in the newsletter.

Sources and further reading

• Margaret Mitchell, Avijit Ghosh and Samir Passi, AI Agents Push Humans Out of the Loop, Hugging Face and Data & Society, arXiv preprint (position paper), August 24, 2026, revised September 6, 2026.
• Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala and Arvind Narayanan, Towards a Science of AI Agent Reliability, Princeton University, arXiv preprint, February 18, 2026, revised June 2, 2026.
• Yijia Shao, Humishka Zope, Yucheng Jiang, Jiaxin Pei, David Nguyen, Erik Brynjolfsson and Diyi Yang, Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforce, Stanford University, arXiv preprint, revised February 1, 2026.
• Microsoft, 2026 Work Trend Index Annual Report: Agents, Human Agency, and the Opportunity for Every Organization, May 5, 2026.
• Julia Gilban-Cohen, Bridges 2026: How Schools Are Putting AI Agents to Work, Government Technology, July 22, 2026.

Dan Fitzpatrick is the founder of The AI Educator, a Forbes contributor, and a keynote speaker and advisor to leaders on AI strategy, readiness and organizational change.