Human in the Loop Is a Feature, Not a Failure
Why automating tasks isn’t enough: learn how healthcare AI agents create real ROI by improving end-to-end workflows, handoffs, and completed outcomes.
When an AI agent hands a case to a person, it’s tempting to count it as a miss. We count it as the system working as designed. What matters is whether the person gets the case already prepared.
This post covers how we think about that and how it works in xMagic: when an agent should stop, what the person receives, how an agent earns more autonomy, and how well-designed handoffs can make future escalations less likely.
In brief
Judge an agent by how much human work is left when a person gets involved, not only by how rarely they do.
Three things should send work to a person: the agent isn’t sure, the next step is too risky to take without approval, or something trips a rule the organization wrote.
Autonomy is earned one workflow at a time, on evidence.
Why automation rate is the wrong score
Automation rate, the percent of cases handled with no human, is the usual headline number. It sounds clean. It also hides the thing that decides whether a deployment is worth it.
Picture two hypothetical workflows, both 70% automated. In the first, the other 30% lands on staff as a dead end: someone calls the patient back and starts from scratch. In the second, the same 30% reaches staff with the history, the extracted details and the open question already written up. The first leaves most of the work to staff. The second has already done most of it.
So the number worth tracking isn’t how rarely a human gets involved. It’s how much human work is left when one does. We go deeper on this in The ROI of Healthcare Agents [7].
Public healthcare examples show why the distinction matters. Surescripts reports an 18-second median approval for automated prior authorizations, but only when all criteria are met, so that figure describes the clean case, not the rest [8]. Microsoft’s healthcare agent orchestrator has specialized agents prepare tumor board material inside Teams, where the board of oncologists still makes the decision [9][10]. These are vendor-reported results. In each, the headline number covers a specific slice of the work, and where a decision carries weight, a person is part of the design.
Three reasons an agent should stop and ask
Three things should send work to a person.
Low confidence. The agent isn’t sure of its answer or of the data it found, for example a patient dispute or an ambiguous clinical detail.
A sensitive action. The next step carries financial, clinical or compliance risk however sure the agent is, for example approving a refund or modifying production data.
A guardrail hit. An input or output trips a rule the organization wrote, for example an output that includes medical advice or personal data it shouldn’t.
The second one matters most. A confident agent can still be wrong about something expensive, so some actions wait for approval no matter what. In xMagic the agent pauses before those steps and asks the right person [4][6]. Guardrails are written in plain language to match how an organization works, and every time one fires it’s flagged for an admin to review [5].
What the person actually gets
A handoff is only as good as what arrives with it. A blind transfer makes staff start from zero and makes the patient repeat themselves. That’s the version people hate, and it’s why handoffs get a bad name.
Here’s what a good one looks like in a prior authorization. The agent pulls the coverage details and the payer’s requirements, then finds the clinical notes are incomplete [1]. It first tries to request the missing documentation. If it can’t close the gap, it doesn’t stall or guess. It puts a worklist item in front of the right person [2][3], containing:
what the request is and who it’s for
what it already pulled together: coverage details, payer requirements, the draft packet
exactly what’s missing and why it stopped
what it recommends doing next
Staff fix the one open item and move on. When the case involves a patient, the loop closes too: once staff resolve it, the agent can call the patient back, so nobody is left wondering what happened to their request [2]. Conversations, actions and escalations are logged, so there’s an audit trail of what happened [1]. The finished packet is still submitted manually, outside xMagic [6].
Autonomy is earned, one workflow at a time
A new hire doesn’t get full authority on day one, and an agent shouldn’t either. We treat autonomy as something each workflow earns in stages, and the institution decides where an agent is deployed, what it can and can’t do, and how fast its scope grows.
An agent starts by connecting to your systems and observing, then shadows staff, then assists under human oversight, and grows into more autonomy as trust, performance and coverage expand [1].
What moves a workflow up a stage should be evidence, not optimism. These are the criteria we’d suggest:
The share of its outputs accepted without a correction is climbing.
The escalation rate is falling, and the cases that still escalate are the ones that should.
It passes the organization’s own test scenarios over repeated runs, not just once.
No guardrail keeps tripping on it.
Every human touch teaches the agent
This is the part that turns oversight from a cost into an asset. Each time a person corrects, approves or takes over a case, that’s a signal, and it goes somewhere.
Corrections can become training examples. Staff corrections can become data for fine-tuning, prepared inside the customer’s own cloud. Repeated edits to how an agent organizes supporting evidence, for example, show the format staff need. It’s optional, and the customer decides whether to turn fine-tuning on and which feedback is appropriate to use [6]. Admins also choose who can give the agent direct feedback, so it learns from your best performers, not from every person who happens to use it. More in [Why Enterprise AI Agents Belong in Your Own Cloud].
Failures become regression tests. A bad interaction in production can be turned into a permanent test case, so the same mistake can’t quietly return in a later release [5].
Approved feedback adapts the agent. It learns from explicit signals, like thumbs up or down with a comment and direct corrections, and from implicit ones, like escalations to a human, sessions ended early and drop-off. Once feedback is approved, the agent adapts within the company’s guidelines, without anyone rewriting prompts by hand [5].
That’s a closed loop: action, outcome, correction, better next time. It’s why we expect the escalation rate to fall as a deployment matures, and why a team that never looks at its handoffs is leaving its best training data on the table. How we test before and after go-live is in How We Evaluate Agent Reliability.
What happens when the loop is badly designed
Four cases from outside our own product show the same lesson from different angles: a legal ruling, a company that reversed course, and two studies of clinicians working alongside AI. Between them they cover what happens when no one is checking, when someone is “checking” but isn’t really, and when the design works.
Air Canada: the organization owns what its agent says
A customer asked Air Canada’s website chatbot about bereavement fares after his grandmother died. The chatbot said he could buy a full-fare ticket and apply for the reduced rate within 90 days. The airline’s own bereavement page said the opposite, and Air Canada refused the refund. It argued, in effect, that the chatbot was a separate legal entity responsible for its own actions [11].
In February 2024 the British Columbia Civil Resolution Tribunal rejected that argument, calling it remarkable. It found negligent misrepresentation and noted that Air Canada never explained why its webpage should be trusted over its chatbot. The total award was CA$812.02, of which CA$650.88 was damages and the rest interest and tribunal fees [11].
The sum was small. The lesson isn’t: a confident answer isn’t a correct one, and the customer can’t tell the difference. Statements and actions that touch money, policy or clinical matters need a checkpoint that doesn’t depend on the agent’s own confidence.
Klarna: the wrong score, and a human option that came back
In February 2024 Klarna reported that its AI assistant had handled two-thirds of customer chats in its first month, about 2.3 million conversations, work it equated to 700 full-time agents [12]. In May 2025 its CEO told Bloomberg that cost had been too dominant a factor in how the service was organized, and that the result was lower quality [13]. Klarna began rebuilding a human support team in a remote, “Uber-type” setup, with the CEO saying customers should always be able to reach a person. The company described this as a dual-track approach, not a step back from automation [14].
These are the company’s own claims, but the pattern is the useful part. “Handled by AI” and “resolved well” are different numbers. It’s the same point as the 70% automated example at the top of this post, played out at scale.
A mammography study: a human in the loop can still be led astray
In a 2023 experiment published in Radiology, 27 radiologists read 50 mammograms with the help of a purported AI system. In the second set of 40 cases, the suggested category was deliberately wrong for 12. Accuracy dropped sharply when the suggestion was wrong, at every experience level. Inexperienced readers got 79.7% right when the AI was right and 19.8% when it was wrong. Very experienced readers went from 82.3% to 45.5% [15][16].
Two cautions. This was an experiment with planted errors, not a deployed tool. And the researchers don’t conclude that people should be removed. They and an accompanying editorial point to safeguards such as showing the system’s confidence, teaching users how it reasons, and keeping them accountable for their own decisions [16]. The lesson: a review step isn’t oversight unless reviewers actually exercise judgment.
The MASAI trial: the loop works when it is designed in
The MASAI trial, run within Sweden’s national screening program, randomized 105,934 women to AI-supported mammography screening or standard double reading. Radiologists still made the final decisions, with the AI’s risk scores and lesion marks in front of them. AI-supported screening found cancers at 6.4 per 1,000 participants versus 5.0 in the control group, a 29% increase, with a 44% reduction in screen-reading workload and no significant increase in false positives. The authors describe showing radiologists the AI’s output at the moment of reading as deliberate [17].
Two caveats. This is an analysis of secondary outcomes, and the primary endpoint, the interval cancer rate, was still to come at publication. And the workload cut came from using the AI to triage exams to single or double reading, in imaging screening rather than administrative work. The lesson transfers anyway: automation took the high-volume routine reading, and people kept the judgment, with better information in front of them.
Reading the cases together
The last two cases can look like they point in opposite directions. They don’t have to. One was built to test what happens when the suggestion is wrong. The other was built around how the AI’s output reaches the reader. They aren’t directly comparable, but together they suggest that outcomes depend less on whether a person is in the loop than on how that step is designed: what the person sees, how much lands on them, and what they’re accountable for.
The risk nobody likes to say out loud
Putting a person in the loop doesn’t guarantee a person is checking. A reviewer who sees approval after approval will start clicking through, and oversight that exists only on paper is worse than none, because everyone assumes it’s working. A few design habits push against it:
Ask less, ask better. If the agent escalates everything, reviewers tune out. Save human review for cases that need judgment and let the high-volume, low-risk steps run fully on their own.
Hand over context. A reviewer who can see the evidence can actually disagree with it.
Audit what the agent finished alone. Reviewing a sample of completed work catches drift that escalations never show.
Watch whether reviewers ever change anything. A 100% approval rate is either a very good agent or a rubber stamp. You want to know which.
How to measure it
Five numbers tell you whether the human side of a deployment is healthy.
Straight-through completion: how often the agent finishes with no human involved.
Assisted completion: how much human work is left when it doesn’t.
Accepted without correction: whether the agent’s output holds up under review.
Escalation rate over time: whether the loop is working. It should fall.
Time to human resolution: whether handed-off cases are actually getting picked up.
Zero escalations isn’t the target. A falling escalation rate with steady quality is.
Where to start
You don’t need a big program to apply this. For one workflow:
Write down which of the three triggers apply, and where the line sits for each.
Decide what every handoff must contain before it reaches a person: the request, what the agent already gathered, what’s missing, and what it recommends.
Start with the agent observing and assisting, and agree on the evidence that would move it up a stage before you begin.
Track the five numbers, and sample the work the agent finished alone.
The questions to ask any AI vendor
Skip “how good are the answers?” and ask this instead: what does it do on its own, and what happens the moment it can’t? Who sees that moment, what do they receive, and does the agent get better because of it? The cases above suggest four follow-ups:
What can the agent say or do without a check, and who decided that line? (Air Canada)
How do you measure quality of outcomes, not just volume handled? (Klarna)
How would you know if reviewers had stopped really reviewing? (the mammography study)
Where does routine volume end and judgment begin, and can that line move with evidence? (MASAI)
That’s the design xMagic is built around: routine work runs end to end, everything else reaches a person with the work already prepared, and the loop makes the next case easier. For the bigger picture on agents that act rather than answer, read From AI Responses to AI Execution.
Sources
https://stochastic.ai/blog/autonomous-healthcare-agents-for-administrative-work
https://stochastic.ai/blog/how-stochastic-is-redefining-new-patient-scheduling-with-ai
https://stochastic.ai/blog/beyond-voice-bots-how-enterprise-ai-agents-turn-calls-into-actions
https://stochastic.ai/blog/mcp-skills-and-the-missing-layer-for-production-ai-agents
https://stochastic.ai/blog/how-we-evaluate-agent-reliability-before-and-after-production
https://stochastic.ai/blog/why-enterprise-ai-agents-belong-in-your-own-cloud
https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do
https://www.pymnts.com/bnpl/2025/klarna-focuses-on-human-powered-customer-service
https://www.rsna.org/news/2023/may/ai-bias-may-impair-accuracy