Building an AI agent has never been easier than it is today. Frontier models can reason through multi-step tasks, open-source frameworks give you an agent loop in a few dozen lines of code, and coding assistants will happily write the integrations for you, so a small team can go from an idea to a working prototype in a couple of weeks rather than months. That speed is real and it is good news, but it has also changed how organizations approach one of the oldest decisions in software: whether to build a capability in-house or buy it from someone who already has.

It usually plays out the same way: a team picks a painful workflow, say preparing prior authorization packets, and wires a model to a handful of tools: something to read faxed clinical notes, a browser to log into payer portals, a script that assembles the PDFs and a shared folder where the finished packet lands. Two weeks later the agent is producing packets that look right, the demo goes well, and someone asks the obvious question: if we got this far in two weeks, why would we pay anyone for this?

It is a fair question, but it tends to get answered at the worst possible moment, right after a successful demo, when the visible part of the work looks almost finished and the invisible part, everything that has to exist before that agent can safely touch real users, real data and real systems, has not even started. Over the past couple of years we have helped many organizations move agents from prototype to production, and we keep seeing the same pattern: the decision to build is made on the cost of the agent, but the real cost sits in the stack around it, in the isolation, guardrails, permissions, approvals, evaluations, audit trails, integrations and compliance work, plus the never-ending effort of keeping all of it current while models and systems keep changing underneath. If the demo is the down payment, the stack is the mortgage, and it keeps arriving every month for as long as the agent runs.

The demo is the easy part

The demo is not lying, but it is incomplete in a way that is very hard to see from the inside. In our experience, roughly 80 percent of an agent comes together remarkably fast: the model understands the task, the tools are wired up and the common cases work well enough to impress a room. The remaining 20 percent is where the product actually lives, in the edge cases, the guardrails, the security controls, the session isolation and the reliability that make an agent safe to put in front of real users, and that last stretch routinely takes several times longer than everything that came before it. Demos are never refined to that level because they do not need to be, which is exactly why the size of the effort stays invisible until a team commits to turning the demo into a real product. Estimating the whole project from the speed of the first 80 percent is one of the most common mistakes we see teams make, and it is usually the mistake behind the decision to build.

A demo lives a forgiving life, with one user, usually the person who built it, running on clean inputs along the happy path. It borrows its builder's credentials, nobody is trying to break it, and nobody will ever ask it to explain what it did. Production takes away every one of those comforts at once. The prior-auth agent now serves 40 staff members across three clinics, and faxes arrive sideways and half-legible, with a page from another patient's file mixed in. Users correct themselves mid-request, ask for two things in the same sentence or don't know which plan the patient is on. And the moment the agent touches real patient data, new people join the conversation and start asking much harder questions:

• When the agent opens a browser or a terminal, can one user's session see another user's files, cookies or portal credentials?

• Who approves before the agent submits, deletes or overwrites something that cannot easily be undone?

• What happens on the day the model we built everything on is retired?

• When an auditor asks what the agent did with a specific record last March, how long does it take to answer?

• Who is on call when it breaks at two in the morning?

The room usually goes quiet, not because the team lacks skill but because none of these questions are about intelligence; they are about operations, and if you build, every one of them becomes something your team owns for as long as the agent runs. We covered what a production platform needs in an earlier post, so here we look at the same ground from a different angle: what each layer costs to build, and what it costs to keep.

The iceberg: what you actually sign up for

1. Isolated execution

The most capable agents are also the riskiest to run, because an agent with a terminal, a browser and a file system executes arbitrary code and processes arbitrary web content right next to sensitive data. In the demo all of that ran on one laptop, but in production every session needs its own sandbox: separate file systems, browser profiles and cookies that never cross between users, resource limits so that one runaway task cannot starve the others, network egress rules so that a compromised session cannot send data out, credential vaulting for the portals the agent logs into and reliable cleanup when the session ends, all of it starting fast enough that users never notice. This is infrastructure work in the most literal sense, made of container orchestration, security hardening, capacity planning and latency tuning, and while none of it makes the agent any smarter, all of it has to be right before a single real user touches the system.

In xMagic, when an agent runs code, touches files or executes shell commands, that execution happens inside a dedicated container for each chat session, so one user never has access to another user's data. We treat this as the minimum standard for any agent that handles sensitive business data, not as an optional extra.

2. Guardrails and prompt injection

An agent that reads documents, web pages and emails can also be instructed by them. Indirect prompt injection, where instructions are hidden inside the content an agent processes, is no longer a theoretical concern. EchoLeak showed that a single crafted email could make Microsoft 365 Copilot leak data without the user clicking anything [1], and ForcedLeak showed the same pattern against Salesforce Agentforce [2]. OWASP now publishes a dedicated Top 10 for Agentic Applications, led by agent goal hijacking and tool misuse [3], and prompt injection has topped its LLM list for the third year running [4].

Defending against this means building input filters for injection attempts, jailbreaks, off-topic requests and PII. It means output checks for leaked internal details, unauthorized advice and policy violations, and, in healthcare, PHI detection and masking at every step. It also means keeping those defenses current indefinitely, because attackers adapt and every new tool you connect opens another path into the system.

With xMagic, every message passes through configurable input guardrails before it reaches the agent, and every response passes through output guardrails before anyone sees it. Rules are written in natural language to match your own policies, and every time a guardrail fires, the event is flagged for administrators to review.

3. Identity and permissions

Controlling who can log in is the easy part; the hard part is controlling what the agent is allowed to do and on whose behalf. A prototype usually runs on one powerful service account, whereas in production the agent should act with the permissions of the person it is working for and nothing more, and administrators need fine-grained control over who can create agents, share them, view their configurations or push changes to a live one. OWASP calls this failure mode "excessive agency," and it rose from eighth to third place in the 2026 rankings precisely because agents now act on systems rather than just answering questions [4].

xMagic supports Single Sign-On and granular, role-based privileges that let you decide exactly who can create agents, share them outside the workspace, view private configurations, or update and delete them. You get specific privileges instead of a single all-or-nothing admin switch.

4. Human approvals and handoffs

Some actions should never happen without a person in the loop, such as submitting a packet to a payer, deleting a file or overwriting a shared document. Building approvals sounds like adding a button, but in practice it requires durable state that lets the agent pause and resume, sometimes hours later, notifications that reach the right reviewer, timeouts and escalation paths, and a handoff that carries the full context when the agent cannot finish on its own, so that staff never have to start from zero.

In xMagic, human-in-the-loop approval is built in for critical actions, such as deleting files or overwriting shared documents, and can be turned on or off as needed. When an agent cannot resolve a case on its own, it creates a worklist item with the full context assembled for whoever picks it up.

5. Evaluation and safe change

Agents are almost never right on the first attempt, so you will change their prompts, tools, knowledge, subagents and models constantly, and every one of those changes can quietly break something that used to work. Changing them safely requires an evaluation harness with component tests, end-to-end scenarios and simulated users running against the real runtime, consistency measured across repeated trials rather than trusted from a single lucky run, a process that turns every production failure into a permanent regression test, and versioning of the entire agent configuration so that a bad change can be rolled back in seconds. Someone also has to write those benchmarks in the first place and keep them representative as the agent gains new tools and users find new ways to ask for things, which is slow, unglamorous work that rarely survives a busy quarter. In LangChain's latest State of Agent Engineering survey, only 52 percent of organizations reported running offline evaluations and just 37 percent evaluated live production traffic [5], which means many teams still find out about regressions from their users.

With xMagic, every agent and subagent configuration is versioned automatically as an immutable release, including prompts, tools, knowledge and memory settings. xMagic can also generate benchmarks automatically from your agent's configuration and its existing conversations, so you can track the agent's accuracy every time you change it without writing each test case by hand. You can try a candidate version in the Playground and run those benchmarks, or your own evaluation sets, against it with simulated users inside the production harness before you deploy it, roll back instantly if anything goes wrong, and turn any failed production conversation into a regression test in a single step.

6. Observability and audit

When something goes wrong, "the model did something weird" is not an answer your compliance team will accept. You need a complete, searchable trace of every message, tool call, decision and action, kept under the right retention policy. You also need analysis that surfaces problems before anyone complains: loops, misrouted requests, repeated fallbacks and frustrated users. Building that means tracing infrastructure, storage and retention rules, conversation analytics and audit exports, and all of them have to keep working as the agent and its tools evolve.

In xMagic, every text and phone conversation is recorded in Threads, with every tool call and action logged and reviewable.

7. Integrations

The prior-auth agent needs a fax inbox, an EHR system, a document store and a dozen payer portals. Each has its own authentication, conventions, rate limits and quirks, and each one changes on its own schedule. Standards like MCP and Skills help, but someone still has to own every connector. Many of the systems that matter most in healthcare have no usable API at all, which pushes you toward browser automation and right back to the isolation problem in layer one. As we have written before, integration debt, not model quality, is usually the real bottleneck.

With xMagic, agents connect to existing MCP servers in a few steps, the SDK turns your proprietary systems into MCP servers, and Skills are managed centrally and reused across agents. SharePoint and Google Drive stay in sync as your source of truth, and where a system has no API, agents can work through the browser the way your staff do.

8. Data residency and compliance

Protecting the source records is only part of the job. Prompts, model responses, conversation history, troubleshooting logs and backups all carry the same sensitive content. Each external model provider that receives it adds another set of terms to review on training, retention and human access. SOC 2 and HIPAA are not boxes you tick once; they mean recurring audits of every layer described above.

xMagic solves this with a control plane, operated by Stochastic, that holds only platform metadata, while the compute plane runs in your cloud, so agents, models, files, prompts and responses stay there. Stochastic models can run inside your environment without retaining your content or training on it, or you can bring your own open-weight model, and xMagic is SOC 2 Type II and HIPAA compliant.

Put together, the gap is easy to describe: the demo had one user, friendly inputs, its builder's credentials, nobody to ask for approval, console logs and data sitting on a laptop, while the product needs isolated sandboxes, guardrails that are constantly maintained, delegated permissions, durable approvals, a living evaluation suite, complete audit trails, a connector for every system and a compliant deployment in your own cloud. Every one of those layers is something you will also have to operate, patch, audit and staff long after the demo has been forgotten.

The treadmill: why the cost never reaches zero

Even if the stack were a one-time project the math would be uncomfortable, but almost everything it depends on keeps moving. Model providers retire the versions you built on, often with only a couple of months' notice, and newer models can change how an agent selects tools and formats output overnight, forcing you to re-run every evaluation. The ecosystem shifts underneath you as well, as protocols get revised, popular agent frameworks are deprecated or merged into new ones, and today's sensible technical choice becomes tomorrow's migration project. Meanwhile, the systems your agent connects to change their APIs and schemas, regulations push the industry toward new interfaces, and new injection techniques appear faster than an internal team can track them.

There is also a second treadmill, which is about getting better rather than standing still. Accuracy that looked acceptable at launch stops being acceptable once staff rely on the agent, every new edge case needs a fix and users soon ask for new channels and workflows. When you build, all of that lands on your own roadmap and competes with your core product, whereas when you buy, a provider whose entire team is dedicated to agent accuracy and features does most of that work and the improvements reach your agents with each release. Building your own stack therefore means staffing an internal platform with exactly one customer and no roadmap beyond keeping up.

Doing the math

Running the layers above takes engineers who own infrastructure and sandboxing, someone responsible for security and guardrails, someone responsible for evaluation, an on-call rotation and a steady stream of compliance work that never really ends. Robert Half puts the base salary of AI and machine learning engineers in the US between $134,000 and $193,250 [6], before bonuses, benefits and the premium that engineers with real production experience command. Industry estimates put ongoing maintenance at 15 to 30 percent of the original build cost every year [7], and that is before counting a single model migration.

The outcome data points in the same direction: MIT's NANDA initiative found that AI tools bought from specialized vendors, or built in partnership with them, succeeded roughly two-thirds of the time, while internal builds succeeded only about a third as often [8]. Gartner expects more than 40 percent of agentic AI projects to be cancelled by the end of 2027, driven by rising costs, unclear business value and inadequate risk controls [9].

The largest cost, however, does not appear on any budget line. It is what your best engineers are not doing while they maintain sandboxes and guardrail pipelines: encoding your workflows, your exceptions, your payer rules and your judgment. That work is what makes your agents different from everyone else's, and it is usually the first thing to get squeezed. There is also a quieter risk, because homegrown agent stacks tend to be built by one or two strong engineers who end up as a platform team nobody planned to create, and the stack becomes legacy the day one of them leaves.

Build the intelligence, buy the foundations

In a previous post we argued that "the foundations will be bought or adopted openly; the intelligence must be built. Mistaking one for the other is how companies end up at stage two permanently." That is why the useful question is not "build or buy?" but "which parts?", and the answer starts with separating what makes you different from what does not. Own the parts that make you different: your workflows and Skills, your domain knowledge, your evaluation sets and your definition of what "correct" means, the feedback and corrections your staff produce every day and any models fine-tuned on that feedback. This is your institutional intelligence, and no vendor can sell it to you because its substance is yours; it compounds the longer you invest in it.

Rent the parts that are undifferentiated and constantly moving: sandboxed execution, guardrail infrastructure, identity and access, versioning and deployment, observability and audit, channel infrastructure, and the plumbing that connects you to models.

If you do buy, some requirements should be non-negotiable:

• It runs in your cloud, and your data never trains the vendor's models.

• It is built on open standards like MCP and Skills, so your capabilities stay portable.

• Your configurations and evaluation sets remain yours.

• Every session is isolated, every action is auditable, and sensitive actions can require human approval.

• It holds the certifications your industry requires.

This is the split we designed xMagic around, with Stochastic operating the foundations, including per-session isolated execution, natural-language guardrails, SSO and granular permissions, built-in human approvals, automatic versioning with rollback, evaluation with simulated users and full audit trails in Threads, all of it running in your cloud through the compute plane. That leaves your team free to spend its time where it matters most: describing and refining agents, encoding workflows as Skills, connecting systems through MCP, defining what good looks like and feeding every staff correction back into a better agent. Because improving accuracy and shipping new capabilities is our full-time job rather than a side project for your team, those improvements, from better reasoning and retrieval to new evaluation tools such as automatically generated benchmarks, reach your agents as soon as they ship.

Ready to spend your time on the intelligence instead of the plumbing? See how xMagic gives your team the production foundations to run AI agents securely and reliably in your own cloud.
Get early access

References

1. The Hacker News. "Zero-Click AI Vulnerability Exposes Microsoft 365 Copilot Data Without User Interaction." June 2025. https://thehackernews.com/2025/06/zero-click-ai-vulnerability-exposes.html

2. Noma Security. "ForcedLeak: AI Agent Risks Exposed in Salesforce Agentforce." September 2025. https://noma.security/blog/forcedleak-agent-risks-exposed-in-salesforce-agentforce/

3. OWASP GenAI Security Project. "OWASP Top 10 for Agentic Applications for 2026." December 2025. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/

4. SD Times. "Prompt Injection tops 2026 OWASP GenAI / LLM Top Ten vulnerabilities." 2026. https://sdtimes.com/security/prompt-injection-tops-2026-owasp-genai-llm-top-ten-vulnerabilities/

5. LangChain. "State of Agent Engineering." https://www.langchain.com/state-of-agent-engineering

6. Robert Half. "AI/ML Engineer Salary." 2026 Salary Guide. https://www.roberthalf.com/us/en/job-details/aiml-engineer

7. Riseup Labs. "The True Cost of Implementing AI in Business in 2026." https://riseuplabs.com/cost-of-implementing-ai-in-business

8. Fortune. "MIT report: 95% of generative AI pilots at companies are failing." August 18, 2025. https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/

9. Gartner. "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027." June 25, 2025. https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027