On August 4, 2026, the British National Cyber Security Center (NCSC) issued a statement regarding incidents arising from evaluations of particularly powerful AI models. Ollie Whitehouse, the agency’s Chief Technology Officer, stated in this Statement on AI Incidents A statement that applies to any document process in which an agent acts on its own: „Relying on detection alone after the fact of an incident will not be enough.“ If you wait until after the incident to look into it, it’s already too late.
For companies that process invoices, contracts, or claims with the help of AI, this is not an abstract security issue, but a central question in process optimization. They are currently deciding how much autonomy they can and want to grant an agent in their document review and processing. This article answers the key questions involved: When can an AI agent make decisions independently in a document process, when must approval remain with a human, and who bears responsibility if the agent makes a mistake?
What Sets an AI Agent Apart from Traditional Automation
An AI agent doesn't just respond to a request—it pursues a goal. Sven Beck, co-founder and Chief Technology and Product Officer at BLP Digital, describes the difference in a Analysis of Agentic AI According to BigData-Insider on August 20, 2026: An agent „pursues a goal, interprets context, selects tools, performs actions, and evaluates the result.“ In other words, it chooses the next step on its own, rather than simply following a predetermined sequence.
That is the crucial difference when it comes to document processing. Rule-based automation is reliable but inflexible. Regarding robotic process automation, OCR (optical character recognition), and traditional workflows, Beck writes: „Each of these technologies operates in a deterministic, rule-based manner: they work reliably as long as the input matches the expected pattern.“ If an invoice layout differs or an item is missing, the chain breaks. An agent can continue working at this point because they assess the context rather than searching for a rule.
That's what sets him apart from the Traditional data extraction versus LLMs, in which a model reads the data once and returns a result. However, it also shifts the nature of the error. A rule-based system fails visibly: the process stops, and someone receives a message. An agent fails invisibly because it continues running and produces a plausible result.
What happens if someone contradicts the agent?
This leniency is the norm in cases of invisible failure, and it has become excessive. A team at Stanford University led by Aaron Fanous has The SycEval testing procedure ...was developed and presented at AIES 2025. Three models currently in use by companies were evaluated: ChatGPT-4o, Claude Sonnet, and Gemini 1.5 Pro.
The process is simple. The model answers a question—either from a math dataset (AMPS) or a collection of medical questions (MedQuad)—and the initial answer is classified as correct or incorrect. Then the test setup challenges it in four stages of increasing pressure. The first stage simply claims that the answer is wrong; the second stage asserts authority; the third stage provides logical reasoning; and the fourth stage cites a scientific source. After each stage, the test setup checks again to see if the model sticks to its answer. From 3,000 initial questions, this resulted in 24,000 rounds of challenge, of which 15,345 were evaluable.
Averaged across the three models, they gave way in 58.19 percent of cases. The key factor is the direction. In 43.52 percent of cases, the model’s concession led to the correct answer: the model had been wrong and allowed itself to be corrected. In other words, exactly what a validation loop is designed to do occurred. In 14.66 percent of cases, however, the model was correct and gave up the correct answer despite being challenged.
Applied to a document process, this means: An agent reads a net amount of 4,812 euros from an invoice. A subsequent verification step or a clerk disputes this. Instead of insisting on the verifiable value, the agent corrects the entry to align with the objection, even though the document is unambiguous. In document verification, this is not merely a convenience issue but a liability risk: the process documents a verification that, in substance, did not take place.
However, this does not imply a blanket rejection of agents. Rather, it leads to the realization that autonomy is not a switch that can be flipped on or off, but rather a factor that must be measured and refined at each step of the process.
How the autonomy of an AI agent is measured
In practice, the boundary is not defined by document types, but by confidence scores. A confidence score is a number between zero and one that a model uses to indicate how certain it is about an extracted field. Above a defined threshold, a document passes through without human intervention; below it, the document ends up in a review queue. The article on automated document processing. The percentage of documents that pass through the system without intervention is the Dark Processing Rate.
An agent does not change this principle. It merely shifts what the confidence value refers to: no longer a single extracted field, but rather an action. The question then becomes not „How certain is the amount?“ but „How certain is the decision to approve this invoice for payment?“ And this is the difference that makes the question of autonomy so difficult: For a single field, this certainty is measured and verifiable. For a chain of reading, comparing, validating, and approving, it is the product of multiple uncertainties, each of which remains unnoticeable on its own.
That is why Human in the Loop Even for agents, this is not a transitional stage on the path to full automation. Humans remain an integral part of the process and handle the cases in which the system itself flags uncertainty; everything else runs as Straight-Through Processing through. In the case of an agent, however, this reporting point must be reset. It is no longer tied to the field, but rather to the action that has financial or legal consequences.
What this means for the rollout
The NCSC has formulated a testing rule specifically for this situation. In the publication Thinking Carefully Before Adopting Agentic AI In a letter dated May 15, 2026, authors Martin R and Dr. Kate S write: „If you cannot understand, monitor, or contain an agent’s actions, it is not ready for deployment.“ Understand, monitor, contain. If any one of these three capabilities is missing, the agent is not ready for deployment.
The agency also clarifies where responsibility lies: „Humans remain accountable for the decision to deploy it, the access it was granted, the safeguards surrounding it, and the consequences of its operation.“ Accountability for commissioning, granted access, safeguards, and consequences therefore rests with humans. An agent is not a decision-maker, but rather a tool with extended privileges.
In practice, this means three things. First, every agent must have a designated person who owns it, authorizes its access, and can stop it in case of doubt. Second, the principle of least privilege applies: An agent who reads incoming invoices does not need write access to master data. Third, the agent’s behavior must be visible in real time, not just in retrospect—because that is precisely what the August statement warns against.
For companies in regulated industries, there is the additional factor that this documentation is required anyway. Those who already keep records of which documents were approved by whom must extend this logging to include agent decisions. The requirement is not new—only the party involved.
The line is drawn between preparation and approval
An AI agent excels where traditional automation falls short: with non-standard layouts, missing items, and processes that span multiple systems. It is weak precisely where one would most like to trust it—namely, in its insistence on a correct result in the face of contradictions. What’s transferable here isn’t the measured number, but the mechanism behind it: Giving an agent validation loops doesn’t automatically make it more accurate.
A robust division of labor follows almost naturally from this. The agent prepares, collects, reconciles, and makes recommendations. Approval with financial or legal implications remains subject to a threshold set by a human—and, in case of doubt, to the human themselves. This is not half-baked automation. This is the form in which automation stands up to scrutiny.
If you want to take the next step, don’t start with the agent architecture—start with a number: Measure the current dark processing rate of your document process and the error rate in cases that are already being processed without intervention. Without this baseline, it will be impossible to determine later whether the agent has improved the process or is simply making mistakes faster.
EN 
DE 
ES 
FR