Making AI Implementation a Success

Most AI projects don't fail because of technical issues. They fail at the second use case—that is, when the first one has worked and no one knows how to apply it to the next process.

Anyone who’s been through it knows the scenario: Invoice processing is up and running, the team is satisfied, and the numbers add up. Six months later, the same technology is supposed to read delivery slips. And the project starts from scratch again because the model, thresholds, and validation process were tailored to exactly one type of document.

The insurance group AXA has drawn a conclusion from this. The group is ending its decentralized AI experiments and, together with Publicis Sapient, has established a vendor-neutral „Global AI Hub,“ as Tristan Fincken in *Computerwoche* reported on September 11, 2026. The hub’s three areas of application are telling: customer email processing, auto claims processing, and company-wide knowledge management. All three rely on documents, and all three require the same foundation.

This post answers the underlying question: How do you implement AI in a way that allows it to grow beyond its initial use case?

Why AI Implementation So Often Ends with a Pilot Project

The figures on this are new and unusually clear. The IT service provider Kyndryl has, for the People Readiness Report 2026 1,100 senior technical and technology leaders in eight countries were surveyed, and the results were published on June 25, 2026. The fact that an IT service provider is assessing the need for consulting is part of the context—yet the individual figures are still revealing.

Two findings contradict each other. 57 percent of respondents say that AI is integrated into their core processes or has been widely rolled out. At the same time, Only 32 percent have achieved at least one of their two most important AI goals.

So AI has long since made its way into business operations—it just isn’t delivering what it was supposed to. This is reflected in the self-assessment: 79 percent agree with the statement that the pace of AI will outpace their workforce, their governance, and their business model.

The minority is more interesting than the majority. Kyndryl identifies a group of 9 percent that the report calls „Pacesetters.“ They have done three things: redefined roles around AI, effectively implemented change management, and prepared their workforce to work with AI. According to the report, this group is 1.5 times more likely to achieve AI-driven revenue growth and 1.6 times more likely to achieve better innovation results.

Mark Paulek, Chief Human Resources Officer at Kyndryl, summarizes the findings as follows: „The leaders pulling ahead are aligning skills, roles, and decision-making with how work is actually changing.“ In other words, those who align skills, roles, and decision-making processes with how work is actually changing are pulling ahead.

The study measures organization, not technology. That is precisely what makes it valuable: The difference between the 9 percent and the rest lies not in the model, but in their preparedness for changes in the way work is done.

What challenges arise when scaling AI workflows?

In document processing, the breakpoints can be precisely identified. They almost always appear in the same order, and there are four of them:

• The filtered dataset: The model was trained using 200 invoices of a single layout. This dataset does not exist for delivery notes, and no one has documented how the training data was generated.
• The implicit threshold: In the pilot project, someone set a confidence threshold above which a document is approved without review. It isn't documented anywhere; it's just in the project manager's head.
• The Evolving Audit Process: The department has become accustomed to an interface designed for a specific document type. Each additional type comes with its own form.
• The lack of a benchmark: The first use case is considered a success, but there is no metric that can be applied to the second one.

The fourth breakpoint is the most expensive because it hides the other three. Without a common metric, it is impossible to tell whether the new process performs better than the old one—and therefore also impossible to tell whether the transition was worth it.

What Needs to Evolve Alongside AI Implementation

There is a common misconception here. Scalability sounds like infrastructure, servers, and throughput. In document processing, however, it is primarily a matter of measurability. Three key metrics must be established from the very first week, because only they allow for a comparison between use cases:

• Average processing time per transaction: How long does it take for a document to be processed from the time it arrives until it is posted? The board of directors understands this figure; it can be compared to the manual process, and it applies equally to every type of document.
• Dark Processing Rate: The percentage of cases that are resolved without intervention. It answers the question of whether the second use case has caught up with the first.
• Error rate in the cases processed in the dark: How many of the automatically processed transactions were incorrect? Without this cross-check, the blind processing rate is worthless.

The last point is the one most often overlooked, and it is the most dangerous. A dark-processing rate can be inflated at any time by lowering the inspection thresholds—the rate goes up, but the defects simply move further down the line in the process. Only when considered alongside the defect rate does it become a reliable indicator.

Confidence and threshold values belong one level down. They are not the metric itself, but rather the adjustment parameter. A confidence value for each field indicates how confident the model is in a single extracted value. The threshold value determines the point at which this value is accepted without human review. You adjust these settings when the error rate is too high or the dark processing rate is too low.

Two things need to be documented; otherwise, they’ll remain only in the project manager’s mind. The threshold varies by field—an incorrect amount has different consequences than an incorrect customer number. And it’s important to note which value applied at what point in time. Without this note, it will be impossible to reconstruct later why a ratio increased.

Throughout this process, the individual remains actively involved. Human in the Loop is not a transitional stage on the path to full automation, but rather the mechanism that calibrates the thresholds in the first place. Anyone who accurately records the test cases has, in the process, generated the training data for the next document type. This is precisely what distinguishes a growing solution from a finished one: corrections from the business department are fed back into the model instead of getting lost in a ticket system.

How can you tell if a platform can scale to handle large volumes of documents?

The decision is rarely based on throughput, since almost every platform processes enough pages per hour. The differences lie elsewhere, and three key factors distinguish a platform from a point solution:

• Reusability of the models: Can a trained model be used as a starting point for a related document type, or does every project start with an empty dataset?
• Uniform test interface: Does the department use the same form for all document types, or does a new application get created for each use case?
• Comprehensive logging: Is it clear which values were automatically accepted and which were manually corrected? In regulated industries, this is mandatory anyway, and it forms the data foundation for any improvement.

These criteria apply regardless of the provider. They are the reason why AXA is building a vendor-neutral hub instead of yet another standalone solution—a central layer on which email processing, claims processing, and knowledge management all use the same building blocks.

How can you introduce AI gradually without committing to a specific approach?

The pragmatic approach starts small, but not haphazardly. The first use case should have three characteristics:

• Enough volume: The measurement is only meaningful once a certain quantity is reached.
• A clear distinction between right and wrong: Only then can errors be detected at all.
• A related process: The transition to the next document type must be realistic.

Incoming invoices usually meet these requirements—they arrive in large numbers, their fields are clearly defined, and the delivery slip is physically right next to them.

After that, the order of steps matters. First, measure what the manual process costs today—processing time per transaction and the error rate in cases that no one checks. Then automate the process, and then collect those same two figures again. Without this baseline, it will not be possible later to demonstrate whether the Automated document processing whether it has improved the process or is simply wrong more quickly—and without this evidence, the second use case won't get any funding.

Conclusion

Implementing AI doesn't just mean putting a model into operation. It means defining the metrics against which the next use case must prove itself.

Kyndryl’s figures reveal where the problem lies: AI has long been integrated into core processes at 57 percent of the companies surveyed, but only 32 percent are achieving their most important goals. The 9 percent who are making progress don’t stand out because of better technology, but because they’ve also adapted roles and workflows.

What can be applied here is not the number, but the mechanism. If you track processing time per transaction, the dark processing rate, and the error rate side by side from the start, you won’t have to reinvent the benchmark for the second document type. Confidence levels and thresholds are the adjustment factors here, not the result.

If you want to take the next step, don’t start by choosing a model. First, measure how long it takes to process a document today and how often something is overlooked in the process. These two figures will later determine whether a pilot project will become a platform.