Start / Blog / Methods / How does document extraction with LLMs work properly?

How does document extraction with LLMs work properly?

Summarize with ChatGPT

Everyone is talking about artificial intelligence and large language models (LLMs). But when it comes to actually using these technologies in document processing and. Document extraction it quickly becomes clear that there is a world of difference between marketing promises and practical reality. A developer insight shows how Konfuzio IDP solves a fundamental problem that causes many other systems to fail. A topic that I have been dealing with for a long time. You can find my article on the subject, which after almost two years seems historic, here: Data extraction from PDF documents.

The problem: If „€1.00“ is not the same as „€1.00“

Imagine a typical situation: An invoice lists various items:

Rechte an Patenten ........................ 1,00 €
Markenrechte .............................. 1,00 €

Your task: The AI should allocate both amounts correctly to the respective items. With clear values, as in the following image, this is of course easier.

It sounds simple - but it's not.

Many AI solutions work like this today: The LLM reads the document and says „I found €1.00 for rights to patents“. So far so good. But now comes the problem: the system has to find this value in the original document in order to mark it there or process it further.

And this is where it gets complicated: The document says „€1.00“ twice. Which one is meant? The first one? The second?

This is difficult for a calculation with two identical values. But just imagine:

Suddenly a simple task becomes a game of chance. The AI cannot reliably say: „This specific amount belongs to this specific item.“

This often results in the following problems

  • LLM Hallucination Document extraction
  • LLM invents data from documents
  • AI extracts incorrect values from PDF
  • Document AI makes errors with identical values
  • LLM cannot distinguish between equal amounts
  • Text extraction LLM inaccurate
  • Assign OCR values correctly Problem
  • Automatically assign multiple invoice amounts
  • AI mixes up values in documents
  • Document processing LLM unreliable

The classic solution does not work

Many providers try to solve the problem with tricks:

  • „We simply take the first occurrence of €1.00“ → Leads to errors
  • „We look at the context“ → Does not work reliably with complex documents
  • „We use approximate values“ → It gets messy with long documents

The basic problem remains: Once the AI has extracted a value, it can no longer be clearly assigned to its original position in the document.

Our solution: The AI knows where to look.

The LLM integration of Konfuzio IDP works fundamentally differently: The LLM not only returns WHAT it has found, but also WHERE exactly it has found it.

This is how it works in practice

Step 1: The document is prepared

If a document is created by the OCR text recognition each recognized word is already assigned its position in the document. This looks something like this:

Wort "Rechte" steht bei Position x=50, y=100
Wort "an" steht bei Position x=75, y=100
Wort "Patenten" steht bei Position x=95, y=100
Wort "1,00" steht bei Position x=125, y=100
Wort "€" steht bei Position x=130, y=100
Wort "Markenrechte" steht bei Position x=50, y=125
Wort "1,00" steht bei Position x=125, y=125
Wort "€" steht bei Position x=130, y=125

Step 2: The LLM receives this information

The LLM does not simply see „Rights to patents €1.00“, but also receives the position for each word.

Step 3: The LLM responds WITH position

Instead of just saying „Rights to patents: €1.00“, the LLM gives back:

Feld "Rechte an Patenten": 
- Wert: 1,00 €
- Position: x=125/y=100 und x=130/y=100
Feld "Markenrechte":
- Wert: 1,00 €  
- Position: x=125/y=125 und x=130/y=125

What does this mean in practice?

1. identical values are no longer a problem

Regardless of whether „€1.00“ appears once or a hundred times in the document - the system always knows exactly which value belongs to which field. This is particularly important for:

2. no invented values

A common problem with LLMs is that they sometimes „hallucinate“ - in other words, they invent data that is not actually in the document. This is impossible with Konfuzio's approach. The AI MUST specify a position that is derived from the OCR recognition originates.

Yes, errors are possible - the AI could assign the wrong value to the wrong field. But it cannot invent amounts or data that are not in the original document.

For Regulated industries such as banks or Insurance this is crucial.

3. you can always see where a value comes from

In the Document Validation UI your employees can see exactly: „This extracted value comes from exactly this place in the document.“

This makes quality control easy:

  • Is the value assigned correctly? → One look is enough
  • Is there an error? → You can see immediately where
  • Do you need to correct anything? → The correct position is marked

4. traceability for compliance

In the Audit, in the public sector or with Financial service providers you often have to prove: „Where does this information come from?“

With the IDP coordinate-based approach, you can trace every extracted value back to the exact pixel in the original document. This creates Transparent, comprehensible AI.

The price: more complexity for AI

Of course, this approach is not without its challenges. It is much more difficult for the LLM:

  • It must not only understand WHAT is to be extracted
  • It must also specify the correct coordinates
  • It must process several levels of information simultaneously

As a result, errors can occur with very complex documents - such as incorrectly specified coordinates or mix-ups.

But: Better a system that finds the right value in the right place for 95% of the cases than one that simply spits out a value for 98% - but nobody knows which one, and that with the Automation of critical processes leads to expensive errors.

Why this is important for your company

In times when every provider promises „AI-supported document processing“, it is worth taking a look behind the scenes. Many solutions work well in demos with simple sample documents. But when it comes to real, complex company documents, the differences become apparent.

Konfuzio's approach is a direct response to the challenges that arise in the practical use of LLMs:

  • Don't just let ChatGPT loose on documents
  • But work in a structured, coordinate-based and comprehensible way
  • Applied AI instead of AI washing

This is particularly relevant if you:

  • Process many documents with recurring values
  • High demands on Dark processing (i.e. documents should pass through without manual checking)
  • must ensure compliance and traceability
  • Your Truly automate processes want to - not only digitize

Conclusion: Engineering skill instead of marketing promises

Our coordinate-based approach shows: Good AI solutions are not created by simply taking the latest LLM and hoping it works. They are created through thoughtful architectural decisions that solve real-world problems.

Yes, it is more complex. Yes, it requires more engineering effort. But that's what makes the difference between a demo that impresses and a system that works reliably in everyday use.


Interested in the technical details? Get in touch for an exchange on a technical level.

Did you find this page helpful?

Thank you for your feedback!

Would you give me feedback? (anonymous)

We develop AI software for companies and deliberately avoid annoying advertising banners. Through our articles, we document topics that occupy and interest us and also finance our daily bread.

As our content is free of charge, your feedback is our praise.

Each author reads your anonymous feedback personally, although AI could automate it, and integrates constructive suggestions directly into the next revision or uses it as inspiration for the next article.



    </article
    • Florian Zyprian
      (Author)

      As CTO at Helm & Nagel GmbH, the company behind the Konfuzio.

    en_USEN