Deepdoctection is a Python library that orchestrates document extraction and document layout analysis tasks using deep learning models.
Deepdoctection focuses on applications and is made for those who want to program real-world solutions to problems related to document extraction from PDFs or scans in various image formats.
Websites vs. documents and scans as information carriers
Suppose you want to analyze the reviews of a website. With a little programming experience, you can create comprehensive studies such as sentiment analyses in a short time.
Now let's assume you want to evaluate customer feedback again. This time, however, the information is available in scanned forms. Perhaps the forms are not all identical and some have been filled in by hand. In this case, you will have to spend considerably more time to achieve your goals.
But what is the big difference?
If you want to obtain your information from websites, you must obtain and process your data using web crawling so that only the information you need for your analysis remains from the raw HTML representation. Up to this point, a rule-based approach is sufficient.
But now to the forms: Perhaps you already have a digitized document, but this document stores its contents in pixel form, i.e. in thousands and thousands of numbers between 0 and 255. OCR engine over it and you will get a combination of words and coordinates.
Now the real work begins, because you still have to give the form structure. For example: How do I recognize the entities in the form? Which word blocks belong together? What structures are there, such as tables or paragraphs?
To address such problems, you need a fund of tools. On the one hand, artificial intelligence can provide help, but on the other hand, artificial intelligence must be orchestrated to be effective in the overall package.
deepdoctection - Pipelines for Document AI
deepdoctection is a Python-based open source framework that supports you in extracting information from documents. You can find the Source code at Github.
Use of image and text
In contrast to common Natural Language Processing frameworks, deepdoctection offers the possibility to create pipelines where multimodal models can be used that read in image information in addition to text. This includes Position information of words and text segments or the entire image. Indeed, it has been shown that especially for documents from the daily business world (forms, reports, presentations), AI models that take visual information into account work better for extraction. This includes the LayoutLM family, a group of models developed by Microsoft.
With deepdoctection you can use these models in combination with OCR for document classification or the Entity extraction try out
By loading the video, you accept YouTube's privacy policy.
Read more
Layout analysis and table extraction
In addition, deepdoctection offers the possibility, Models for document layout analysis to be used. These are extremely useful for structuring the document into specific areas such as tables, figures or list elements.
Tables can also be reconstructed and extracted as a csv file or even as an HTML structure. Deepdoctection you can simply test online.
OCR
OCR is a key technology for Intelligent Document Processing and primarily enables the extraction of text from optical formats such as PDF. For this purpose, deepdoctection offers the option of using various OCR frameworks. These include the best-known open source framework Tesseractbut also the more modern DocTr, which provides more accurate results for many use cases.
Train and evaluate models
No model works for all use cases. But accuracy can be significantly improved by training models on your own data. Deepdoctection offers preconfigured training scripts for various basic models, so you don't have to search for a suitable setting. Before you can use a model productively, you need to evaluate it to see if the prediction results are accurate enough on unseen data.
For this purpose, deepdoctection offers metrics and evaluation tools that you can use to comprehensively assess the predictive power of your model. In addition, deepdoctection offers the option of creating and registering your own data sets. This makes it easier to train basic models and also offers the option of combining several data sets.
Open Source Base Models
Overall, various base models from different high-quality open source libraries can be used with the help of deepdoctection. The execution of several AI models in one simple call is the particular strength of the framework. This is currently one of the most important needs of enterprises. With this in mind, it also makes sense to look at some open source AI models. Here is a selection from the various application areas of document processing.
Vision
DiT - Document Image Transformer is a pre-trained and self-monitored model that works with large text images without labels.
BEiT - This Vision Transformer is based on Google's BERT language model and was able to outperform supervised trained models for the first time thanks to self-monitored pre-training.
Language
Donut - OCR-free Document Understanding Transformer achieves impressive results in document understanding and text extraction - without the need for OCR.
UniLM - This pre-trained language model from Microsoft can be fine-tuned for both the processing and generation of language.
Multimodal
LayoutXLM - This multimodal further development of LayoutLM for multilingual document analysis takes text, layout and image information into account - with higher performance than previous state-of-the-art models.
DocLLM - As an extension of classic large language models, DocLLM is suitable for multimodal processing of the spatial layout structure.
Other PDF-specific Intelligent Document Processing Tools are, for example
Enterprise requirements for AI architecture
With these rapidly growing possibilities and the number of available models for document AI, the technical requirements of large companies have also increased. In the early days of Large Language Models, it initially seemed foreseeable that organizations could continuously train their own models to their individual data and needs. However, the number of applicable and deployed solutions has reached a dimension where this approach is losing its cost efficiency. 60% of enterprises use a variety of models, including Menlo's Enterprise AI report. The majority of willingness to pay is therefore not related to training, but to AI inference, i.e. the standardized usability of various AI models.
At this point, deepdoctection faces certain challenges - despite all the robustness and flexibility that the framework has to offer. Despite preconfigured training and evaluation transcripts, the use of an open source framework is often not sufficient for enterprises. Additional resources are needed to reduce development costs and ensure high security standards. The possibility of in-house development and the integration of other models are also frequent requests. Enterprises are therefore well advised to look for a secure and flexible platform for operating the framework and/or integrating additional models.
Enterprise AI for deepdoctection and other basic models
deepdoctection can be operated with the help of Konfuzio, a platform for enterprise AI. This enables the rapid implementation of individual use cases and the testing of new models in short iteration cycles. This results in Various advantages:
- Optimal use of AI models - deepdoctection also allows the orchestration of the available AI models - in a customized software environment. In addition, any other open source model, e.g. those listed above, can be used with constant availability.
- Less development effort - The general development effort is considerably reduced thanks to a provided interface as well as its own API, SDK and database.
- Marketplace Feature - In addition to using the models shown, numerous pre-trained, industry-specific and ready-to-use AI models are available. Here you can Carry out non-binding tests.
- User Management - In large organizations, data protection and governance policies can be more complex. Konfuzio supports implementation with a sophisticated user management system that allows individual access authorizations to be assigned.
- Enterprise Security - Konfuzio is operated with high security standards at the server location in Germany. Another widely used option is on-premises operation, so that no data leaves the company's own servers.
Document AI in the Python SDK from Konfuzio
Konfuzio makes it possible to Upload your own AI, Individual information extraction and much more. A ready-made software development kit is available for this purpose.
This is how you proceed if you want to use deepdoctection or various document AI models via the Python SDK - using the example of an extraction AI:
- The installation of konfuzio_sdk
!pip install konfuzio-sdk- The required packages can be imported in the following way.
import os
import sys
import konfuzio_sdk
from konfuzio_sdk.data import Project
from konfuzio_sdk.trainer.information_extraction import RFExtractionAI
from konfuzio_sdk.tokenizer.regex import WhitespaceTokenizer
from konfuzio_sdk.api import upload_ai_model- The next step is to initialize the project
from tests.variables import OFFLINE_PROJECT, TEST_DOCUMENT_ID
project = Project(id_=None, project_folder=OFFLINE_PROJECT)- Determination of the corresponding category.
category = project.get_category_by_id(63)- Initialization of the training pipeline. The RFExtraction category is interchangeable.
pipeline = RFExtractionAI(use_separate_labels=True)
pipeline.category = category- For later evaluation, test_documents is required as an attribute of the pipeline.
pipeline.test_documents = category.test_documents()- To query the documents in the specified category:
documents = category.documents()- Apply the pipeline to the documents.
pipeline.fit(documents)- Extraction from new PDFs and uploaded files is then possible.
text = "..."
document = category.create_document(text, filename="test.pdf")
pipeline.process_document(document)
annotations = document.annotations()Deeptoctection as well as your own or the basic models outlined above can be uploaded to Konfuzio using the Python SDK via the upload_ai_model function.
Conclusion
As an open-source framework in the field of document AI, Deepdoctection shows great potential for the orchestration of AI models. It enables efficient information processing through the integration of text and image analysis, which is particularly relevant for the processing of real documents such as forms and reports. The combination of various deep learning models and OCR techniques underlines the flexibility and adaptability of the framework. Despite these extensive possibilities, deepdoctection also has certain limits, particularly with regard to enterprise requirements and ensuring data protection and security.
Cooperation with platforms such as Konfuzio can solve these challenges by simplifying development and accelerating model application. In addition, there are numerous other options for model integration using Python SDK in order to achieve optimal results even for highly complex, individual use cases.
Would you like to operate deepdoctection, the AI models shown here or your own AI models in a secure enterprise environment? Get in touch with us. We will be happy to support you.
About the author of deepdoctection
Dr. Janis Meyer has been working as an independent machine learning engineer and business consultant for many years.
The trigger to deal with Document AI topics for him was a project that involved extracting specific key figures from financial reports.
He has developed deepdoctection to develop in short iteration cycles new Machine Learning models, for solving Document AI issues in Pipelines to integrate.
Dr. Janis Meyer advises end customers and platform operators on implementation and innovation topics of Intelligent Automation topics. This includes general consulting as well as project, implementation, test and production monitoring.
EN 
DE 
ES 
FR