Data annotation with LLMs - The future of data labeling

In the rapidly evolving landscape of AI technologies, data annotation plays a crucial role in the training of machine learning models. Accurately labeled data is the foundation for model performance. Traditionally, manual labeling of data has been the preferred method, but for modern companies it is increasingly becoming a thing of the past.

In this blog post, we will explore the evolution from manual to automated data annotation and finally to the superior form of automated labeling with large language models (LLMs). We will also look at the concept of hybrid annotation, which combines human assistance with LLMs to achieve the best possible result.

Manual data annotation - the traditional approach

Manual labeling, also known as human annotation, is a fundamental process in data annotation and plays a crucial role in various machine learning projects and AI applications. It involves human annotators reviewing and assigning labels to data based on specific criteria or guidelines.

Although this method offers a high degree of precision, it is labor-intensive, time-consuming and expensive. Furthermore, in the context of modern data labeling and machine learning applications, manual labeling is being used less and less by modern companies.

Automated data annotation - a step towards efficiency

As companies seek to overcome the limitations of the manual labeling process, they are increasingly turning to automated solutions. These often use rule-based algorithms and predefined guidelines to automatically label data based on text or images. With the rise of machine learning algorithms. it became possible to automate the assignment of labels to data with high precision.

Labeling with large language models - The superior form of automation

Large language models (LLMs) are advanced AI models that have revolutionized data annotation. They use vast amounts of data and sophisticated algorithms to understand, interpret and create text in human language. LLMs have the ability to understand the context, linguistic nuances and even the specific goals of a labeling task.

Hybrid labeling - combining human expertise with LLMs

Although LLMs offer unparalleled efficiency and quality in automated data annotation, there are still scenarios where human expertise is indispensable. Hybrid data annotation combines the strengths of humans and LLMs. In this approach, LLMs create pre-labeled data, and human annotators review and refine the annotations to ensure accuracy and compliance with specific requirements.

Use of the Konfuzio SDK to automate data labeling

We will now look at how you can Konfuzio SDK to automate data annotation with LLMs. We will go through the steps of creating a project, uploading documents, creating categories, separating documents with LLMs, assigning categories, and creating labels to achieve a fully annotated dataset whose information is available in the Konfuzio DVUI can be checked.


Before you start, make sure that you have installed the Konfuzio SDK and have access to a Konfuzio server. Install the SDK with the following command:

pip install confuzio_sdk

Step 1 - Setting up your Konfuzio project

First, we need to create a new project and upload documents.

from konfuzio_sdk.api import Project

# Creating a new project
project = Project.create(name="My LLM labeling project", description="Project for labeling data with LLMs")

# Upload documents
document paths = ["path/to/document1.pdf", "path/to/document2.pdf"]
for path in document paths:

Here we create a new project with a name and a description. We then upload documents to the project. These documents will be automatically annotated later.

Step 2 - Creating categories and separating documents

Once the documents have been uploaded, we need to create categories and divide the documents into these categories using LLMs.

from konfuzio_sdk.api import Category

# Create categories
category1 = Category.create(project=project, name="Category 1")
category2 = Category.create(project=project, name="Category 2")

# Separate documents with LLMs
# Assuming `split_document_with_llm` is a user-defined function that uses LLM to split documents into categories
def split_document_with_llm(document):
# Pseudocode for splitting documents
splits = []
# Here would come the LLM code that analyzes and splits the document
# Example splits:
splits.append({'category': category1, 'content': '...'})
splits.append({'category': category2, 'content': '...'})
return splits

for document in project.documents:
splits = split_document_with_llm(document)
for split in splits:
split_document = project.upload_document(content=split['content'])

We create two categories and define a function split_document_with_llmwhich analyzes a document and splits it into different parts, each of which is assigned to a category. The split documents are uploaded and assigned to the corresponding categories.

Step 3 - Assigning documents to categories

In this step, we assign the documents to the categories created.

# Assign documents to categories
for document in project.documents:
if some_condition_for_category1(document):

Here we define a condition (some_condition_for_category1), which determines which category a document is assigned to. The documents are then assigned to the corresponding categories.

Step 4 - Creating labels

Now we create the labels and annotate the documents.

from konfuzio_sdk.api import Label

# Create labels
label1 = Label.create(project=project, name="Label 1")
label2 = Label.create(project=project, name="Label 2")

# Annotate documents with labels
for document in project.documents:
for page in document.pages:
for annotation in page.annotations:
if condition_for_label1(annotation):

We create two labels and define a condition (condition_for_label1), which determines which label is assigned to an annotation. The documents are then annotated accordingly.

Step 5 - Review in the Konfuzio DVUI

With all labeled documents, you can now check the labeled record in the Konfuzio DVUI to ensure accuracy and completeness of the information.


Data annotation is a crucial step in the training of machine learning models. The manual annotation method is used less and less by modern organizations due to its limitations in terms of scalability, cost efficiency, accuracy and speed. Automated approaches, especially those that large language models have emerged as superior alternatives that address these shortcomings. Hybrid labeling, which combines human expertise with LLMs, represents a pragmatic approach that leverages the strengths of both methods to achieve the highest levels of accuracy and scalability.

Platforms like Konfuzio offer seamless integration of LLMs and human annotators, enabling organizations to leverage the full potential of data annotation.

In addition to general information about data annotation, this guide has shown how to set up a Konfuzio project, upload documents, create categories for the data, separate documents with LLMs, assign categories and create labels to get a fully labeled data set that can be reviewed.

Glossary in the area of data annotation and automation

Data annotation and automation

Data annotation is an essential part of training machine learning models. Data annotation services play a key role in providing high-quality, annotated data that is used for various AI applications. The process of data annotation can be manual or automated and involves tagging data sets with relevant text labels that help the models recognize and learn patterns in the data.

Data Annotation Companies

Data annotation companies are specialized service providers that offer high-quality annotation services for various industries. These companies use human annotators or advanced algorithms to label data and ensure that it is suitable for machine learning models.

Annotated Data

Annotated data is data that has been provided with labels or tags to highlight certain characteristics or information. These annotations help machine learning models to better understand and process the data by identifying and classifying relevant information.

Automated data analysis and classification

Automated data analysis and classification refers to the use of software and algorithms to process and interpret large amounts of data without human intervention. These technologies enable companies to gain insights into their data more quickly and efficiently and make informed decisions.

Automated Data Analysis

Automated Data Analysis is the process of using algorithms to automatically examine and analyze data sets. This method saves time and resources by detecting patterns and anomalies in large amounts of data that are difficult for the human eye to recognize.

Automated Data Analytics

Automated data analytics is an advanced form of data analysis that uses advanced algorithms and machine learning models to provide deeper insights and predictions. These analytics can be implemented on platforms such as AWS (Amazon Web Services) to ensure scalability and efficiency.

Automated data collection and classification

Automated data collection and classification encompass technologies and methods that automate the collection and organization of data. These processes are crucial for managing large volumes of data and preparing data for analysis or further processing.

Automated Data Collection

Automated Data Collection is the use of technology to automatically collect data from various sources. This method reduces manual effort and ensures that data is collected in real time, which is beneficial for current analyses and decision-making processes.

Automated Data Classification

Automated Data Classification is the process of automatically assigning data to predefined categories. This is done by algorithms that analyze data characteristics and classify the data accordingly in order to increase the efficiency and accuracy of data processing.

Automated data labeling

Automated data labeling refers to the use of algorithms to automatically assign labels to data sets. This is an important step in data preparation for machine learning models and significantly reduces the time and effort required compared to manual data annotation.

Automated Data Labeling

The technology uses advanced algorithms to automatically assign labels to data. This method improves the efficiency of data annotation and enables companies to process large volumes of data quickly and accurately.

Automatic Data Labeling

Automatic Data Labeling is a synonym for Automated Data Labeling and also refers to the automatic assignment of labels to data sets. This technique is particularly useful in applications that require fast and scalable data annotation.

Automated image annotation and special applications

Automated image annotation and specialized applications include advanced technologies and vision models for automatic annotation of image data as well as specialized methods for cell type annotation in biological datasets. These techniques are of great importance in areas such as biomedical research and image processing.

Automated Image Annotation

Automated image annotation is the use of algorithms to automatically annotate image data. This method is often used in computer vision to identify and label objects in images.

Maximilian Schneider Avatar

Latest articles