What is data annotation and why does AI need it?

Regardless of whether one speaks of annotating data, annotating data or data annotation, this refers to the enrichment of raw data with contextual knowledge.

I will explain this possibly cryptic sentence in more detail in the following article.

Data annotations always begin with raw data, which is the starting point for every analysis. It is the foundation on which systems recognize, interpret and categorize content. To understand what is behind annotations, it is worth taking a look at raw data.

What is raw data?

Raw data are observations in technical form and form the preliminary stage of information.

5 Examples of raw data

When you read these examples, you associate them with a memory. If you look at the birthday party, the notary contract, the heartbeat, the job promise or the full bathtub in their technical form, they become abstract.

As a simple example, I will show you the raw data of an image in the coding Base64.

data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAIkAAACJAQMAAAAi6omKAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAADUExURQAAAKd6PdoAAAAJcEhZcwAAMsAAADLAAShkWtsAAAAaSURBVEjH7cGBAAAAAMOg+VNf4AhVAADw1AAKKwABPzKW5wAAAABJRU5ErkJggg==

Data annotation starts with the raw data. The following presentation by Mr. Gossen provides a comprehensive overview.

What does data annotation mean?

Data annotations contextualize raw data. Put simply, the content of raw data is enriched by annotating it.

To stay with the example of the chain of letters: If you copy the aforementioned chain of letters and save it as black.png on your computer, you can open this file. You will then see the following image.

The name thus gives the data an initial context. You could also create a folder on your computer that you call "Colors". This procedure would allow you to "annotate" further context. Through this folder, you annotate that this image is one of the images that shows a color. You could then place it in the blue.png, red.png etc. folder. By doing this, you would, unimpressed by some popular culture debateand annotate that black is a color for "you".

This simple example already demonstrates this. Even the simplest annotations, in the form of the file name or the folder in which you store, or more precisely classify, the image, are an interpretation.

In another context, e.g. when sorting vacation pictures, you would put this picture in the trash. If you are a collector of works of art, the picture could be the Black square by Kazimir Severinovich Malevich.

In short: data annotation is the interpretation of raw data.

It is precisely these interpretations that are used for controlled machine learning. This annotated data is used in the supvervised training approach, compared to the unsupervised trainingso that machines can learn this human interpretation of the raw data.

All data, whether audio, video, sound, image or text, can be annotated. However, the quality, consistency and granularity of the annotation significantly determine the performance of the resulting models.

What are the limits of annotating data?

Before we look at a cat picture as an example, it is worth taking a short mental detour. Because behind every annotation there is not only technology, but also an interpretation - sometimes factual, sometimes contradictory, sometimes even charmingly human.

Annotations structure data, they decide what is visible and how it is evaluated. In banks and insurance companies, for example, this classification is crucial: Is a customer a risk case or an opportunity? Is a loss standardized or are there special features? The quality of the annotation determines the precision of the models - and therefore the basis for every decision. Clarity and consistency in interpretation are not an option, but a duty.

A small digression on a simpler topic - cats

Recognizing different objects in one image was already very easy in 2017, see Computer Vision on the Web with WebRTC and TensorFlow - webrtcHacksTwo cats, a bed and a laptop. Google's TensorFlow framework, which was popular at the time, offered thousands of different types of objects with its Object Detection API, which could be "automatically" recognized using annotated data. Many years later, the technology is much more accurate. However, the following aspect still needs to be considered today:

What at first glance looks like a neutral assignment - "this is a laptop", "this is a bed" - can become complex in detail: Should the two cats be recognized not only as cats, but by their respective names. But these annotations could probably only be made by the master, or more precisely, the expert of these two cats, Chad Hart attach.

This is precisely where the power and pitfalls of annotations become apparent: They make data tangible, but they do so through our eyes - and these eyes are never completely objective. And that is not a weakness, but a call for attention: because machine learning always starts with us.

This image is more than just a technical snapshot: it is proof that machines are beginning to see our world - but only as we describe it to them.

Development and future of data annotation

Regardless of whether we are talking about annotating data, annotation or data interpretation, it is always about enriching raw data with context so that machines are able to learn with this data. Data annotation is the key to turning an untreated technical observation into a meaningful basis for machine learning. However, the way in which this annotation is carried out has changed massively in recent years.

• From manual annotation to automation: In the past, the task of data annotation was simple but labor-intensive: people assigned a meaning to every single observation - whether pixel, text or signal. Every single image, every paragraph of text was meticulously reviewed and labeled manually. This was precise, but not scalable. The increasing demand for data was accompanied by a change in methods.
• Support through pre-annotated data: With the shift towards pre-annotated data, humans became the checker instead of the creator. Algorithms made suggestions that were subsequently corrected. This hybrid method saved time, but also led to new challenges, such as the uncritical acceptance of algorithmic errors and thus the tightening of existing assumptions. It remained a middle ground - faster, but not perfect.
• The entry into synthetic data: Synthetic data was the next logical step. It was created completely artificially using algorithms, simulations or generative models such as GANs, but automatically annotated due to its origin. They solved the problem of data availability with high scalability at the same time. However, there were also limitations here. The problem of domain validity remained unsolved: synthetic data usually reflects idealized conditions and often cannot fully represent real applications.
• Unsupervised transformers and machine understanding: With the rise of unsupervised transformer-based models like GPT, the game changed again. Data annotation seemed to become superfluous. The starting point was the idea that machines could independently infer meaning by analyzing patterns in large amounts of unprocessed data. Instead of explicit labels, the raw flow of data sufficed; instead of annotation, correlation took over. Models like these aimed to recognize meanings from data autonomously - a paradigm shift.

A step backwards? Why annotation will become more relevant again in 2025

Even if unsupervised learning can theoretically do without annotation, in practice it is clear that annotations remain indispensable in this new era. Large language models such as GPT have repeatedly shown that they reach their limits in specific applications if the training data does not undergo clear human verification or adaptation.

Fine, targeted data annotations are important in various areas:

• Domain-specific knowledge: Language models that are to be used in companies or specialized applications require precise, industry-specific adaptations.
• Fairness: Without targeted annotation, unsupervised models reinforce existing biases, as they rely on the data pool in an uncontrolled manner.
• Quality check: High-quality annotation serves as the gold standard to validate, explain or challenge the performance of a model.

Thesis 1: Unsupervised learning does not replace annotation - it makes it more targeted.

The conflict of objectives remains: Unsupervised learning has never replaced annotation, only displaced it. With the multitude of agnostic data sets, it is the precise human correction that makes the difference.

Annotation is no longer carried out en masse, but serves as a Strategic fine-tuning. It is the bridge between machine generality and business relevance. Companies that want to use their models effectively must first invest in a new, targeted form of data annotation.

Thesis 2: Even unsupervised learning is never unsupervised

Even so-called Unsupervised Learning is - on closer inspection - never completely unsupervised. Because even if no explicit labels are specified during the learning process itself, the selection of data always remains a profoundly human decision.

Which data is collected, stored and processed is neither random nor complete - and this is precisely where the hidden supervision lies: a machine cannot know what it is doing. not sees. She lacks the ability to be purposefully curious, to actively observe or even to question her own perspective.

Instead, it is dependent on the world as it is presented to it - selectively, fragmentarily, often distorted. In this way, every supposedly objective discovery becomes a mirror of human assumptions, selection processes and blind spots.

Conclusion

The annotation of data allows raw data to be interpreted. Even if it is assumed in research that annotations can be correct or incorrect, annotations interpret the data through the human annotator.