LLM Fine-Tuning with your own data - A practical guide

This guide shows you how to customize a large language model (LLM) to meet your specific needs. From an introduction to low-rank adaptation to efficient data preparation and deployment, you will receive clear guidance on how to build a model that meets your specific needs.

Step by step, you will learn how to use modern methods such as LoRA, work with minimal resources and develop a powerful, customized model. Whether for your company or your own projects - this guide provides you with a practical and comprehensible basis.

With these clear instructions, you will be able to optimize an LLM independently and successfully implement your application ideas. Start with a model that is tailored to your requirements.

Why use a vehicle registration document as an example?

A vehicle registration certificate (registration certificate part 1) may seem inconspicuous at first glance, but in practice it presents an impressively complex challenge. This document contains numerous structured data points in a clearly defined format: from the vehicle identification number (VIN) to the date of first registration and technical data such as total mass, engine capacity or CO₂ values. In addition to all other documents, Konfuzio IDP can also Read out vehicle registration document and automatically validate the data contained therein.

The layout of the vehicle registration document is also challenging: the data is distributed in a very small space, accompanied by descriptive key numbers and elements that are sometimes forgery-proof. In addition, there are variable spellings or abbreviations for manufacturer and type designations, which make correct extraction even more difficult. It is particularly difficult with poorly exposed scans or images taken from unfavorable angles - challenges that often cause conventional OCR systems to fail.

Why is the vehicle registration document complex?

• High data volume: A lot of relevant information has to be identified, extracted and brought into the correct relationship with each other.
• Different formatting: Elements such as tables, fields and line breaks vary slightly depending on the exhibition.
• Counterfeit protection: Security features such as holograms, special color embossing or shading make text recognition more difficult.
• Precision required: The extracted data must not contain any errors; even an additional letter in the VIN has serious consequences.

Using a vehicle registration document as an example, we can show how an LLM can be tuned to process these complex data structures securely and precisely. The aim is to create a system that remains efficient even under difficult conditions - precise, scalable and robust.

Instructions: Adjust LLM to your own data

What does low-rank adaptation do?

Low-rank adaptation is an efficient method for fine-tuning large language models. Instead of changing all model weights, only targeted, compact adjustments are made - in a reduced subspace of the original parameters. This preserves the performance of the model, while the Significant reduction in resource consumption. Less memory, shorter training times, lower computing load - ideal for production-related applications or edge deployments.

Why QLoRA?

QLoRA combines low-rank adaptation with quantized storage - i.e. an extremely compact representation of the model weights. This makes it possible to run even large models efficiently on limited hardware, e.g. on a single GPU server or even locally. For developers, this means: no complete fine-tuning, no high-end infrastructure, but still powerful results. QLoRA reduces the hurdles of model customization - without compromising performance.

Areas of application

QLoRA is the perfect partner for any project that requires precisely tailored language models. Be it in local operation, for specific workflows or for data protection-sensitive tasks. No expansion of unnecessary systems, no waste of resources. Just precision - for today and tomorrow.

Prepare and use data

In order to train a reliable extraction model, unstructured documents must be converted into structured data. Konfuzio supports this process in several steps:

1. Define labels and categories - e.g. Receipt date, Vehicle type or License plate.
2. Annotate texts - mark relevant text passages and assign them to the corresponding labels.
3. Carry out quality assurance - check the annotations for completeness and correctness and correct them if necessary.

Konfuzio converts documents into structured, machine-readable data - a central basis for the training and productive use of AI models for information extraction.

By loading the video, you accept YouTube's privacy policy.
Read more

Load video

Always unlock YouTube

A data record in which certain information (e.g. "Date of approval") is securely marked and exportable.

From text to training data set

Requirements for the input data set

A model requires structured specifications in order to be trained. Standard formats include:

• Description of the task (instruction).
• The input text (input).
• An example of the answer (output).

Automate data formatting

Use a Python script to transfer the annotated data from Konfuzio into a format suitable for training. Example code:

import json
from copy import deepcopy
from konfuzio_sdk.data import Project
# Projekt und Kategorie laden (IDs z. B. aus der Weboberfläche)
project = Project(id_="DEIN_PROJEKT_ID", update=True)
category = project.get_category_by_id("DEINE_KATEGORIE_ID")
for (name, documents) in [("train", category.documents()), ("test", category.test_documents())]:
    dataset = []
    for doc in documents:
        copy_doc = deepcopy(doc)
        entry = {
            "instruction": "Extrahiere Informationen aus Dokumenten.",
            "input": copy_doc.text,
            "output": {label.name: ann.offset_string[0] for label in doc.labels for ann in doc.annotations() if ann.label.id == label.id},
            "id": copy_doc.id_
        }
        dataset.append(entry)
    with open(f"{name}_dataset.json", "w", encoding="utf-8") as f:
        json.dump(dataset, f, indent=4, ensure_ascii=False)

After execution, you will have a JSON file, e.g:

{
    "instruction": "Extrahiere Kennzeichen und Zulassungsdaten.",
    "input": "1. Kennzeichen: B-MW123...",
    "output": {
        "Kennzeichen": "B-MW123",
        "Zulassungsdatum": "01.01.2020"
    },
    "id": "abc123"
}

We use instruction in the Alpaca Format.

# Definition der ALPACA-Prompt-Vorlage
alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
{}
### Input:
{}
### Response:
{}"""
EOS_TOKEN = tokenizer.eos_token  # Sicherstellen, dass das EOS-Token angehängt wird

def formatting_prompts_func(examples):
    instructions = examples["instruction"]
    inputs = examples["input"]
    outputs = examples["output"]
    texts = []
    for instruction, input_text, output in zip(instructions, inputs, outputs):
        text = alpaca_prompt.format(instruction, input_text, output) + EOS_TOKEN
        texts.append(text)
    return {"text": texts}

This file can now be loaded.

import json
import random
import pandas as pd
from datasets import Dataset
from transformers import AutoTokenizer
file_path = "train_dataset.json"  # Pfad zur Datei (bei Bedarf anpassen)
with open(file_path, "r", encoding="utf-8") as f:
    dataset = json.load(f)
# In ein Pandas DataFrame umwandeln
random.shuffle(dataset)
df = pd.DataFrame(dataset)
# Formatierung anwenden
hf_dataset = Dataset.from_pandas(df)
dataset = hf_dataset.map(formatting_prompts_func, batched=True)

Customize model

Procedure for fine-tuning

For fine tuning you need:

1. A basic model (e.g. Llama or Mistral series).
2. A training tool such as the Hugging Face Framework.
3. Formatted data from step 3.

Training with LoRA

LoRA is efficient because only specific model areas are changed. Steps:

• Configure adapter.
• Load data.
• Set parameters such as batch size and learning rate.
• Perform training.

Example:

import torch
from unsloth import FastLanguageModel
# Lade das ursprüngliche Modell
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Llama-3.2-3B-Instruct",
    max_seq_length = 4096,
    dtype = None,
    load_in_4bit = True,
)
# Füge QLoRA hinzu
model = FastLanguageModel.get_peft_model(
    model,
    r = 128,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 32,
    lora_dropout = 0,
    bias = "none",
    random_state = 3407,
    use_rslora = False,
    loftq_config = None,
)
from transformers import TrainingArguments, Trainer
training_args = TrainingArguments(
    output_dir="./results",
    per_device_train_batch_size=4,
    num_train_epochs=5,
    learning_rate=2e-4,
)
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=formatted_data,
)
trainer.train()
model.save_pretrained("finetuned_model") 
tokenizer.save_pretrained("finetuned_model")

The model is adapted so that it can process specific tasks of your data efficiently.

Conclusion

The system described here impressively demonstrates what can be achieved with optimized LLM fine-tuning and modern technologies. With just 4 GB VRAM impressive performance is achieved:

• 99.57 % F1 score: A measure of accuracy and precision, measured on character-precise level - a level that offers maximum reliability even for demanding documents.
• 1.5 seconds per documentA processing speed that is not only efficient, but also scalable.

This setup shows how AI-based data processing is ready for production even with limited resources - without compromising on quality or speed.