Provide a finetuned LLM with REST API

Fine-tuning a language model is a good way to adapt it to a specific domain or use case. In our previous Konfuzio guide for fine tuning we showed how to train an LLM on structured data from German vehicle registration documents. These types of documents, with highly specific fields and legally relevant data formats, benefit greatly from a customized LLM that understands context and terminology.

But once your model is trained, the next step is to make it accessible - ideally in a scalable, maintainable and secure environment. In this blog post, we build on this preliminary work. Now the customized LLM will be implemented with Docker, a REST API, the Konfuzio SDK and BentoML provided.

Why provide your fine-tuned model?

Fine-tuning a Large Language Model (LLM) allows you to use general language understanding while adapting it to the unique characteristics of your data - be it processing legal documents, financial reports or - as in our case - vehicle registration documents. But a fine-tuned model only reveals its value when it is reliably ready for use.

Therefore, the provision of your LLM is important:

• Provision via an HTTP interfaceApplications require simple interfaces to interact with the model. A REST API is the standard for providing predictions from an LLM.
• Integration into other applicationsWhether it's a document pipeline, chatbot or backend service - you want to be able to integrate your model seamlessly.
• Reproducibility and isolationLocal experiments can fail in production if environments are not stable. Docker ensures consistency.
• Scalable accessAs a container and provided via API, your model becomes a scalable microservice - cloud-compatible and Kubernetes-ready.
• Versioned deploymentsBentoML allows you to track, update or roll back model versions - without any infrastructure effort.

The combination of Docker for isolation and BentoML for packaging and deployment lays the foundation for robust, maintainable deployment - with minimal overhead. The model is made easily accessible via an automatically generated REST API based on FastAPI.

Step 1: Integrate LLM into a Konfuzio SDK pipeline

We expand AbstractExtractionAI and build a complete extraction pipeline for vehicle licenses with a fine-tuned LLM. We assume that the trained weights are in the current working directory. Here in the Vinlora folder:

from konfuzio_sdk.trainer.information_extraction import AbstractExtractionAI
class LLMExtractionAI(AbstractExtractionAI):
    alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
        ### Instruction:
        {}
        ### Input:
        {}
        ### Response:
        {}"""
    llm_model_name = "vinlora"
    llm_model = None
    llm_tokenizer = None
    def check_is_ready(self):
        pass
    def load_llm(self):
        from unsloth import FastLanguageModel
        max_seq_length = 2048
        self.llm_model, self.llm_tokenizer = FastLanguageModel.from_pretrained(
            model_name=self.llm_model_name,
            max_seq_length=max_seq_length,
            dtype=None,
            load_in_4bit=True,
        )
        FastLanguageModel.for_inference(self.llm_model)
    def run_llm(self, instruction: str, text: str) -> str:
        self.load_llm()
        inputs = self.llm_tokenizer([
            self.alpaca_prompt.format(instruction, text, "")
        ], return_tensors="pt").to("cuda")
        outputs = self.llm_model.generate(**inputs, max_new_tokens=128, use_cache=True)
        response = self.llm_tokenizer.batch_decode(outputs)
        return response[-1].split("\n")[-1].split("<|end_of_text|>")[0].strip()
    def extract(self, document: Document) -> Document:
        json_str = self.run_llm("Extract values from german car registration certificate and return them as JSON.", document.text)
        self.parse_json_to_annotations(json_str, document)
        return document
    def parse_json_to_annotations(self, json_str: str, document: Document) -> None:
        import re, json
        def normalize_ws(s: str) -> str:
            return re.sub(r'\s+', ' ', s).strip()
        data = json.loads(json_str)
        annotation_set = document.default_annotation_set
        label_mapping = {label.name: label for label in document.project.labels}
        for field_name, value in data.items():
            if field_name in label_mapping:
                label = label_mapping[field_name]
                value_pattern = re.escape(value.strip()).replace(r'\ ', r'\s+')
                match = re.search(value_pattern, document.text)
                if match:
                    span = Span(start_offset=match.start(), end_offset=match.end())
                    Annotation(
                        document=document,
                        label=label,
                        confidence=1.0,
                        label_set=label.label_sets[0],
                        annotation_set=annotation_set,
                        spans=[span]
                    )

Step 2: Build bento service automatically

You do not need your own BentoML service. Our LLMExtractionAI contains a method build_bentowhich provides the entire model workflow as a Bento package:

project = Project(id_=PROJECT_ID, update=True)
pipeline = LLMExtractionAI(category=project.categories[0])
bento = pipeline.build_bento(pipeline)

This method:

• Copies Bento service templates from the konfuzio_sdk
• Integrates metadata and model ID
• Packs everything into a ready-to-use BentoML package

Step 3: Build a Docker container

Use the command line for containerization. This creates a Docker image:

bentoml containerize extraction_<NAME>:latest

Start container:

docker run -p 3000:3000 extraction_<NAME>:latest

Test call:

curl -X POST http://localhost:3000/predict -H "Content-Type: application/json" -d '{"text": "..."}'

The API provided is already documented by a Swagger Interface with Open-API specification.

Conclusion

This guide shows you how to take your fine-tuned LLM from the local development environment to production-ready deployment - integrated in the Konfuzio SDK, delivered as a REST API in the Docker container. Whether you are a data scientist or ML engineer, this architecture creates a structured and scalable foundation.

The method build_bento is a useful tool for packaging models together with metadata, model weights and dependencies and making them versionable - but it is not a panacea. The focus is on a robust overall architecture: a finely tuned model, embedded in a versioned pipeline, provided in a containerized service.

This is a concrete step towards production-ready AI - maintainable, testable and ready for use. More background information on containerization and its advantages can be found in the article: Why companies should use containerization