Container orchestration deep dive

Container orchestration is the automation of deployment, scaling and management containerized applications. For AI-intensive workloads such as Konfuzio's document processing, Kubernetes (K8s) enables elastic scaling, GPU resource management and cross-cloud portability. The modern infrastructure for Cloud hosting and On-premise deployments is increasingly relying on container technology to maximize flexibility and scalability.

What is container orchestration?

Containers 101

Container technology has revolutionized the way applications are developed and operated. Unlike virtual machines, which virtualize an entire operating system, containers share the kernel of the host system and encapsulate only the application itself, including its specific dependencies. This architecture leads to significantly lower resource consumption and makes it possible to run dozens of containers on hardware that would only be able to handle a few VMs.

container pack applications with all their dependencies into isolated, portable units.

Advantages:

• PortabilityRuns everywhere (laptop, on-premise, cloud)
• Insulation: No conflicts between applications
• EfficiencyShare OS kernel (more lightweight than VMs)
• SpeedStart in seconds (VMs: minutes)

Docker example:

# Konfuzio OCR-Service
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "ocr_service.py"]

# Build & Run
docker build -t konfuzio-ocr:v1 .
docker run -d -p 8000:8000 konfuzio-ocr:v1

Why orchestration?

Problem without orchestration:

• Manual deployment on each server
• No auto-restart in the event of a crash
• No automatic scaling
• Configure load balancing manually

orchestration:

• ✅ Automatic deployment via cluster
• Self-healing (restart in the event of an error)
• ✅ Horizontal Pod Autoscaler (HPA)
• ✅ Service discovery & load balancing
• Rolling updates without downtime

Kubernetes: de facto standard for orchestration

Kubernetes architecture

Kubernetes Cluster
├── Control Plane (Management)
│   ├── API-Server (Zentrale Schnittstelle)
│   ├── etcd (Key-Value-Store für Cluster-State)
│   ├── Scheduler (Platziert Pods auf Nodes)
│   └── Controller Manager (Steuert Desired State)
└── Worker Nodes (Ausführung)
    ├── kubelet (Agent auf jedem Node)
    ├── kube-proxy (Netzwerk-Routing)
    └── Container Runtime (Docker/containerd)

Core concepts

Pod:

• Smallest deployable unit
• One or more containers
• Shared Network & Storage

Deployment:

• Describes desired state
• Automatic ReplicaSet management
• Rolling Updates

Service:

• Stable network endpoint for pods
• Load balancing via replicas
• DNS name (e.g. ocr-service.default.svc.cluster.local)

ConfigMap & Secret:

• Configuration separate from code
• Secrets encrypted

PersistentVolume:

• Persistent storage for pods
• Outlasts pod restarts

Konfuzio on Kubernetes: example deployment

Minimal setup (Docker Compose → Kubernetes)

Alt (Docker Compose):

version: '3'
services:
  ocr:
    image: konfuzio/ocr:latest
    ports:
      - "8000:8000"
    environment:
      - DB_HOST=postgres

New (Kubernetes):

# ocr-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ocr-service
spec:
  replicas: 3  # 3 Instanzen für Load Balancing
  selector:
    matchLabels:
      app: ocr
  template:
    metadata:
      labels:
        app: ocr
    spec:
      containers:
      - name: ocr
        image: konfuzio/ocr:latest
        ports:
        - containerPort: 8000
        env:
        - name: DB_HOST
          value: postgres-service
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
---
# ocr-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: ocr-service
spec:
  selector:
    app: ocr
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8000
  type: LoadBalancer  # Extern erreichbar

Deployment:

kubectl apply -f ocr-deployment.yaml
kubectl get pods  # 3 OCR-Pods sollten laufen
kubectl get service ocr-service  # Externe IP holen

Auto-scaling with HPA

Horizontal Pod Autoscaler:

# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ocr-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ocr-service
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
      - type: Percent
        value: 100  # Verdopplung möglich
        periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300  # 5 Min Cooldown
      policies:
      - type: Pods
        value: 1  # Max 1 Pod/5min runterskalieren
        periodSeconds: 300

Functionality:

Normal-Betrieb (1.000 Docs/Tag):
- CPU: 30% durchschnittlich
- HPA: 2 Pods (Minimum)
Lastspitze (10.000 Docs/Tag):
- CPU: 85% (über Threshold 70%)
- HPA skaliert hoch: 2 → 4 → 8 → 12 Pods
- CPU sinkt auf 60%
Nach Lastspitze:
- CPU: 20% (unter Threshold)
- HPA skaliert runter: 12 → 11 → 10 → ... → 2 Pods
  (langsam, 1 Pod alle 5 Minuten)

GPU management in Kubernetes

NVIDIA Device Plugin:

# Installation
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/main/nvidia-device-plugin.yml
# Verifizierung
kubectl get nodes -o json | jq '.items[].status.allocatable'
# Sollte "nvidia.com/gpu: 2" zeigen (wenn Node 2 GPUs hat)

GPU pod:

apiVersion: v1
kind: Pod
metadata:
  name: ocr-gpu
spec:
  containers:
  - name: ocr
    image: konfuzio/ocr-gpu:latest
    resources:
      limits:
        nvidia.com/gpu: 1  # 1 GPU anfordern

GPU sharing (MIG - Multi-Instance GPU):

# A100 GPU in 7 Instanzen aufteilen
apiVersion: v1
kind: Pod
metadata:
  name: ocr-mig
spec:
  containers:
  - name: ocr
    image: konfuzio/ocr:latest
    resources:
      limits:
        nvidia.com/mig-1g.5gb: 1  # 1/7 einer A100 (5GB VRAM)

Advantages:

• ✅ Multiple workloads share an expensive GPU
• ✅ Better GPU utilization (no idle time waste)
• ✅ Isolation (one workload does not crash the GPU for others)

Storage: Persistent volumes for ML models

Problem: Pods are ephemeral (can restart at any time, data gone)
Solution: PersistentVolumes for models, documents

PVC (PersistentVolumeClaim):

# models-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: models-storage
spec:
  accessModes:
    - ReadWriteMany  # Mehrere Pods gleichzeitig
  storageClassName: nfs  # NFS für Shared-Access
  resources:
    requests:
      storage: 100Gi
---
# ocr-deployment mit PVC
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ocr-service
spec:
  template:
    spec:
      containers:
      - name: ocr
        image: konfuzio/ocr:latest
        volumeMounts:
        - name: models
          mountPath: /app/models
      volumes:
      - name: models
        persistentVolumeClaim:
          claimName: models-storage

Advantages:

• Models do not need to be downloaded with every pod start
• Shared access: All OCR pods use the same models
• Outlasts pod restarts

Kubernetes distributions: Which one to choose?

Managed Kubernetes (Cloud)

AWS EKS (Elastic Kubernetes Service):

• Advantages: Deeply integrated with AWS services (IAM, VPC, EBS)
• Disadvantages: AWS-specific configuration (less portable)
• Cost: 0.10€/h for Control Plane + Worker Node costs
• See: Cloud hosting

Google GKE (Google Kubernetes Engine):

• Advantages: Best K8s integration (K8s was developed at Google)
• Disadvantages: Smaller ecosystem than AWS
• Cost: Free Control Plane + Worker Node costs

Azure AKS (Azure Kubernetes Service):

• Advantages: Best Microsoft integration (AD, DevOps)
• Disadvantages: More complex network configuration
• Cost: Free Control Plane + Worker Node costs

Self-Managed Kubernetes (On-Premise)

Rancher (RKE/RKE2):

• Advantages: Simple on-premise setup, web UI
• Disadvantages: Less enterprise support than Red Hat
• Cost: Open Source (free of charge)

Red Hat OpenShift:

• Advantages: Enterprise support, security features, operator hub
• Disadvantages: High license costs (approx. 50€/core/year)
• Cost: from €15,000/year for small clusters

Vanilla Kubernetes (kubeadm):

• Advantages: Maximum control, no vendor lock-in
• Disadvantages: Complex setup, no support
• Cost: Open Source (free of charge)

K3s (Lightweight K8s):

• Advantages: Minimal footprint (40MB binary), ideal for Edge/IoT
• Disadvantages: Some features removed (cloud provider integrations)
• Cost: Open Source (free of charge)

Konfuzio recommendation:

• Cloud: GKE (best K8s experience) or EKS (largest ecosystem)
• On-Premise: Rancher (simple) or OpenShift (Enterprise with support)
• Edge: K3s

Microservices architecture with Kubernetes

Monolith (old):

┌───────────────────────┐
│   Konfuzio Monolith   │
│  (API + OCR + DB)     │
└───────────────────────┘

• A large container
• Difficult to scale (all or nothing)
• One mistake brings everything down

Microservices (new):

┌─────────┐   ┌─────────┐   ┌──────────┐
│   API   │ → │   OCR   │ → │ Extract  │
│ Gateway │   │ Service │   │ Service  │
└─────────┘   └─────────┘   └──────────┘
     ↓              ↓               ↓
┌───────────────────────────────────────┐
│         PostgreSQL (Database)         │
└───────────────────────────────────────┘

• Each service scales independently
• API gateway: 2 replicas (low load)
• OCR service: 10 replicas (computationally intensive)
• Extract service: 5 replicas

Kubernetes deployment:

# api-gateway (wenig Last)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: api
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
---
# ocr-service (rechenintensiv, GPU)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ocr-service
spec:
  replicas: 10
  template:
    spec:
      containers:
      - name: ocr
        resources:
          requests:
            cpu: "2000m"
            memory: "4Gi"
            nvidia.com/gpu: "1"
---
# extraction-service (mittel)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: extraction-service
spec:
  replicas: 5
  template:
    spec:
      containers:
      - name: extraction
        resources:
          requests:
            cpu: "1000m"
            memory: "2Gi"

Advantages:

• Optimum use of resources (each service as required)
• Isolated errors (OCR crash does not affect API)
• Independent deployments (extraction update without OCR downtime)

CI/CD with Kubernetes

GitOps: Infrastructure as Code

Concept:

• Git repository as a single source of truth
• Changes via Git commit (no manual changes) kubectl apply)
• Automatic deployment via CI/CD pipeline

Flux/ArgoCD (GitOps operators):

# FluxCD installieren
flux bootstrap github \
  --owner=konfuzio \
  --repository=k8s-manifests \
  --path=clusters/production
# Flux überwacht Git-Repo
# Bei neuem Commit:
git commit -m "Update OCR to v2.3"
git push
# → Flux erkennt Änderung automatisch
# → Deployed neue Version auf Cluster

Advantages:

• Audit trail (every change in Git history)
• ✅ Rollback simple (git revert)
• Disaster recovery (cluster recreation from Git)

CI/CD pipeline example (GitHub Actions)

# .github/workflows/deploy.yml
name: Build and Deploy
on:
  push:
    branches: [main]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Build Docker Image
      run: |
        docker build -t konfuzio/ocr:${{ github.sha }} .
    - name: Push to Registry
      run: |
        docker push konfuzio/ocr:${{ github.sha }}
  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
    - name: Update Kubernetes Manifest
      run: |
        kubectl set image deployment/ocr-service \
          ocr=konfuzio/ocr:${{ github.sha }} \
          --record
    - name: Wait for Rollout
      run: kubectl rollout status deployment/ocr-service
    - name: Run Smoke Tests
      run: |
        curl https://api.konfuzio.com/health
        # Wenn Fehler: kubectl rollout undo

Monitoring & Observability

Prometheus + Grafana Stack

Installation (helmet):

# Prometheus Operator
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack
# Zugriff
kubectl port-forward svc/prometheus-grafana 3000:80
# → http://localhost:3000 (User: admin, Password: prom-operator)

Key Metrics:

• Node level: CPU, RAM, Disk, Network
• Pod level: Container CPU/RAM, restart count
• Application: Request rate, latency, error rate
• GPU: GPU utilization, VRAM usage, temperature

Grafana dashboard (example):

┌─────────────────────────────────────────┐
│  Konfuzio OCR Service Dashboard         │
├─────────────────────────────────────────┤
│  Pods Running: 12 / 12 (100%)           │
│  Request Rate: 150 req/s                │
│  P95 Latency: 1.8s                      │
│  Error Rate: 0.05%                      │
├─────────────────────────────────────────┤
│  [Graph: CPU Usage over Time]           │
│  [Graph: Memory Usage]                  │
│  [Graph: GPU Utilization]               │
└─────────────────────────────────────────┘

Logging: ELK stack or Loki

Loki (lightweight alternative to Elasticsearch):

# Loki + Promtail (Log-Collector)
helm install loki grafana/loki-stack
# Logs in Grafana
# → Explore → Data Source: Loki
# → Query: {app="ocr-service"} |= "error"

Advantages:

• Central log aggregation across all pods
• Correlation with Metrics (Prometheus + Loki in one dashboard)
• Label-based queries (similar to Prometheus)

Security: Best Practices

1. network policies (micro-segmentation)

# Nur API-Gateway darf auf OCR-Service zugreifen
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ocr-service-policy
spec:
  podSelector:
    matchLabels:
      app: ocr
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api-gateway
    ports:
    - protocol: TCP
      port: 8000

2. pod security standards

# Restrictive Pod Security
apiVersion: v1
kind: Pod
metadata:
  name: ocr-secure
spec:
  securityContext:
    runAsNonRoot: true  # Nicht als Root laufen
    runAsUser: 1000
    fsGroup: 1000
  containers:
  - name: ocr
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true  # Read-Only-FS
      capabilities:
        drop:
        - ALL  # Alle Linux-Capabilities droppen

3. secret management

# Secrets verschlüsselt in etcd
kubectl create secret generic db-password \
  --from-literal=password=SuperSecure123
# In Pod nutzen
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-password
          key: password

Better alternative: Sealed Secrets:

# Sealed Secrets: Verschlüsselte Secrets in Git
kubeseal < secret.yaml > sealed-secret.yaml
git add sealed-secret.yaml
# Sicher für Git (nur Cluster kann entschlüsseln)

Best alternative: Vault Integration:

• See Container Security Scanning

Cost optimization

1. spot instances / preemptible VMs

AWS:

# Node Group mit Spot Instances (70% günstiger)
apiVersion: v1
kind: Node
metadata:
  labels:
    node.kubernetes.io/lifecycle: spot

Kubernetes Tolerations:

# Pods tolerieren Spot-Instance-Evictions
apiVersion: v1
kind: Pod
spec:
  tolerations:
  - key: "node.kubernetes.io/lifecycle"
    operator: "Equal"
    value: "spot"
    effect: "NoSchedule"

Important: Only for stateless workloads (stateless)

2. set resource requests correctly

❌ Error: Over-provisioning

resources:
  requests:
    cpu: "4000m"
    memory: "8Gi"
# Tatsächliche Nutzung: 500m CPU, 2Gi RAM
# → 75% Ressourcen verschwendet

✅ Correct: Optimize after monitoring

# Prometheus-Query: Tatsächliche Nutzung
avg(rate(container_cpu_usage_seconds_total[1d]))
# Ergebnis: 550m
# Requests entsprechend setzen (+ 20% Buffer)
resources:
  requests:
    cpu: "700m"
    memory: "2.5Gi"

3. vertical pod autoscaler (VPA)

# VPA empfiehlt/setzt automatisch Resource Requests
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: ocr-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ocr-service
  updatePolicy:
    updateMode: "Auto"  # Automatisch Pods mit neuen Requests neu starten

Troubleshooting

Pod does not start

# Status prüfen
kubectl get pods
# NAME                   READY   STATUS              RESTARTS
# ocr-6d5b8f9-xxx        0/1     ImagePullBackOff    0
# Logs
kubectl describe pod ocr-6d5b8f9-xxx
# Events: Failed to pull image "konfuzio/ocr:v99": not found
# → Image-Tag falsch oder Registry-Auth fehlt

Pod crashes repeatedly

# Logs des abstürzenden Pods
kubectl logs ocr-6d5b8f9-xxx --previous
# OOMKilled → Speicher-Limit zu niedrig
# Memory-Limit erhöhen
resources:
  limits:
    memory: "8Gi"  # vorher: 2Gi