Test LLM prompts | pytest, without API calls, without mocking.

The problem

You have a Prompt, which tells an LLM: Return JSON. Your code parses this JSON. Someone edits the prompt - renames a field, changes a value from String to Int, deletes a key. Your code breaks at runtime. In production.

This happens continuously. Prompts live in Markdown files. Code lives in Python. Nothing connects the two. The prompt says "score", but the code reads "rating". The prompt shows "status": "done", but the code checks for "ready". The prompt has 5 fields, the code expects 6.

The gap between what your prompt tells the LLM and what your code expects is a contract. If no one tests this contract, it will break.

The solution: Prompt Contract Tests

Write JSON-examples into your prompts. Extract them into tests. Validate the structure against what your code actually reads.

No API calls. No mocking. No LLM involved. Only regex + JSON parsing + assertions. Tests run in milliseconds.

Minimal example

Suppose you have a prompt file that instructs an LLM to analyze customer feedback:

<!-- prompts/analyze_feedback.md -->
# Analyse Kundenfeedback
Pruefe das Kundenfeedback und klassifiziere es.
## Output Format
Antworte ausschliesslich mit JSON:
​```json
{
  "status": "ready",
  "sentiment": "positive",
  "topics": ["pricing", "support"],
  "urgency": 3,
  "suggestions": {
    "summary": "Kunde ist zufrieden mit dem Support, aber besorgt ueber Preise.",
    "follow_up": true
  }
}
​```
Falls mehr Kontext noetig ist:
​```json
{
  "status": "needs_info",
  "questions": ["Welches Produkt nutzt der Kunde?", "Wann trat das Problem auf?"],
  "suggestions": {}
}
​```

And your code does something like this:

# app/feedback.py
def process_feedback(llm_response: dict):
    status = llm_response["status"]         # muss existieren
    if status == "ready":
        sentiment = llm_response["sentiment"]   # muss String sein
        topics = llm_response["topics"]         # muss Liste sein
        urgency = llm_response["urgency"]       # muss Int 1-5 sein
        summary = llm_response["suggestions"]["summary"]  # muss existieren
    elif status == "needs_info":
        questions = llm_response["questions"]   # muss nicht-leere Liste sein

Here is the test that combines both:

# tests/test_prompt_contracts.py
import json
import re
from pathlib import Path
PROMPTS_DIR = Path(__file__).resolve().parent.parent / "prompts"

def _read_prompt(filename: str) -> str:
    path = PROMPTS_DIR / filename
    assert path.exists(), f"Fehlt: {path}"
    return path.read_text(encoding="utf-8")

def _extract_json_blocks(text: str) -> list[dict]:
    """Alle ```json ... ``` Bloecke aus einer Markdown-Datei extrahieren."""
    pattern = r"```json\s*\n(.*?)```"
    matches = re.findall(pattern, text, re.DOTALL)
    results = []
    for match in matches:
        results.append(json.loads(match.strip()))
    return results

# --- Hat der Prompt ueberhaupt parsebare JSON-Beispiele? ---
def test_prompt_has_output_format():
    text = _read_prompt("analyze_feedback.md")
    assert "## Output Format" in text
def test_json_blocks_are_valid():
    text = _read_prompt("analyze_feedback.md")
    examples = _extract_json_blocks(text)
    assert len(examples) >= 2, "Brauche mindestens ein 'ready' und ein 'needs_info' Beispiel"
# --- Passen die Beispiele zu dem, was der Code liest? ---
def test_status_values_are_valid():
    text = _read_prompt("analyze_feedback.md")
    for example in _extract_json_blocks(text):
        assert example["status"] in {"ready", "needs_info"}, (
            f"Code prueft auf 'ready' oder 'needs_info', bekommen: '{example['status']}'"
        )
def test_ready_example_has_required_fields():
    text = _read_prompt("analyze_feedback.md")
    ready = [e for e in _extract_json_blocks(text) if e["status"] == "ready"]
    assert len(ready) > 0, "Kein 'ready'-Beispiel im Prompt"
    for example in ready:
        # Genau die Felder, die process_feedback() liest
        assert "sentiment" in example
        assert isinstance(example["sentiment"], str)
        assert "topics" in example
        assert isinstance(example["topics"], list)
        assert "urgency" in example
        assert isinstance(example["urgency"], int)
        assert 1 <= example["urgency"] <= 5
        assert "summary" in example["suggestions"]
def test_needs_info_has_questions():
    text = _read_prompt("analyze_feedback.md")
    needs_info = [e for e in _extract_json_blocks(text) if e["status"] == "needs_info"]
    assert len(needs_info) > 0, "Kein 'needs_info'-Beispiel im Prompt"
    for example in needs_info:
        assert "questions" in example
        assert isinstance(example["questions"], list)
        assert len(example["questions"]) > 0, "Leere Fragenliste"

Execute:

pytest tests/test_prompt_contracts.py -v

test_prompt_has_output_format       PASSED
test_json_blocks_are_valid          PASSED
test_status_values_are_valid        PASSED
test_ready_example_has_required_fields  PASSED
test_needs_info_has_questions       PASSED

What this reveals

Imagine someone edits the prompt and names it "urgency" in "priority" around. Or changes"status": "ready" at "status": "complete". Or delete the "summary"-field from the suggestions.

The tests fail immediately. Before deployment. Before the LLM ever sees the new prompt. Before a customer hits the bug.

Real examples of prompt drift that this pattern captures:

Scaling

For multiple prompts: parameterize.

PROMPTS = {
    "analyze_feedback.md": "feedback",
    "classify_ticket.md": "ticket",
    "summarize_report.md": "report",
}
@pytest.mark.parametrize("filename", PROMPTS.keys())
def test_has_output_format(filename):
    text = _read_prompt(filename)
    assert "## Output Format" in text
@pytest.mark.parametrize("filename", PROMPTS.keys())
def test_json_is_valid(filename):
    text = _read_prompt(filename)
    for block in _extract_json_blocks(text):
        assert "status" in block  # jeder Prompt muss Status zurueckgeben

Then prompt-specific test classes for the fields that only occur for this prompt:

class TestFeedbackContract:
    def test_has_sentiment(self):
        ...
class TestTicketContract:
    def test_has_priority_level(self):
        ...

The principle

The JSON examples in the prompt are the Specification. The field accesses in the code are the Implementation. The test combines both.

┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│ prompt.md   │────▶│  test.py     │◀────│  app.py     │
│ (JSON-Spec) │     │ (Vertrag)    │     │ (liest JSON)│
└─────────────┘     └──────────────┘     └─────────────┘

If the prompt changes, the test fails. If the code reads other fields, you update the test - which forces you to check the prompt. The contract remains synchronized.

What this does NOT test

This pattern validates Structure, not Quality. It answers:

• Does the prompt show the LLM which fields it should return? Yes/No.
• Do these fields match what the code parses? Yes/No.
• Are the types (String, Int, List) correct? Yes/No.
• Are enum values valid („ready“ instead of „complete“)? Yes/no.

It does not answer:

• Does the LLM follow the prompt reliably? (This requires integration tests.)
• Is the prompt well written? (You need people for that.)
• Does the LLM hallucinate field values? (This requires output validation in the code).

But structural drift is the number one cause of prompt-related production bugs, and this pattern catches it at no cost and no latency. Run in CI, with every commit, before every deployment.

Try it out

1. Output Format write to your prompt files with JSON examples
2. _extract_json_blocks() implement (15 lines, simply copy from above)
3. Assert that the fields your code reads exist in the examples
4. Integrate into CI

This is the entire pattern. No framework, no library, no dependency. Just Regex, JSON and pytest.