The Cheaper Answers First

By the end of this session you will be able to:

• Sort a model failure into one of three buckets and pick the cheapest fix that actually addresses that bucket
• Serve Ministral 3 8B under vLLM and force schema-valid output with response_format, without training anything
• Run a two-variant head to head on 30 real failures and read the result as an early go or no-go signal

The three failures that all look like "the model is bad"

Someone hands you a folder of bad outputs and a budget. Before you spend either, read twenty of those outputs and tag each one. Almost everything lands in one of three buckets.

It does not know. The question turns on a fact the model was never shown: a policy written last month, an internal product code, a customer's balance. Fine-tuning is a poor way to install facts, because facts change and weights do not. Tune to fix this and you get a model that states last quarter's refund policy with total confidence for the next year. This bucket wants retrieval.

It will not comply. The answer is right and the shape is wrong. Prose wrapped around the JSON, an enum value the model invented, a missing field, a code fence you did not ask for. This bucket wants constrained decoding, and it is the single most common reason people reach for a tune they did not need.

It does not sound right. Correct, well formed, wrong register. Too long, too hedged, leads with the caveat instead of the answer, uses the wrong word for a thing your industry names differently. Few-shot examples move this partway. This is the bucket where tuning eventually earns its cost, and it is usually the smallest of the three.

If more than half your tagged failures land in the first two buckets, you have a plumbing problem wearing a training problem's clothes. Fix the plumbing, retag, and see what is left.

Constrained decoding, before few-shot, before anything

Schema enforcement is not prompting. Asking nicely for JSON changes the probabilities. A grammar-constrained decoder masks the logits so an invalid token cannot be sampled at all. Your parse rate goes to 100 percent by construction, and it costs you nine lines.

Serve the candidate model first. Ministral 3 8B ships FP8 weights under Apache 2.0 and fits on a single H200:

pip install "vllm>=0.26" openai pydantic
vllm serve mistralai/Ministral-3-8B-Instruct-2512 \
  --tokenizer_mode mistral --config_format mistral --load_format mistral \
  --max-model-len 8192

The default context is 262144 tokens, which reserves far more KV cache than a classification task needs. Setting --max-model-len is how you fit on the card you actually have.

Now pin the output shape:

from typing import Literal
 
from openai import OpenAI
from pydantic import BaseModel
 
 
class Ticket(BaseModel):
    account_id: str
    severity: Literal["low", "medium", "high"]
    summary: str
 
 
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
model = client.models.list().data[0].id
 
completion = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": "Extract the ticket fields from the message."},
        {"role": "user", "content": "Acct 88213. Card declined twice at the pump, I am stranded."},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "ticket", "schema": Ticket.model_json_schema()},
    },
    temperature=0,
)
print(Ticket.model_validate_json(completion.choices[0].message.content))

Two details that will bite you. First, guided_json and its siblings were removed in vLLM 0.12.0, so a pasted 2024 snippet gets you an unknown-field error. The current spellings are the OpenAI-style response_format above, or extra_body with a structured_outputs block for the choice, regex and grammar variants. Second, describe the schema in the prompt as well as passing it. The constraint guarantees the shape, not the content, and telling the model what severity means is what makes it pick the right one.

Few-shot and retrieval, priced per request

In-context examples are not free, they are rented. A 900 token system prompt plus eight worked examples at 300 tokens each is 3,300 tokens of overhead on every request you will ever serve. Weights pay that once. That is a real argument for tuning, and one you are not allowed to make yet, because you do not know if the behavior is reachable at all. So the order is: get it working with eight examples in context, then decide whether to bake it in.

There is a useful negative signal here too. If eight well chosen examples cannot move a style failure at all, a few hundred in a training set usually will not either. Something else is wrong: the task is underspecified, or your labels disagree with each other.

Keep the fixed block first and the variable input last. vLLM can reuse the KV cache for a shared prefix, so a stable system prompt plus stable examples is close to free after the first hit. Interleave the user's text with your instructions and every request recomputes the lot.

Retrieval belongs to the first bucket only. It adds a new failure mode, retrieved the wrong chunk, which looks exactly like a model error in your logs and is not one.

The afternoon head to head

Thirty real failures, two variants, one number each. Write your cases to cases.jsonl, one JSON object per line with a text field and the severity you believe is right.

import json
from typing import Literal
 
from openai import OpenAI
from pydantic import BaseModel, ValidationError
 
 
class Ticket(BaseModel):
    account_id: str
    severity: Literal["low", "medium", "high"]
    summary: str
 
 
SYSTEM = (
    "Extract account_id, severity and summary from the support message. "
    "severity is low, medium or high. Reply with JSON only."
)
SCHEMA = {
    "type": "json_schema",
    "json_schema": {"name": "ticket", "schema": Ticket.model_json_schema()},
}
 
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
model = client.models.list().data[0].id
cases = [json.loads(line) for line in open("cases.jsonl")]
 
 
def run(text, constrained):
    kwargs = {"response_format": SCHEMA} if constrained else {}
    out = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": text},
        ],
        temperature=0,
        **kwargs,
    )
    return out.choices[0].message.content
 
 
for constrained in (False, True):
    parsed = correct = 0
    for case in cases:
        try:
            ticket = Ticket.model_validate_json(run(case["text"], constrained))
        except ValidationError:
            continue
        parsed += 1
        correct += int(ticket.severity == case["severity"])
    print(f"constrained={constrained} parsed={parsed}/{len(cases)} correct={correct}/{len(cases)}")

Thirty cases is not an evaluation set and this is not an evaluation harness. It is a smoke test that tells you in twenty minutes whether the cheap fix already clears your bar.

Try it

Take 30 outputs your current system got wrong. Tag each with one of the three buckets. Then build cases.jsonl from them and run the script above.

You are done when you can write one sentence with four numbers in it: unconstrained parsed X of 30 and got Y right, constrained parsed 30 of 30 and got Z right. Then state which bucket the remaining errors sit in.

Success condition: if Z is at or above the accuracy the product needs, you have your answer for today, and the answer is do not train. If Z is well short and the remaining errors are style or knowledge, you have a candidate for the go side of the memo, and a real baseline to beat.

Common mistakes

• Fine-tuning to fix a format bug. It works, which is the trap. You spend two days and a few hundred dollars in GPU time teaching the model something a JSON schema enforces for free, and you still have no schema, so the next model you swap in breaks the same way.
• Comparing a tune against a prompt you wrote in thirty seconds. The baseline has to be the best prompt you can write, with the schema attached. Any lift measured against a lazy prompt is a number about your prompt, not about tuning.
• Toy few-shot examples. Hand-written examples that are cleaner than production input teach the model a distribution it will never see. Pull your examples from real logs, keep the typos.
• Leaving temperature at its default while comparing variants. Set temperature=0 for anything you intend to count, or run three seeds and report the spread. Otherwise you are measuring sampling noise and calling it a result.
• Counting retrieval as free. It adds latency, an index to maintain, and a silent failure mode. Still cheaper than weights, but price it.

Where this goes next

You now know which failures a prompt, a schema, or a retriever can absorb. The next session, What Tuning Actually Buys, takes the residue: format compliance you cannot constrain, tone no example set reaches, latency from a smaller model, and behavior no prompt encodes because you cannot ship the prompt.