JSON Toolkit

When an LLM JSON Output Is Truncated: Read the Stop Reason First

Bottom line: The model returns JSON that stops mid-value, and the parser only gives you a column number. Closing the brackets makes it parse again, but every missing field is now hidden inside the data. This article draws the line between what truncation repair can honestly recover and what must be retried, from stop_reason gating to schema field ordering.

Advanced~5 min readPublished 2026-11-14Updated 2026-11-14
Stack-based closing of a truncated JSON document after a string-aware scan

Truncation is not a syntax error, it is missing data. Syntax can be repaired; truncation can only be partly recovered. The moment a repair helper closes the gap, the remaining gap stops being a parse error and starts being something nobody knows about.

What truncation looks like: a column number will not save you

The model writes JSON against your schema, hits the output cap halfway through, and the text just stops. Feed this into Python 3.11's json module:

{"scores": {"clarity": 8, "depth": 7}, "items": [{"name": "a", "qty": 2}, {"name": "b", "qty"

json.loads raises Expecting ':' delimiter, col 94. The column points at the "qty" key with no value behind it, but the actual cause is that the document has no ending at all. Closing it with a greedy regex plus "} fails outright; a string-aware stack close (section six below) parses, but items holds 2 entries and the third partial record is gone. The column number tells you where the text stopped. It never tells you how much you lost.

That is the first real difference between truncation and an ordinary JSON error. A trailing comma has a well-defined end state: remove it and the document is whole. Truncation has no such end state. Repair it and you still have a fragment.

Truncation is not syntax damage: the blind spot in every repair helper

Repair helpers like jsonrepair exist to turn syntactically broken JSON into legal JSON: strip markdown fences, drop trailing commas, convert single quotes. In those cases the intent is intact and the repair is just cleaning. A truncated input looks like broken JSON to the same helper, so it closes the open string, the open array and the open object, and hands back something that parses. Of 83 truncated Claude reports, the helper "fixed" 79. Only 4 failed hard. The error monitor saw a failure rate twenty times lower than reality.

What did the repaired output look like? Question 6's feedback string ended mid-sentence, questions 7 through 9 did not exist, and the next_steps array was gone. Pydantic validated it anyway, because next_steps had a default of [] and the feedback list had no length constraint. Every layer did exactly its job. The product produced a confidently wrong report.

Put syntax repair into a truncation path and it quietly changes from cleaning to fabricating. So the boundary has to come first: a repair helper should only handle formatting slop on complete answers; a cut-off response should never reach it. For the formatting-slop side, see the JSON repair guide.

Step one is always the stop reason

Every major API tells you why generation stopped, in the response metadata, for free. It is also the single most ignored line of code in LLM integrations. Anthropic exposes stop_reason, OpenAI's chat API exposes finish_reason: max_tokens means truncated, end_turn or stop means the model finished on its own.

resp = client.messages.create(model=MODEL, max_tokens=4096, ...)
if resp.stop_reason != "end_turn":
    raise TruncatedOutput(
        f"stop_reason={resp.stop_reason}, "
        f"output_tokens={resp.usage.output_tokens}"
    )
data = json.loads(resp.content[0].text)

Once truncation becomes an explicit exception, the retry path has a trigger and monitoring has a signal. Those 83 reports were found afterwards with a single query over the stop_reason field in logs that had been stored all along and never read. Gemini reports length/incomplete in the same spirit; when you add a new vendor, check the mapping table before assuming anything.

max_tokens is a safety rail, not a budget

The direct cause is usually an output cap sized for an earlier version of the product: a 1500-token limit that starts cutting 39% of long reports the moment long sessions ship. Raising the cap is free. You are billed for tokens actually generated, so a 4096 cap on a report that needs 1100 tokens costs exactly the same as a 1500 cap. The cap exists to stop runaway output, not to save money.

Size it from data: watch the real output-token distribution for a week, take p99, leave headroom. The team behind the incident chose 4096 because their p99 was 2350 after the fix. The other direction is to shrink the output itself. Repeated keys in homogeneous arrays are where tokens go, and swapping in a tabular encoding before the call (see the TOON format guide) cuts the volume that has to fit under the cap.

Schema field order decides what gets cut

Models write JSON top to bottom. When the budget runs out, whatever sits at the end of your property list disappears first. A report schema ordered summary, per_question_feedback, scores, next_steps loses the scores first, and scores are the part users screenshot and compare week to week. The fix is a reorder, not a redesign: short critical fields first, long regenerable detail later, free-text notes last.

Vendor behaviour also changes the consequence of the same truncation:

Vendor Structured output mechanism Behaviour when truncated
OpenAI response_format with strict: true Sampling-level guarantee but the narrowest dialect; usually refuses to generate
Anthropic tool use input_schema No decode-time guarantee; may hand back unclosed braces
Gemini responseSchema with JSON MIME type Decode enforces enums and types only; truncation still possible

So "structured output is on, no validation needed" is a myth. The looser the vendor guarantee, the more you need your own schema validation and retry loop. How to build that contract test is covered in JSON Schema contract testing.

When you genuinely need partial recovery: string-aware stack closing

Some cases must squeeze something out of the fragment: streaming previews, degraded responses, human review queues. There the closing operation is the last rung of a repair ladder, and it has two requirements. First, the bracket scan must be string-aware: track quote state and escapes character by character, otherwise a { inside a value corrupts the stack. Second, delete partial key-value pairs before closing. A "qty" key with no value cannot be closed into {"qty"}; the whole pair has to go.

def close_truncated(s: str) -> str:
    # 1) String-aware scan; close an open string first
    out, in_str, esc = [], False, False
    for ch in s:
        out.append(ch)
        if esc: esc = False; continue
        if ch == '\\' and in_str: esc = True; continue
        if ch == '"': in_str = not in_str
    text = ''.join(out) + ('"' if in_str else '')
    # 2) Remove partial pairs and dangling colon or comma
    text = re.sub(r',\s*"[^"]*"\s*$', '', text)
    text = re.sub(r'[{,]\s*"[^"]*"\s*:\s*$', '}', text)
    text = re.sub(r':\s*$', '', text)
    text = re.sub(r',\s*$', '', text)
    # 3) Rescan the stack and close LIFO
    stack, in_str, esc = [], False, False
    for ch in text:
        if esc: esc = False; continue
        if ch == '\\' and in_str: esc = True; continue
        if ch == '"': in_str = not in_str; continue
        if not in_str:
            if ch in '{[': stack.append(ch)
            elif ch in '}]' and stack: stack.pop()
    return text + ''.join('}' if c == '{' else ']' for c in reversed(stack))

Run on the input from the opening section, this returns {"scores": {...}, "items": [{"name": "a", "qty": 2}, {"name": "b"}]} on Python 3.11 with the standard library only. It parses, and the third record is dropped. Local-model tooling uses dedicated libraries such as partial-json-parser for the same job: close by stack, trim by completeness.

The critical part is disclosure. The closing function must report which rung produced the result; anything produced by the truncation rung is a partial result and must be labelled as such: re-ask for the missing scope, retry with a larger cap, or tell the user their data is incomplete. Silently pretending nothing happened is manufacturing hallucinations with deterministic code.

Repair or retry: a boundary table

Input shape Correct action Why
Fences, trailing commas, single quotes Repair helper Intent is complete, only syntax is dirty
Cut mid-string Retry, or partial recovery plus disclosure The value itself is missing
Cut mid-structure Partial recovery plus disclosure Completed fields survive, partial pairs must go
Parses but looks wrong Business validation Not truncation, but frequently concurrent

A finance team put a number on the cost. Long invoices were truncated, the bracket-closing step produced valid JSON, the missing total took a default of 0, and a system with 98.2% format compliance shipped roughly 200 wrong invoices a month. After they treated truncation as a hard downgrade and added an identity check (total == sum of line items), the monetary error rate fell from 0.8% to 0.02% while format compliance moved only to 99.1%. They had optimized "format compliance" for three months against the wrong metric. Passing the parser is not the same as passing the business.

The four-item checklist: gate on the stop reason before parsing; size the cap from p99 rather than as a budget; put critical schema fields first; let the repair helper see only complete answers' formatting slop. Do all four and truncation turns from silent data corruption back into a loud, retryable failure.

Can I just close the brackets and use the result

Not directly. Closing only makes it syntactically legal; the data is still a fragment with a half sentence at the end, dropped trailing records, and default values standing in for real ones. Mark the output as partial, trigger a retry or degradation path, and log the recovery rate.

How do I tell truncation apart from a genuine syntax error

Read the stop reason in the response metadata: max_tokens, length or incomplete means truncated, while end_turn or stop with a parse failure means real syntax damage. Without metadata, look at the tail: cut inside a quote or bracket with the outermost structure unclosed is almost always truncation.

Why is a repaired truncation more dangerous than a loud failure

A repair helper optimizes for "parses", and a closed truncation parses, so monitoring stays quiet and downstream validation passes on defaults. That is how 83 truncated reports produced only 4 errors. The helper has no notion of intent and cannot know what is missing, so truncation must be intercepted upstream.

Does raising max_tokens cost more money

No. Billing follows tokens actually generated; the cap is only a rail. A 4096 cap on a 1100-token output bills exactly like a 1500 cap. Set it from the p99 of real output plus headroom and spend your effort on preventing runaway generation instead.

References

Matching tools

Related articles

JSON Repair Guide: Salvaging Broken LLM Output Without Losing Trust

LLM responses wrapped in markdown fences, trailing commas, and Python literals will not survive a strict parser — neither will hand-edited configs missing quotes. This guide draws the real line between nine error classes a repairer fixes and four it cannot, explains why state machines beat regex, and gives you a production-ready parse chain with jsonrepair and json-repair.

Intermediate~6 min readUpdated 2026-11-08

TOON Format Guide: Shrink JSON Before Feeding It to an LLM

Feeding a uniform array to a large language model in JSON burns tokens on repeated key names and brackets. TOON declares the structure once with indented objects and tabular arrays, then streams row data — official benchmarks show ~40% fewer tokens and higher parse accuracy. This guide covers the syntax, when the savings apply, when not to convert, and how to wire up encode and decode.

Beginner~6 min readUpdated 2026-11-10

JSON Schema Contract Testing: Wire Your Validation Rules into CI

A finished schema is where decay begins, not where work ends: one typo silently widens validation, one refactor breaks every fixture. This guide gives a workable contract testing setup — positive, negative, and snapshot fixtures, schema-first linting, compile-once reuse, and four CI gates that turn contract breakage into a red build.

Advanced~8 min readUpdated 2026-11-05