Working with JSONL Files: Open, Convert, and Debug Bad Lines

The bottom line: every line of a JSONL file is an independent, valid JSON value, but the file as a whole is not a single JSON document. If handing the whole file to JSON.parse or json.load fails, nine times out of ten the file is fine — your reading strategy is wrong.

One record per line, no commas between lines, no outer array — JSONL trades away "whole-document structure" to gain append-only writes, streaming reads, and per-line fault isolation. All three benefits only materialize when you process the file line by line, and all three turn into parse errors the moment you try to parse the whole file at once.

1. Three names, one format: JSONL, NDJSON, JSON Lines

Beginners often assume these names imply syntax differences. They don't:

Any tool that reads one reads the other. The only criterion for picking a name is your downstream ecosystem's convention: write .jsonl when feeding Hugging Face datasets, set application/x-ndjson when serving a streaming API. A portable JSONL file satisfies three rules: UTF-8 encoding (no BOM), one complete JSON value per line, lines separated by \n. A trailing newline at the end of the file is strongly recommended, and blank lines are not valid records. The protocol details and design rationale are covered in our JSON Lines and NDJSON overview; this article is strictly about what to do once you have the file.

2. Inspecting: head first, then grep, and only then an editor

Double-clicking a 500 MB .jsonl file to open it in an editor is the most common mistake: three million lines of text plus syntax highlighting plus an undo buffer will freeze practically any editor — and none of that is JSON's fault. Parsing a 45 MB document takes Node 22 about 208 ms; rendering is what hangs. Line-oriented formats are built for line tools, because records align with physical lines:

复制# Size it up first: record count and file size, without reading content
wc -l events.jsonl && du -h events.jsonl

# See what a record looks like (confirm fields and structure)
head -n 1 events.jsonl | jq .

# Find target records directly: line tools just work — a JSONL privilege
grep '"level":"error"' events.jsonl | head -n 5
tail -n 100 events.jsonl | jq 'select(.latency_ms > 500)'

Note the last command: every line coming out of tail is still an independent JSON value, so it feeds straight into jq — no need to rebuild an array first. This approach works identically on multi-gigabyte files with near-zero memory overhead, which is the same principle behind why large JSON files slow pages down.

3. The three standard line-reading patterns

Line-by-line reading has to handle three things: skip blank lines (especially the trailing newline), report errors with line numbers, and never let one bad line kill the whole job. The standard patterns in all three languages revolve around these points.

Python, standard library only:

复制import json

with open("events.jsonl", encoding="utf-8") as f:
    for line_number, line in enumerate(f, start=1):
        if not line.strip():          # skip blank lines
            continue
        try:
            record = json.loads(line) # parse each line independently
        except json.JSONDecodeError as e:
            print(f"Line {line_number} failed: {e}")  # report, don't abort
            continue                  # isolate the bad line
        handle(record)

Node with the built-in readline module, constant memory regardless of file size:

复制import fs from "node:fs";
import readline from "node:readline";

const input = fs.createReadStream("events.jsonl", "utf8");
const lines = readline.createInterface({ input, crlfDelay: Infinity });
let lineNumber = 0;
for await (const line of lines) {
  lineNumber += 1;
  if (!line.trim()) continue;
  try {
    handle(JSON.parse(line));
  } catch (e) {
    console.error(`Line ${lineNumber}: ${e.message}`);
  }
}

Go 1.27 promotes encoding/json/v2 to a stable default, whose per-value stream decoding fits JSONL perfectly; use gojq if you need to embed jq expressions. On library choices, don't rush to install packages: Python's jsonlines reports the line number and raw text of any failing line, and orjsonl reads and writes gzip/bzip2/zstd compressed files directly; the Node ndjson package hasn't published a release since 2020 — the built-in readline is enough.

4. Converting to and from JSON arrays

Direction matters; get it backwards and every command is wrong. Remember one rule: array → JSONL is a map (each record compacted to one line), JSONL → array is a join (lines collected into a list).

复制# JSON array → JSONL: -c compacts each element to one line
jq -c '.[]' records.json > records.jsonl

# JSONL → JSON array: -s collects the whole input into one array
jq -s '.' events.jsonl > records.json

# Filter JSONL line by line: no -s needed, jq processes input streams natively
jq -c 'select(.active == true)' events.jsonl > active.jsonl

Python one-liners are equivalent: array → JSONL writes json.dumps(r, ensure_ascii=False) + "\n" per record; the reverse collects json.loads per line into a list. Each direction has one trap. Going over (array → JSONL), any real newline inside a record must be escaped by json.dumps — otherwise one record gets torn into multiple lines. Coming back (JSONL → array), don't concatenate the lines into one giant string and parse it — with millions of lines that is one unnecessary full-memory copy. Aggregations (sums, group-bys, averages) require -s, which loads everything; on multi-GB files switch to awk or a Python streaming accumulator.

5. Six kinds of bad lines and how to troubleshoot them

Almost every JSONL error lives at the line level. Validating line by line and reporting line numbers is the only reliable diagnosis. Ordered by frequency:

The diagnostic routine is always three steps: head -n 1 to confirm encoding and first-line structure; run a line-by-line script that reports every bad line number instead of stopping at the first; then copy the bad lines out individually — one line is one complete record, so inspecting it in isolation costs nothing. For batch imports the recommended policy is "quarantine, don't abort": write bad lines plus their line numbers to errors.jsonl, keep loading the good ones, then decide about re-importing based on the bad-line ratio.

6. JSONL over HTTP: line-wise consumption and backpressure

The typical NDJSON-over-HTTP pattern is a server writing continuously and a client consuming continuously. On the client, do not call response.json() — it buffers the entire body before parsing, which defeats the point of streaming. Read the body stream directly, split on \n, and carry any incomplete trailing fragment into the next iteration:

复制const reader = response.body.pipeThrough(new TextDecoderStream()).getReader();
let buffer = "";
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  buffer += value;
  const lines = buffer.split("\n");
  buffer = lines.pop();               // last chunk may be an incomplete line
  for (const line of lines) {
    if (line.trim()) handle(JSON.parse(line));
  }
}

On the server, two things are mandatory: first, when res.write() returns false, await the drain event (backpressure), otherwise memory grows without bound whenever the consumer is slower than the producer; second, disable gzip middleware on streaming routes — gzip must buffer the whole response before it can compress anything, which neutralizes NDJSON streaming entirely. Both behaviors are cross-confirmed by local testing and multiple independent English-language sources, not guesses.

7. When JSONL is the wrong choice

The negative space of format selection matters too. Three scenarios should stay with a JSON array:

One-sentence rule of thumb: if records arrive over time or get consumed line by line (logs, events, exports, training sets), use JSONL; if a small payload is produced once, consumed once, and needs whole-document semantics, use a JSON array. Framed variants (RS-delimited JSON Text Sequences) are a separate topic — see the RFC 7464 article.

FAQ

Is there any real difference between JSONL and NDJSON?

Not in engineering terms. Both mean "one complete JSON value per line, no outer wrapper, no commas between lines," and any tool supporting one supports the other. The difference is branding and ecosystem habits: .jsonl dominates data engineering and ML datasets; .ndjson and application/x-ndjson dominate streaming HTTP APIs. When integrating with an external system, follow whatever extension and media type its documentation specifies.

Can I read a .jsonl file directly with JSON.parse?

Not the whole file. JSON.parse accepts exactly one JSON text, while JSONL is many independent values arranged line by line, so whole-file parsing always fails with "Unexpected token" or "Extra data". Split into lines and parse each non-empty line individually, reporting the line number on failure.

If one line in a JSONL file is broken, is the whole file ruined?

No — per-line fault isolation is the format's core design. A bad line affects only itself: skip it, log its line number, keep reading. For batch imports, quarantine bad lines into a separate file, load the good records normally, then decide whether to fix and re-import based on the bad-line ratio.

How do I convert JSONL to Excel or CSV?

First turn it into an array or extract fields per line, then run a table conversion: compact with jq -c '.[]' data.jsonl and feed the result to a converter, or stream each line through json.loads into pandas. Watch out: flattened nested fields produce dotted column names, array fields can only be serialized as text inside a cell, and long ID columns should be declared as strings up front to avoid precision loss.