JSON Lines and NDJSON: Streaming Data

Bottom line: use a JSON array when you process a whole batch at once, and JSON Lines when you need streaming appends and bad-line isolation; if you still need to validate the overall structure, do not switch to a line-delimited format.

Once the data exceeds memory, or keeps arriving continuously, the "one big array" structure becomes the enemy.

1. Format rules

JSON Lines (.jsonl) and NDJSON (.ndjson) are practically synonyms, and the rules are short in both cases:

1. Encoding is UTF-8, without a BOM;
2. Each line holds one complete, valid JSON value;
3. Lines are separated by \n; the final line may or may not end with a newline;
4. No unescaped newline may appear within a line — a newline inside a string must be written as \n;
5. Empty lines are usually ignored, but a strict implementation should treat them as invalid.

复制{"time":"2026-10-30T10:41:45+08:00","level":"info","message":"boot"}
{"time":"2026-10-30T10:41:46+08:00","level":"warn","message":"slow query","costMs":812}
{"time":"2026-10-30T10:41:47+08:00","level":"error","message":"timeout","traceId":"a1b2"}

The subtle difference between the two: the JSON Lines website emphasizes "one value per line, and the value can be any JSON (including scalars)", while the NDJSON specification adds stricter implementation rules such as "no empty lines and no top-level scalars". In engineering, following the stricter set keeps you out of trouble.

2. Comparison with a JSON array

The conclusion is direct: whether the data is "a batch of records" or "one document" decides which format to use.

3. Reading: line-by-line streaming

In Node, the simplest approach is readline:

复制import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';

/**
 * Reads a JSON Lines file line by line and invokes a callback for each record.
 * @param filePath Path to the file
 * @param onRecord Handler for a single record
 * @returns Statistics containing the number of successful and failed lines
 */
async function readJsonLines(
  filePath: string,
  onRecord: (record: unknown, lineNumber: number) => void,
): Promise<{ okCount: number; badCount: number }> {
  const lineReader = createInterface({
    input: createReadStream(filePath, { encoding: 'utf8' }),
    crlfDelay: Infinity,
  });

  let okCount = 0;
  let badCount = 0;

  for await (const rawLine of lineReader) {
    const line = rawLine.trim();
    if (!line) {
      continue;
    }
    try {
      onRecord(JSON.parse(line), okCount + badCount + 1);
      okCount += 1;
    } catch {
      badCount += 1;
    }
  }

  return { okCount, badCount };
}

The memory usage of this code depends only on the size of a single line, not on the total file size — that is the greatest value of a line-delimited format.

3.1 When you only need a few fields, don't JSON.parse first

If each record has 50 fields and you only need 3, JSON.parse builds objects for all 50 of them. A streaming parser (stream-json, clarinet) can extract only the target fields, at the cost of higher implementation complexity. The rule of thumb: when a single record is under a few hundred KB, calling JSON.parse directly is usually still the better deal.

3.2 Chinese text and BOMs

.jsonl files generated on Windows often carry a BOM, and JSON.parse on the first line is guaranteed to fail. Strip the BOM before reading, or inspect the first three bytes while streaming the first chunk.

4. Writing: appends and atomicity

复制import { appendFile } from 'node:fs/promises';

/**
 * Appends a single JSON Lines record.
 * @param filePath Target file
 * @param record The record object to write
 * @returns A promise that resolves when the write completes
 */
async function appendRecord(filePath: string, record: unknown): Promise<void> {
  const line = `${JSON.stringify(record)}\n`;
  await appendFile(filePath, line, { encoding: 'utf8' });
}

Two things to note:

1. JSON.stringify output contains no newlines (unless a string's \n is escaped into two characters), so one line is always exactly one record;
2. When several processes append concurrently, a single write smaller than PIPE_BUF (usually 4096 bytes on Linux) is atomic; anything larger is not guaranteed. For high-concurrency scenarios, switch to "one file per process, merged after flushing to disk", or use storage with transactions.

5. Handling bad lines

This is the decision a line-delimited format forces you to make in advance:

Whichever you choose, record the line number and a fragment of the original content, otherwise you cannot locate the problem afterwards. The line number is not present in the JSON.parse error message, so you have to maintain it yourself.

6. Compression and sharding

• gzip compresses .jsonl at a consistently high ratio (field names repeat), so .jsonl.gz is recommended;
• gzip compresses in blocks and natively supports streaming decompression, so there is no need to decompress the whole file;
• Split large files into slices of roughly 1GB, with sequence numbers in the names, to make parallel processing and resumable transfer easier;
• Never preprocess .jsonl by "merging it into an array" — that throws away all of its advantages.

7. Trade-offs against other big-data formats

The advantage of JSON Lines is that any language and any tool can read and write it immediately; the disadvantage is the lack of a schema and of columnar compression. Once the data no longer fits on a single machine and you need frequent column-oriented queries, it is time to consider Parquet.

8. Checklist

• [ ] The file extension is consistent (.jsonl or .ndjson) and never mixed;
• [ ] The writer guarantees "one record per line", with newlines inside strings already escaped;
• [ ] The reader strips the BOM, skips empty lines, and maintains line numbers;
• [ ] The bad-line strategy is explicitly defined and monitored (bad lines / total lines);
• [ ] Large files are sliced by size and paired with gzip and parallel reads;
• [ ] Where a field contract exists, an additional schema is provided and validated line by line.

FAQ

How do I choose between a JSON array and JSON Lines?

Look at whether the data is a batch of records or a single document. Choose JSON Lines when records are appended and read as a stream: appending means writing one line at the end, and reading only requires splitting by line. Keep a JSON array when the whole structure must be validated in one pass, because a JSON array validates everything at once while JSON Lines can only be validated line by line.

Why does one corrupt line not break the whole file?

Each line in JSON Lines is an independent, complete JSON value, and parsing is naturally line-by-line, so corrupt data affects only that line. In a JSON array, one syntax error makes the whole file unparsable, which is exactly why line-delimited data suits logs and analytics events.

Should I convert JSON Lines into a JSON array?

You can: read line by line, call JSON.parse on each record, and collect them into an array. Doing so requires the entire content to fit in memory and gives up streaming reads and one-line appends. Preprocessing .jsonl into an array throws away all the advantages of the format, so it is only reasonable for one-off delivery where the data fits in memory.