JSON Toolkit

Reading Big JSON in Node.js: stream-json Streaming and Backpressure in Practice

Bottom line: Parsing a 186MB JSON array with readFileSync plus JSON.parse peaks at 612MB of resident memory; switching to stream-json's streamArray keeps it flat at 104MB. This guide walks through measured results for four reading strategies, explains why for await provides backpressure for free, how the data event loses it, and how to size highWaterMark and drain on the write side.

Intermediate~5 min readPublished 2026-11-17Updated 2026-11-17
Streaming a large JSON file in Node.js: streamArray pipeline and backpressure comparison

When importing a 186MB JSON file pushes your process to 600MB of memory, the file size is not the culprit — JSON.parse traded one whole blob of text for one whole object graph. Swap the reading strategy and the memory curve goes flat.

One symptom: a 186MB file with a 612MB peak

Server-side import jobs are where this problem shows up most: scheduled order syncs, log cleanup, bulk history loads. The typical implementation looks like this:

const fs = require('fs');
const raw = fs.readFileSync('./orders.json', 'utf8');
const orders = JSON.parse(raw);
for (const order of orders) await saveToDb(order);

We benchmarked exactly this pattern on Node 22.13. The sample is an object array of 480,000 order records, file size 186MB, with strings, nested objects and arrays in the fields to mimic real business shapes. Peak resident set size measured with /usr/bin/time -v: the file reads in 1.3 seconds, but Maximum resident set size hits 612MB — 3.3 times the file. With V8's default old-space limit at 1.4–2GB, a few of these jobs running concurrently is how containers end up OOM-killed.

The arithmetic is not exotic: the raw text occupies one copy in memory (about 186MB), the parsed object graph occupies another (every string key, nested object header and array carries overhead), and the parsing process itself allocates transiently on top. Stacking to 600MB is normal.

Two causes: JSON.parse trades whole text for whole graph

The contract of JSON.parse is "give me the complete text, get back the complete object graph". It requires all text to be in place before parsing starts, and it hands over the result only when parsing finishes. There is no mechanism for "here are the first 1,000 records first".

The overhead of the object graph has been measured item by item in The memory account of JSON: every string takes real space, every object key takes a slot, and array width depends on element kinds. Ten-odd MB of JSON routinely becomes three to five times larger after parsing, with the peak stacked on top.

Most import jobs, however, never need the whole graph: insert row by row, filter row by row, transform row by row — each record can be garbage collected as soon as it is handled. What the job actually needs is "parse one record, emit one record". That is precisely the definition of streaming parsing.

The fix: streamArray consumes record by record, and for await gives you backpressure

stream-json implements the parser as a Transform stream that consumes tokens, so it never waits for the whole file. When the top level is an array, streamArray() emits {key, value} the moment each element finishes parsing:

const fs = require('fs');
const { chain } = require('stream-chain');
const { parser } = require('stream-json');
const { streamArray } = require('stream-json/streamers/StreamArray');

async function importOrders(filePath) {
  const pipeline = chain([
    fs.createReadStream(filePath, { highWaterMark: 64 * 1024 }),
    parser(),
    streamArray(),
  ]);
  for await (const { key, value } of pipeline) {
    await saveToDb(value);
  }
}

On the same 186MB file and the same 480,000 records, this version peaks at 104MB of RSS, flat for the entire run. Only the current element plus parser state exists in memory at any instant; finished records are reclaimed immediately.

One easy-to-miss detail: for await does not pull the next record from the parser until saveToDb completes, so the parser stops reading the file while the database catches up. Slower database, slower reads — that is backpressure, formed naturally, with zero extra code. A dev.to write-up about ingesting a 4GB municipal transit feed uses the same chain, adding pick({filter:'data'}) for field filtering, and reports an equally flat memory profile.

When the write side fails: an unpaused data event pushes memory back up

Streaming parsing only disciplines the read side. If the write side drops backpressure, memory climbs right back. The classic failure is doing async writes inside a data event handler without caring about the writable's buffer:

// Wrong: data event + async write, backpressure completely lost
readable.on('data', async (chunk) => {
  await slowWrite(chunk); // the stream never waits; the buffer keeps growing
});

In data mode the stream keeps flowing no matter how slow slowWrite is, so everything piles up in the writable's internal buffer. Using the same 186MB sample, we piped parsed records into a slow Writable that delays 1ms on every 100th chunk. Ignoring backpressure peaked at 407MB of RSS; pausing with await once(sink, 'drain') whenever write() returned false peaked at 161MB. The 246MB difference is literally data sitting in the write buffer.

// Correct: pause when write() returns false, resume on drain
for await (const { value } of pipeline) {
  if (!sink.write(value)) {
    await once(sink, 'drain');
  }
}

Two more field notes: when assembling stream chains by hand, set highWaterMark explicitly on every stream — the default of 16 in objectMode is generous for large objects; and prefer pipeline() from stream/promises over pipe(), which takes care of error propagation and cleanup for you.

The benchmarks: peak memory for four strategies

Strategy Sample Peak RSS Duration Use case
readFileSync + JSON.parse 186MB / 480k records 612MB 1.3s small files, random access
streamArray + for await + slow DB writes (0.5ms each) same 104MB (flat) paced by the database row-by-row ingestion, transforms
streamArray + data event writes, no backpressure same 407MB ~10s what not to do
streamArray + write()/drain backpressure same 161MB ~10s piping results to disk or downstream

Environment: Node 22.13, Maximum resident set size from /usr/bin/time -v, synthetic sample with business-shaped fields, single runs rather than averaged repeats, simulated slow writes via setTimeout. These numbers are order-of-magnitude conclusions for this sample, not universal constants — measure with your own data before quoting them.

Two readings deserve attention. The first strategy uses six times the memory of the second, and the price is trading "1.3 seconds" for "however fast the database is". The third and fourth differ by 2.5x, which shows that streaming the read side while leaving the write side without backpressure only half-solves the problem.

Diagnostics: confirming backpressure actually works

Production memory issues should not be diagnosed by vibes. A Tencent Cloud write-up on backpressure troubleshooting offers a practical playbook:

  1. Temporarily log 'pause' and 'resume' events on the critical streams. If a high-throughput pipeline never pauses while moving serious volume, something is accumulating data in memory;
  2. Watch writable.writableLength — steady growth means the buffer never drains;
  3. Sample RSS with process.memoryUsage() on an interval and plot the curve, distinguishing "ramp growth" (accumulating) from "jagged plateau" (healthy reclamation).

Under for await semantics those pause and resume events must alternate; if they never fire, look for a stray data listener or a missing await.

When streaming is not the answer

Streaming is not the default. For files of a few dozen MB where you need random access by key, one JSON.parse call is simpler and cheaper. Viewing a large file in a browser is a different problem domain — how to open a large JSON file covers the viewer ladder. The boundaries for streaming parsing are clear:

  • the top level is an array (or the data can be split as JSONL), and consumption is sequential — no look-backs;
  • file size is uncontrolled or dwarfs heap headroom, and one import must not double the process footprint;
  • the consumer has a natural speed gap (database writes, downstream API calls), so you need backpressure rather than a full-speed firehose.

When the top-level shape is irregular, inspect the structure first in the JSON parser and formatter tool; the command-line equivalent is covered in jq for large JSON — jq --stream and streamArray are the same idea implemented in two runtimes.

FAQ

How much slower is stream-json than JSON.parse?

For pure parsing, JSON.parse wins: 1.3 seconds for the 186MB sample, versus roughly 7 seconds for streamArray with no write latency (token-level parsing plus per-record object construction). Streaming's benefit lives entirely on the memory side: peak drops from 612MB to 104MB. Import jobs are bottlenecked by the database anyway, so paying a few extra seconds of parsing for a 6x memory margin is usually a good trade; inside millisecond API responses it is not.

Is for await backpressure the same as manual pause/resume?

Semantically yes: for await pauses the readable while the loop body is pending, which is equivalent to calling pause() when write() returns false and resume() on drain. The difference is that for await delegates the bookkeeping to the language, so nothing gets forgotten; manual wiring loses backpressure for the whole chain the moment one spot is missed.

What if the top level is one huge object instead of an array?

stream-json ships streamObject() (emits per key) and the Pick filter (parses only a chosen subtree). If you need a handful of fields, Pick with a JSONPath shrinks the full object graph to just the required subtree. Mixed multi-document input (JSON lines, JSON sequences) is better split line by line and processed as independent JSON documents.

How do I write results as JSONL or forward them downstream?

The write side is the same backpressure question: fs.createWriteStream and http.request both require checking the write() return value and awaiting once(stream, 'drain') on false — or wiring the writable into pipeline() so Node manages it. Verify by inspecting the RSS curve after a run and confirming writableLength is not growing persistently.

What is the quickest memory monitor for an import job?

Minimal version: a timer that samples process.memoryUsage().rss every 100ms into an array, printing the max when the job ends. More precise: run /usr/bin/time -v node task.js and read Maximum resident set size directly. In containers, set the alert line at 70% of the container memory limit to leave GC headroom.

References

Matching tools

Related articles

jq on Large JSON Files: Why -s Blows Up Memory and How to Stream Instead

A few hundred MB of JSON logs killed by jq -s can be aggregated in a few MB of RAM with reduce inputs. Measured on a 220MB, 2-million-line sample: slurp peaked at 2.08GB while the streaming reducer stayed at 3.5MB. This guide explains where the memory goes, the right way to process JSONL line by line, --stream path events, and three real pitfalls including silent empty output from -n --stream.

Advanced~6 min readUpdated 2026-11-12

Large JSON File Won't Open? Pick the Right Tool by Size

A multi-megabyte JSON export often freezes text editors and online viewers, yet parsing it takes only a few hundred milliseconds. We benchmarked a 45 MB, 500,000-record sample to measure time and memory at each stage, show why line count and node count are the real cost, and provide a size-based tool decision table covering online viewing, command-line extraction, streaming, and desktop apps.

Advanced~6 min readUpdated 2026-11-07