jq on Large JSON Files: Why -s Blows Up Memory and How to Stream Instead
Bottom line: A few hundred MB of JSON logs killed by jq -s can be aggregated in a few MB of RAM with reduce inputs. Measured on a 220MB, 2-million-line sample: slurp peaked at 2.08GB while the streaming reducer stayed at 3.5MB. This guide explains where the memory goes, the right way to process JSONL line by line, --stream path events, and three real pitfalls including silent empty output from -n --stream.

jq parses fast, but on a few-hundred-MB file almost every out-of-memory death traces back to one flag:
-sslurps the whole input into one giant array, and the peak RSS easily reaches several times the file size.
One file, two queries, 580x memory difference
Here is a measurement. The sample is a 220MB JSONL log with 2,000,000 lines — one request record per line with timestamp, level, path, status, duration_ms and uid. Environment: jq 1.7 on Linux x86_64, peak RSS taken from /usr/bin/time -v:
| Query | Task | Time | Peak RSS |
|---|---|---|---|
jq -s 'group_by(.level) | ...' |
count by level | 10.1s | 2.08 GB (~9.5x file size) |
jq -n 'reduce inputs ...' |
same counting | 4.7s | 3.5 MB |
jq -r 'select(...) | .path' |
per-line filter | 2.6s | 3.6 MB |
Same statistic, 580 times less memory, and faster. That is not a jq bug — -s means "read everything into one array" by definition; most people just never price in that default. Third-party numbers line up: a 2.3GB JSONL write-up on Huawei Cloud Community shows the identical arc, slurp OOM, then the streaming reducer finishing in 47 seconds at just over 30MB RSS. Dratini.tech measured jq -s being killed by the OOM killer on a 16GB+ box at 2.5GB of input, while the plain pipeline finished in 48.2s at 84MB.
The takeaway first: if the file is bigger than half your available RAM, do not consider -s. Measure three things before writing anything: ls -lh for size, free -m for headroom, wc -l for line count.
Where the memory actually goes
jq -s (slurp) is documented as "read all inputs into an array". The real cost is much larger than the text: jq keeps a tree of C structs after parsing, and every object, array, string and number carries its own node and metadata. Our 220MB of text became just over 2GB of structures — roughly 9.5x inflation. group_by then sorts and buckets in memory, amplifying the peak further.
Single large documents behave the same way. Packing the same log into one 300MB top-level array and extracting with an ordinary filter:
| Query | Time | Peak RSS |
|---|---|---|
jq '.[] | select(.level=="ERROR") | .uid' (whole-document) |
4.7s | 1.85 GB |
jq --stream 'select(...)' (path events) |
13.4s | 3.4 MB |
Whole-document parsing still peaks near 1.8GB — jq builds the entire tree before the filter runs. --stream trades roughly 3x runtime for a 540x smaller memory footprint. Where exactly that inflation happens across the pipeline is modeled in Where JSON memory goes; the conclusion is what matters here.
JSONL logs: line-by-line is already streaming
If your "large file" is actually JSONL — one object per line — jq's default behavior is already streaming: every input line is parsed and emitted independently, and -s is simply wrong:
# count records per level: line-by-line, aggregate in the shell
jq -r '.level' app.jsonl | sort | uniq -c | sort -rn | head
# filter + extract: select first, so non-matching lines are dropped immediately
jq -r 'select(.level=="ERROR") | .path' app.jsonl | sort | uniq -c | sort -rn | head -5
Two habits worth forcing:
- Put
selectat the front of the pipe. jq filters as soon as a line is parsed; filtering after transformations just wastes the work. - Use
-rfor raw strings. Without it the output is a quoted JSON string, anduniq -cwill happily count"ERROR"andERRORas two different values.
For dirty lines use fromjson?: jq -r 'fromjson? | .level' silently skips lines that fail to parse, so one truncated line does not kill the whole pipeline. The format itself and when to choose it are covered in The JSONL file handbook; this article is about the commands that process it.
Global aggregation: reduce inputs keeps state bounded
Line filtering belongs in the shell, but when the aggregation must happen inside jq — counts per level, average latency per path — the key is reduce with inputs, so the state stays bounded:
# count per level: memory tracks the result set, not the file
jq -n 'reduce inputs as $x ({}; .[$x.level] = ((.[$x.level] // 0) + 1))' app.jsonl
# average duration per path: state is a fixed {count, total} pair
jq -n 'reduce inputs as $x ({}; .[$x.path].count += 1 | .[$x.path].total += $x.duration_ms)
| map_values(.total / .count)' app.jsonl
Two details:
-nmeans jq should not treat the first input as the program's initial value;inputsemits every record itself. Omit-nand the first record is swallowed as the initial state.- The reducer state must have a fixed shape — counters and accumulators only. The moment you push matching rows into an array inside the reducer, memory grows with the data again and streaming buys you nothing.
That is where the 3.5MB figure comes from: two gigabytes stream past, and what remains is one hash table.
Single large documents: --stream path events
When the input is one huge top-level document — a few-hundred-MB export with no line boundaries — use --stream. Instead of complete values it emits a sequence of "path + leaf value" events:
# events look like: path array + scalar value
$ echo '{"rows":[{"id":"a"},{"id":"b"}],"meta":{"id":"top"}}' | jq -c --stream .
["rows",0,"id"]
"a"
["rows",1,"id"]
"b"
["meta","id"]
"top"
Extracting from the event stream means matching the full path shape — depth, member names, and index positions — because checking only the last component will also catch same-named keys in nested objects:
# only rows[i].id, skipping meta.id and deeper lookalikes
jq --stream 'select(length == 2 and .[0][0] == "rows" and .[0][-1] == "id") | .[1]' huge.json
Aggregation still uses the fixed-shape reduce inputs pattern (with -n --stream you must consume events via inputs — see the pitfalls below). For reassembly — turning a matched event stream back into whole objects — combine truncate_stream with fromstream: strip the leading path segments first, then rebuild subtrees. The Huawei Cloud Community article has a worked example.
The price of --stream is speed: unfolding structure into scalar events costs about 3x the runtime of whole-document parsing on the same task. So the decision order is: JSONL first, line by line; a single large document only when you need a few leaves out of it. For the bigger picture of choosing tools by file size, see Can't open a large JSON file?.
Three real pitfalls
Pitfall one: -n --stream without inputs silently outputs nothing. Hit it on this machine: jq -n --stream 'select(...)' file.json exits 0 with no output and no error — under -n, jq never reads the input, so the select in the filter has no events to pick from. Either drop -n and use plain --stream, or pair -n --stream with reduce inputs / inputs to consume events explicitly. This class of "success but empty" failure is the hardest to debug; validate against a small sample before wiring up a pipeline.
Pitfall two: --stream only emits scalar leaves; empty objects and arrays have no value event. {"a":{}} produces only a path-close event (the length == 1 kind). A filter matching length == 2 will silently swallow empty containers — decide explicitly whether your counts include them.
Pitfall three: partial results after a mid-file parse error. A streaming pipeline may have already emitted half its output before hitting bad JSON near the end. jq 1.7 fails loudly by default — [1,] produced parse error: Expected another array element at line 1, column 4 with exit code 5 in our test — but if your shell redirects stdout into the target file, the truncated result is already on disk. The engineering fix is staging plus rename: write to a temp file and mv it over the target only after jq succeeds. For finer control, --stream-errors turns parse errors into events you can branch on and fail explicitly. JSONPrettifier's article has the full atomic-publish template.
When not to use jq: Miller and hybrid pipelines
jq is a functional tree-transformation language, great for deep restructuring. But once the job degenerates to "filter and count over gigabytes of flat logs", better-fitted tools exist. Dratini.tech's benchmark on a 2.5GB JSONL (filter + column selection):
| Tool | Time | Peak RSS |
|---|---|---|
| jq plain pipeline | 48.2s | 84 MB |
| jq -s | OOM (killed on >16GB RAM) | — |
| Miller (mlr --json) | 11.4s | 18 MB |
| ripgrep + jq hybrid | 8.1s | 32 MB |
The rule of thumb: deep nesting, structural reshaping, and ubiquity in CI — use jq. Multi-GB flat JSONL statistics, p95 percentiles, JSONL-to-TSV — one mlr --json stats1 replaces a jq script with constant memory. And when ripgrep can discard 90% of irrelevant lines before jq ever sees them, the hybrid pipeline is fastest. More routes for extreme file sizes — ijson, sharding, dedicated viewers — are in Can't open a large JSON file?
FAQ
How big can a file get before jq -s breaks?
A practical threshold: once the file exceeds half your available RAM, be suspicious. Parsed memory typically runs 3-10x the text size (9.5x in our 220MB test; third-party reports show a 4GB file OOM-killing a 16GB machine). Compare ls -lh with free -m first, and replace -s with reduce inputs when it's close.
Why does reduce inputs need -n in front?
-n tells jq not to consume the first input as the program's initial value, so inputs can emit every record itself. Without -n, the first record becomes the reduce's initial state and silently disappears from the result — a data-loss bug that never raises an error.
When should I pick --stream over plain jq?
Look at the file shape first. JSONL logs are already streaming line by line; you don't need --stream. For one large document where you only need a few leaf values, --stream trades about 3x runtime for hundreds of times less memory — worth it. Use truncate_stream + fromstream to rebuild subtrees. If the file fits in memory, plain jq is simply faster.
What about corrupt lines during streaming?
In JSONL, fromjson? skips unparseable lines so one bad record doesn't kill the pipeline. In a single large document, a mid-file parse error fails the run — publish atomically via a staging file and rename so partial output never lands in the target, or use --stream-errors for explicit control.