JSON Toolkit

Parsing Large JSON in Python, Benchmark: stdlib, orjson, and msgspec

Bottom line: The same NDJSON star-counting script takes 754 ms with the standard json library but only 146 ms with msgspec declarative decoding, and multi-core parallelism pushes it another order of magnitude. This article benchmarks stdlib, orjson, and msgspec step by step on a 61.4 MB, 300,000-line sample, and explains exactly where each speedup comes from and when it runs out.

Advanced~6 min readPublished 2026-11-15Updated 2026-11-15
Python large JSON parsing speedup routes compared: stdlib, orjson, msgspec, and multi-core parallelism

What slows Python parsing down is never the JSON text itself — it is the cost of turning every field into a Python object. Swap the default parser, build fewer objects, use all your cores, and the same counting job goes from 0.08 GB/s to tens of GB/s.

One observation: the same code, an order of magnitude apart

The task is a common one: given a GitHub Events-style NDJSON log, find which repositories get starred the most among records whose type is WatchEvent. On my machine (Ubuntu 22.04, Python 3.12, orjson 3.13, msgspec 0.22) I generated a 61.4 MB sample file with 300,000 lines and ran four implementations three times each, keeping the best result:

Implementation Best time Scaled throughput Relative
json.loads line by line 754 ms 0.08 GB/s 1× (baseline)
orjson.loads line by line 248 ms 0.24 GB/s 3.0×
msgspec decoding to dict 201 ms 0.30 GB/s 3.8×
msgspec declarative Struct 146 ms 0.41 GB/s 5.2×

All four produced exactly the same star counts (60,108 records), so the entire difference comes from parsing itself. Third-party data backs up the same order of magnitude: in a benchmark published by World Programming on 2026-11-14, the same statistic over 7.6 GB and 2.2 million GitHub Archive events ran at 0.28 GB/s with the standard library (about 27 seconds), roughly 8× faster with msgspec, while SIMD-based fastsimdjson finished the whole dataset in 55 ms on 192 threads (138 GB/s), peaking at 277 GB/s on a single file. Note that their machine is an AWS c8a.metal-48xl with 192 cores, so those numbers do not transfer directly to a laptop — but the shape of the ladder is the same.

Where the standard library loses time

The line-by-line json.loads approach performs a "full translation" at every step: the parser scans {"id": 1, "type": "WatchEvent", ...}, allocates a fresh dict, at least seven str objects, and several ints for that one record, then hands references to your code. Three hundred thousand lines means millions of small object allocations, and you never touch most of them — the counting job only reads two fields, type and repo.name.

The second limit is the GIL. In CPython only one thread executes Python bytecode at any moment, and even if the C parsing code releases the GIL, your filtering and accumulation logic inside the loop stays serial. Spawning eight threads on the default interpreter therefore solves nothing; you need a different strategy.

First speedup tier: swap the parser

This is the highest value-for-effort step — the only change is the import:

import json
import orjson
import msgspec

dec = msgspec.json.Decoder()

# The three parsers share almost the same interface, except dumps:
# orjson.dumps(...) returns bytes, not str
for name, load in [("stdlib", json.loads), ("orjson", orjson.loads), ("msgspec", dec.decode)]:
    stars = {}
    with open("events.ndjson", "rb") as f:
        for line in f:
            e = load(line)
            if e["type"] == "WatchEvent":
                stars[e["repo"]["name"]] = stars.get(e["repo"]["name"], 0) + 1

orjson is written in Rust, produces UTF-8 bytes directly during parsing, and leans heavily on SIMD instructions; on my machine it was three times faster. There is one API trap worth remembering: orjson.dumps() returns bytes rather than str, so handle that when writing files or concatenating strings.

msgspec's default decoding is in the same league as orjson (3.8×), but its real weapon is declarative decoding — it only builds Python objects for the fields you declare:

import msgspec

class Event(msgspec.Struct):
    type: str
    repo: msgspec.Raw          # not decoded yet, raw bytes kept

dec = msgspec.json.Decoder(Event)
stars = {}
with open("events.ndjson", "rb") as f:
    for line in f:
        e = dec.decode(line)
        if e.type == "WatchEvent":
            name = msgspec.json.decode(e.repo)["name"]
            stars[name] = stars.get(name, 0) + 1

Once repo is declared as msgspec.Raw, the 240,000 non-WatchEvent lines skip building the repo object entirely and only do token scanning — that is exactly where the jump from 3.8× to 5.2× comes from. The more fields a record has and the higher the discard ratio, the bigger this step pays off.

Second speedup tier: multi-process sharding, with diminishing returns

Once a single core is saturated, the classic answer is a process pool: split the file into N byte ranges (aligned to line boundaries), give each slice to a worker process, and merge the Counters at the end. Each process returns only a small result table, so communication cost is tiny — an ideal map-reduce shape.

The World Programming benchmark shows what this path looks like in practice: a msgspec process pool peaked at 116 GB/s with 96 processes, and adding more processes made it slower — scheduling and merging overhead ate the gains. Two practical constraints apply: slices must be aligned to newline characters or the last line of each slice fails to parse; and counting jobs (which return small results) are the best case for a process pool. If you need to pull every record out and use it, inter-process transfer itself becomes the new bottleneck.

Third speedup tier: SIMD and free-threaded Python

fastsimdjson takes a different road: 8 GB/s on a single core (about 3.5× msgspec), because it builds on simdjson's structured indexing — it pre-scans punctuation to locate every token without building a full object tree; combined with fields=["/type", "/repo/name"] it reads only the declared JSON Pointer paths, and the callback receives just those two values.

It is also multi-threaded: paired with the free-threaded build of Python 3.14 (python3.14t, the official PEP 703 attempt to remove the GIL), 192 threads processed 7.6 GB in 55 ms. Across six public benchmark files of roughly 1 GB each, four exceeded 100 GB/s and the fastest, the twitter dataset, hit 277 GB/s. Keep in mind these are independent re-runs beyond vendor claims, and the sample is the most favorable scenario — "read one or two fields per record." Deeply nested documents do not reach those numbers; one of them stayed around 20 GB/s.

Selection cheat sheet and rollout order

Route Mechanism Throughput (local / reference) Best for
json (stdlib) full object building, single core 0.08 GB/s small files, zero dependencies
orjson Rust + SIMD, full object building 0.24 GB/s drop-in replacement, a few lines
msgspec dict Rust, full object building 0.30 GB/s pick one of orjson/msgspec
msgspec Struct decodes declared fields only 0.41 GB/s large arrays, high discard ratio
msgspec multi-process sliced parallelism reference 116 GB/s aggregations, small result sets
fastsimdjson structural index + threads reference 138-277 GB/s NDJSON field extraction, 3.14t

A sensible rollout order: switch to msgspec or orjson first (a few minutes of work for a free 3-5×); if that is not enough, convert the hot-path fields to a Struct; add a process pool for counting jobs; consider fastsimdjson only when you extract fields line by line from NDJSON and your runtime allows it. If the bottleneck is memory rather than speed — the file is too large to read whole — the direction changes: move to ijson streaming, or jq --stream on the command line; see Streaming large JSON files with jq and Parsing, rendering, and streaming large JSON. For line-oriented NDJSON rules, see The JSONL file handbook.

Measurement discipline and three pitfalls

First, fix your timing methodology: in this article every implementation runs three times and keeps the best result (to filter scheduling jitter), and the file is fully generated before timing starts. When you cite third-party numbers, state the machine spec — never compare a 192-core 138 GB/s figure against your laptop's single core.

Second, cross-validate correctness: all four implementations counted 60,108 stars. Any "speedup" that changes the count has changed parsing semantics (duplicate key handling, big integer precision), not made things faster. Align results before you align speeds.

Third, three common traps: orjson.dumps() returns bytes, and mixing it with str silently produces dirty data with a b'...' prefix; process-pool slices must be aligned to line boundaries — after f.seek(offset), discard the partial line first; big integers beyond 2^53 are safe in every Python parser, but if the result crosses over to a JavaScript side, precision trouble is waiting there instead.

FAQ

Should I pick orjson or msgspec?

If you only want a faster loads, both deliver 3× or more — pick based on maintenance activity and team familiarity. Choose msgspec when you need partial decoding (msgspec.Struct + Raw), type validation, or TypedDict/dataclass output; its declarative decoding adds an independent layer of gains on large arrays.

Is multi-process always faster than one core?

No. Splitting, spawning processes, and merging all carry fixed overhead, and with a file of a few dozen MB the trade is usually a loss. Jobs that return large results (you need every record) lose their gains to transfer cost. As a rule of thumb, swap libraries on a single core first, and parallelize only at GB scale on pure counting jobs.

Why doesn't my orjson benchmark show 3×?

Three usual reasons: the sample is too small (parsing time is drowned by file reading and loop overhead); the real bottleneck is network or disk, not parsing; or the data is mostly long strings, so object building is a minor share. Use a bigger sample, wrap only the parsing loop in time.perf_counter, and only then draw conclusions.

How does ijson relate to these libraries?

ijson is a streaming iterator — it trades throughput for constant memory and suits files far larger than RAM; orjson/msgspec chase throughput but build every record into objects. The two do not conflict: for GB-scale NDJSON, read line by line and decode fast with msgspec; reserve ijson or jq --stream for single JSON documents that are huge on their own.