When two JSON documents differ: positional diff, id-based diff, or normalize first

Bottom line: normalize first, then compare — use jq -S or stable serialization to erase key order, indentation, and number spelling, sort or index arrays by a business key when order is meaningless, and only then turn the diff into a patch.

A JSON difference is not a text difference. Compare the files as strings and a key reorder lights up the whole page; compare arrays by index and inserting one record marks the other fifty as changed.

A patch shipped to the orders service, and all you have is two responses: fifty records before, fifty-one after, with one new record inserted in the middle. The diff tool reports fifty-one changes, a wall of red, and you still cannot tell which record actually moved.

The tool is not broken. "Comparing" means three different things on JSON, and most people reach for the least useful one by default.

Reproducing the false alarm: one insert, fifty-one changes

Thirty lines of Node 22: build fifty orders ({id, sku, qty}), then append one at the tail and insert one at the head, counting differences with index-by-index alignment.

Same "one extra record", fifty-one times the noise. The reason is mundane: when you compare by index, the new element lands at position 0 and every existing record shifts down one slot. Position 0 versus position 0, position 1 versus position 1 — no pair is equal.

Build a Map keyed by id instead and the same data reports exactly one entry: $[id=ORD-999] added.

That experiment gives you the rule that matters more than any tool recommendation: whether the array is an ordered sequence or an unordered set decides which comparison you should use.

Three ways a text diff lies about JSON

diff a.json b.json compares characters. JSON carries a data structure. That mismatch produces three reliable kinds of false positive.

1. Reordered object keys

RFC 8259 section 4 is explicit: an object is "an unordered collection of name/value pairs". These two documents are the same value:

复制{ "id": 42, "name": "Ada", "active": true }
{ "active": true, "name": "Ada", "id": 42 }

A line diff calls all three lines changed. I reversed the key order of a ten-key object, pretty-printed both, and compared line by line: all ten lines flagged, while a structural comparison says they are equal. Noise ratio 10:0.

Why does key order change at all? Switching serializers, running a formatter with sorted keys, rebuilding the object from a dictionary or hash map, moving between language runtimes — any one of those is enough to make identical data look different on disk.

2. Whitespace and indentation

{"a":1} and {\n "a": 1\n} parse to the same value. A text diff sees newlines and spaces and flags everything.

3. Different spellings of numbers and escapes

1.0 and 1, 1e2 and 100, "\u00e9" and "é" — same value after parsing, different text after serializing. This one is milder because it touches few lines, but in config reviews it is enough to make you think a number changed.

The reverse case is what text cannot see and structure can: 5 versus "5". One extra pair of quotes, but the type drifted — usually a bug rather than an update. A structural comparison reports it as modify without hesitating.

Three comparison modes, three jobs

Structural diff solves the "key order and formatting" layer. Semantic diff goes one step further and solves "array element identity". The difference is not implementation complexity; it is whether you are willing to tell the tool which field is a record's passport.

Arrays are the dividing line

Ask one question: is the order of this array itself data?

• Order is data: migration steps, search rankings, chart points, approval chains. A position change is a behaviour change, so positional comparison is correct.
• Order is meaningless: tags, permission ID sets, order line items, inventory lists. An insertion should not affect how other items are judged.

When order is meaningless, index the records by a key instead of hammering them by position:

复制// Match by id: an insert or delete reports one entry, other items stay put
function diffById(before, after, path = '$') {
  const out = [];
  const beforeMap = new Map(before.map((item) => [item.id, item]));
  const afterMap = new Map(after.map((item) => [item.id, item]));

  for (const [id, value] of afterMap) {
    if (!beforeMap.has(id)) out.push({ path: `${path}[id=${id}]`, type: 'added' });
  }
  for (const [id, value] of beforeMap) {
    if (!afterMap.has(id)) out.push({ path: `${path}[id=${id}]`, type: 'removed' });
    else if (JSON.stringify(value) !== JSON.stringify(afterMap.get(id))) {
      out.push({ path: `${path}[id=${id}]`, type: 'changed' });
    }
  }
  return out;
}

Mature libraries turn this into an option. jsondiffpatch uses an LCS to find the longest common subsequence, but object elements inside arrays are matched by position by default — the official docs emphasize that you must supply an objectHash function to match them by content. First-time users often conclude the library is dumb; in reality they never passed a key.

Choosing the key matters too. id is the safe choice. sku works only when SKUs are unique. Hashing the whole object turns "one field changed" into "one removed plus one added", which is worse than comparing by position.

The fastest terminal trick: normalize with jq

If you do not want a library and just need a look, normalize first and diff second. jq -S sorts object keys recursively and applies consistent indentation:

复制# Normalize, then compare line by line; key order and indentation noise disappear
diff -u <(jq -S . before.json) <(jq -S . after.json)

# When the array is genuinely unordered, sort by key before comparing
diff -u <(jq -S '.items |= sort_by(.id)' before.json) \
        <(jq -S '.items |= sort_by(.id)' after.json)

I ran printf '{"b":2,"a":1,"arr":[1,2]}' | jq -S . and got {"a":1,"arr":[1,2],"b":2} — keys sorted, arr untouched. Remember this: -S sorts object keys only, never array elements. Either sort arrays explicitly with sort / sort_by, or accept positional noise.

Two traps:

• jq cannot edit in place; it writes to standard output. Use process substitution (<(...)), or write a temp file, verify, then replace;
• -r strips quotes from strings, so the output may contain newlines or shell-significant characters. Quote untrusted data before piping it onward.

What this site's /diff does, and what it does not

The built-in JSON compare tool follows the same structural approach: both sides are parsed into trees and aligned recursively by path, producing path + type(add/remove/modify) + before/after. Object key order does not affect the result, and everything runs in the browser — pasted payloads are never uploaded.

Its arrays, however, are aligned by index. That is a deliberate trade-off, not a defect:

• Appending at the tail, modifying the same position, or replacing wholesale gives you clean output;
• Inserting a record at the head or in the middle marks everything after it as modify, and you have to spot that "actually only one record is new".

So when you compare order lists, the correct move is to sort both sides by id before pasting. Sorted input pins the insertion point and collapses the noise to one entry. No installation required, and it covers most day-to-day cases.

If you need "generate an applicable patch" or "detect moved elements", the in-page tool does not do that. That is the next section.

Turning a diff into a patch: two formats, do not mix them

Humans read diffs; programs apply patches. JSON has two patch formats with completely different semantics.

JSON Patch has one operation people skip: test.

复制[
  { "op": "test", "path": "/version", "value": 3 },
  { "op": "replace", "path": "/status", "value": "shipped" }
]

If test fails, the whole patch is rejected. That is optimistic concurrency built into the protocol — no extra If-Match header needed to guarantee "I only wrote because I saw version 3". For a PATCH endpoint this is worth more than any documentation convention.

Paths use JSON Pointer (RFC 6901): ~0 escapes ~, ~1 escapes /. Keys containing slashes (MIME types as keys, for instance) must be escaped, or the pointer lands somewhere else. The details live in JSON Pointer and Patch.

One practical problem remains: patches generated automatically from a diff usually address arrays by index. Indexes are fragile in production — one concurrent insert and /items/3 no longer points at the same record. Either restrict generated patches to object fields, or validate the critical fields with test before applying.

Put comparison into CI: assert values, not text

Three rules are enough:

1. Assert parsed values in tests, never the output of JSON.stringify. In Jest use toEqual; in Python assert a == b after json.loads. Key order and indentation do not participate.
2. Whitelist dynamic fields. requestId, timestamp, traceId differ every run; without exclusions the test is permanently red. propertyFilter in jsondiffpatch and exclude_paths in deepdiff exist for this.
3. Print paths on failure, not the whole document. Nobody reads 5 MB of payload in a log; $.data.items[id=ORD-042].qty: 2 → 3 locates the problem instantly.

Config drift works the same way: jq -S both the staging and production configs, sort arrays by a business key, and whatever remains is a real difference. This class of problem — the same data looking different across environments — is usually an undefined contract, which is what API contract design is about.

FAQ

Does key order count as a difference?

No. RFC 8259 defines objects as unordered collections, so {"a":1,"b":2} and {"b":2,"a":1} are the same value. Any tool that reports reordering as a change is doing text comparison, not JSON comparison.

Why does inserting one record mark every entry as changed?

Because the tool aligns arrays index by index. The new element sits at position 0, every existing record shifts down one, and no two positions match. Sort or index by a key such as id before comparing instead of comparing positionally.

Do online comparison tools upload my data?

It depends on the implementation. Server-rendered tools send the payload to a backend, where it may be logged; client-side tools parse in the browser and upload nothing. Before comparing production configs or user payloads, confirm the page computes locally, or disconnect the network and try again.

Can a diff result be used directly as a patch?

Not directly. A diff says where things differ; JSON Patch says which operations to run in order. A conversion layer sits between them. Generated patches also tend to address arrays by index, which breaks under concurrency — add test operations before applying.

How do I compare files of tens of megabytes?

Stream first (jq -c writes one record per line), or extract only the subtree you care about, such as jq '.data.items'. A full recursive comparison of tens of megabytes still fits in memory; past a hundred megabytes, shard it or index by key. For concrete memory and timing figures, see large JSON performance.

Should I sort arrays by id before comparing?

Yes. The built-in JSON compare tool aligns arrays by index, so inserting one record at the head or in the middle marks everything after it as modify; sorting both sides by id before pasting pins the insertion point and collapses the noise to one entry. The exception is when array order is itself data — migration steps, search rankings, chart points — where positional comparison is correct and sorting is wrong.