Parsing, Rendering, and Streaming Large JSON

Bottom line: measure parsing, traversal, rendering, and transport separately first, then work in this order — cut copies, expand on demand with virtual scrolling, move work into a Worker, switch to streaming, and only then change the format. Streaming is worth it only when the file does not fit in memory or when you need a small subset of its fields.

The conclusion "JSON.parse is slow" is wrong eight times out of ten. What is actually slow is what you do after parsing.

1. Measure First, Then Optimize

For the same 10MB JSON file, the time spent in different stages differs by orders of magnitude:

So measure before you act: in the browser, wrap each stage with performance.now(), or use console.time; in Node, use process.memoryUsage() to watch heap growth. No baseline, no optimization.

2. Parsing Layer: Three Strategies

2.1 Full parse (the default choice)

复制const data = JSON.parse(text);

The native implementation does the scanning and object construction in C++, so a pure JS library can hardly beat it. Use it as long as the following three conditions hold:

• The text fits entirely in memory (usually < 100MB);
• Most fields are actually needed after parsing;
• The degree to which the data is untrusted is manageable.

2.2 Streaming incremental parsing

With a SAX-style parser such as stream-json, you can scan out the target fields without building a complete object tree:

复制import { createReadStream } from 'node:fs';
import { parser } from 'stream-json';
import { pick } from 'stream-json/filters/Pick';
import { streamArray } from 'stream-json/streamers/StreamArray';

const pipeline = createReadStream('huge.json')
  .pipe(parser())
  .pipe(pick({ filter: 'items' }))
  .pipe(streamArray());

for await (const { value } of pipeline) {
  // Process one element at a time; memory is independent of element count
  handleItem(value);
}

Costs:

• Code complexity rises significantly and error handling gets messier;
• Only sequential access is possible, with no random access;
• It is unfriendly to requirements that need global aggregation.

Decision rule: streaming is only worth it when the file is too large to fit, or when you need only a very small subset of its fields.

2.3 Change the format

If the data is "a batch of records" rather than "a single document", you should have used JSON Lines from the start. A line-delimited format turns "streaming" into "reading line by line", lowering implementation complexity by an order of magnitude.

3. Traversal Layer: One Less Copy Wins Half the Battle

复制// Anti-pattern: deep copying first "for safety"
const safeCopy = structuredClone(bigData);   // Memory doubles immediately

// Good pattern: pass references directly in read-only scenarios
function renderSummary(data: OrderDocument): Summary {
  return { total: data.items.length, amount: data.total };
}

Common hidden copies:

• JSON.parse(JSON.stringify(x)) used as a "deep copy" — this is the most expensive kind;
• { ...data } copies only one level, but for a large object it still copies the reference array of all top-level keys;
• Array.prototype.map creates a new array, which is also costly when there are many elements;
• Framework reactivity proxies (Vue's reactive) proxy large objects recursively, and the overhead can be higher than parsing itself.

Practical advice: keep the "raw data" as a plain immutable object and do not wrap it in reactivity; compute and cache the derived data you need to display separately.

4. Rendering Layer: Don't Stuff the Whole Tree into the DOM

A 5MB JSON file has roughly 30,000 to 100,000 nodes; rendering them all as DOM elements will make the browser stop responding.

Three viable approaches:

4.1 Expand on demand

Render only the direct children of the root node, and compute the next level only when the node is expanded. This is the standard approach for tree viewers:

复制type TreeNode = { key: string; value: unknown; expanded: boolean };

/**
 * Returns only the currently visible nodes, avoiding building the whole tree at once.
 * @param rootNode Root node
 * @param expandedPaths Set of paths that have been expanded
 * @returns List of visible nodes
 */
function collectVisibleNodes(rootNode: TreeNode, expandedPaths: Set<string>): TreeNode[] {
  const visibleNodes: TreeNode[] = [];

  /**
   * Recursively collects the child nodes of expanded branches.
   * @param currentNode Current node
   * @param currentPath Current path
   */
  const walk = (currentNode: TreeNode, currentPath: string): void => {
    visibleNodes.push(currentNode);
    if (!expandedPaths.has(currentPath) || typeof currentNode.value !== 'object' || currentNode.value === null) {
      return;
    }
    for (const [childKey, childValue] of Object.entries(currentNode.value)) {
      walk({ key: childKey, value: childValue, expanded: false }, `${currentPath}/${childKey}`);
    }
  };

  walk(rootNode, '');
  return visibleNodes;
}

4.2 Virtual scrolling

Once expanded nodes exceed a few hundred, render only the few dozen inside the viewport and reserve space for the rest with placeholder heights. It is most effective on list-style views (tables, logs).

4.3 Show summaries only

First present "structure + type + count" (for example items: Array(12000)); only when the user opens an item do you fetch the concrete value. This trick works best for the "I just want to confirm the fields are right" scenario.

5. Main Thread: Move Work to a Worker When You Should

Parsing, formatting, and diffing anything larger than 1MB should go into a Web Worker:

复制// json.worker.ts
self.onmessage = (event: MessageEvent<{ text: string }>) => {
  try {
    const parsed = JSON.parse(event.data.text);
    self.postMessage({ ok: true, data: parsed });
  } catch (parseError) {
    self.postMessage({ ok: false, message: (parseError as Error).message });
  }
};

复制/**
 * Parses JSON inside a Worker so the main thread is not blocked.
 * @param text JSON text
 * @returns Parse result
 */
function parseInWorker(text: string): Promise<unknown> {
  return new Promise((resolvePromise, rejectPromise) => {
    const worker = new Worker(new URL('./json.worker.ts', import.meta.url), { type: 'module' });
    worker.onmessage = (event) => {
      worker.terminate();
      if (event.data.ok) {
        resolvePromise(event.data.data);
      } else {
        rejectPromise(new Error(event.data.message));
      }
    };
    worker.postMessage({ text });
  });
}

Mind the communication cost of Workers: postMessage uses structured cloning by default, and cloning a 10MB object once also takes tens of milliseconds. If all you need is to "parse it and hand it to the main thread", do the heavy work inside the Worker (parse + transform + validate) and send back a result summary.

6. Transport Layer: Size Optimization

For pages meant for humans, the correct combination is: the API sends compact, compressed JSON, and the UI prettifies it only for display. Many tool sites by default send prettified text to the frontend again, shipping 30% more bytes for nothing.

7. How to Benchmark

复制/**
 * Measures the median duration of a synchronous routine so a single jittery run does not skew the result.
 * @param runOnce Function under test
 * @param repeatCount Number of repetitions
 * @returns Median duration in milliseconds
 */
function measureMedian(runOnce: () => void, repeatCount = 9): number {
  const durations: number[] = [];
  for (let roundIndex = 0; roundIndex < repeatCount; roundIndex += 1) {
    const startedAt = performance.now();
    runOnce();
    durations.push(performance.now() - startedAt);
  }
  durations.sort((firstValue, secondValue) => firstValue - secondValue);
  return durations[Math.floor(durations.length / 2)];
}

Three rules of discipline:

1. Use a fixed sample and never generate data randomly, or the numbers are not comparable;
2. Run at least 9 times and take the median; the first run includes JIT warm-up and must be discarded;
3. Record the runtime environment (Node version, CPU), or cross-machine comparisons are meaningless.

8. Checklist

• [ ] Baseline numbers exist so before and after can be compared;
• [ ] Parsing anything larger than 1MB runs in a Worker;
• [ ] Tree views expand on demand and lists use virtual scrolling;
• [ ] No pointless deep copies or reactive wrapping;
• [ ] JSON Lines is used when the data is "a batch of records";
• [ ] The API sends compact JSON and formatting happens only for display.