JSON Schema Contract Testing: Wire Your Validation Rules into CI

Bottom line: a schema is code, and code needs tests — positive fixtures prove it accepts what it should, negative fixtures prove it rejects what it must, snapshots watch for unintended changes, and ajv compile plus --ci snapshots stand guard in your pipeline.

When a validation rule breaks, nothing raises an error: widened acceptance goes unnoticed, tightened rejection makes the whole team suffer. A schema is not a config file — it is code nobody tests.

1. When the schema breaks, who finds out first

Start with a pitfall we hit for real. Declare a schema against the 2020-12 draft and compile it with Ajv's default package, and it throws immediately:

复制Error: no schema with key or ref "https://json-schema.org/draft/2020-12/schema"

Ajv 8's default entry only knows draft-07; for 2020-12 you must import from ajv/dist/2020. Another pitfall: with strict mode on, writing format: "email" without installing ajv-formats also throws at compile time:

复制Error: unknown format "email" ignored in schema at path "#/properties/email"

Seeing these errors locally on first compile is fine. The problem is that many projects defer compilation until a request arrives, so a broken schema waits for the first unlucky user. More insidious is the half-broken schema — required accidentally written as a string instead of an array throws schema is invalid: data/required must be array at compile time, but if that step never runs in CI, it sits in production waiting.

So the first gate does not validate data — it validates the schema itself. One line:

复制node scripts/lint-schemas.mjs

复制// scripts/lint-schemas.mjs
import { readFileSync, readdirSync } from 'node:fs';
import Ajv2020 from 'ajv/dist/2020.js';
import addFormats from 'ajv-formats';

const ajv = new Ajv2020({ allErrors: true, strict: true });
addFormats(ajv);
for (const file of readdirSync('schemas')) {
  const schema = JSON.parse(readFileSync(`schemas/${file}`, 'utf8'));
  ajv.compile(schema);
  console.log(`ok ${file}`);
}

Why not npx ajv compile: the ajv v8 package ships no CLI, so npx ajv fails with could not determine executable to run; the CLI lives in the separate ajv-cli package (5.0.0 in our run). But ajv-cli 5 supports only draft-07 and draft-2019-09 — pointed at the 2020-12 schema from section 1, it throws the very same no schema with key or ref "https://json-schema.org/draft/2020-12/schema". So a 2020-12 project lints with the same Ajv2020 entry point as the article; npx ajv-cli compile -s '**/*.schema.json' --strict is right only for draft-07 projects.

Run it like a linter and typos, deprecated keywords, and unregistered formats all surface at PR time. This is the first thing schema testing has in common with ordinary code testing: confirm it compiles.

2. The trio: positive, negative, snapshot

Schema testing has a three-piece skeleton.

Positive fixtures: samples captured from real responses that must pass. Capture them from staging, not from memory — hand-written positives tend to cover only the happy path, missing nulls, empty strings, and required-only boundaries.

Negative fixtures: for every constraint in the schema, supply at least one violating input. minimum: 0 gets -1 (reject) and 0 (accept); enum gets an off-list value; additionalProperties: false gets an extra field. Negatives matter more than positives: a schema that silently accepts bad data is a silent bug that pollutes your database all the way down.

A naming convention borrowed from open-source projects makes filenames into assertions:

复制schemas/order.schema.json
fixtures/order/valid-minimal.json        # must pass
fixtures/order/valid-full.json           # must pass
fixtures/order/invalid-type.json         # must be rejected
fixtures/order/invalid-extra-field.json  # must be rejected

Node's built-in node:test is enough — you do not need Jest:

复制import { test } from 'node:test';
import { readFileSync, readdirSync } from 'node:fs';
import assert from 'node:assert/strict';
import Ajv2020 from 'ajv/dist/2020.js';
import addFormats from 'ajv-formats';

const ajv = new Ajv2020({ allErrors: true });
addFormats(ajv);
const validate = ajv.compile(
  JSON.parse(readFileSync('schemas/order.schema.json', 'utf8'))
);

for (const f of readdirSync('fixtures/order')) {
  const data = JSON.parse(readFileSync(`fixtures/order/${f}`, 'utf8'));
  if (f.startsWith('valid-')) {
    test(`${f} should pass`, () => { assert.ok(validate(data), JSON.stringify(validate.errors)); });
  } else {
    test(`${f} should be rejected`, () => { assert.ok(!validate(data)); });
  }
}

Snapshots: serialize validate.errors to a file and compare. If someone unintentionally changes a constraint, the snapshot goes red instantly. In Jest/Vitest it is one line — expect(errors).toMatchSnapshot() — and in CI always pass --ci, which makes new snapshots fail instead of being silently created, so drift cannot sneak in.

3. Collect all errors in one pass

A commonly missed configuration when testing negatives: by default, validators stop at the first error. Your data breaks in three places, you see one, fix it, rerun, see the next. With allErrors: true (Ajv) or iter_errors (Python), you see everything at once:

复制from jsonschema import Draft202012Validator

validator = Draft202012Validator(schema)
errors = sorted(validator.iter_errors(bad), key=lambda e: list(e.absolute_path))
for e in errors:
    print(list(e.absolute_path), e.validator, e.message)

Python has a counterintuitive quirk too: which error validate() reports is not "the first" but the one best_match picks — in our test with three violations (wrong id type, off-list role, unexpected extra field), it reported only the additionalProperties error, hiding the type error on id. In CI, always use iter_errors for the full picture.

An even easier trap: format is not an assertion by default. Draft 2020-12 positions format as an annotation; Python's jsonschema does not validate it unless you pass a format_checker — in our test, "not-an-email" sailed through. Ajv requires ajv-formats and registration. If a format matters (email, date, uuid), there must be a negative fixture proving your validator configuration actually checks it.

4. Compile once — stop treating the compiler as disposable

Local measurements (Node v22.13.1, Ajv 8.17, same three-level nested schema, same violating data):

A 6000x difference. Putting the validator inside a request handler and constructing a new Ajv instance per call is the most common performance accident. Also note a widespread misconception: "ajv.validate(schema, data) recompiles every time" — in our measurement, 100k calls took 8.2ms, comparable to 11.2ms for a reused compiled function, because Ajv caches schemas internally. The real problem is not a cache miss — it is not reusing the instance. Python mirrors this: a reused Draft202012Validator is about 5000x faster than calling validate() each time (2000 iterations: 0.3ms vs 1566ms), because the latter re-checks the schema itself on every call.

5. Wire the gates into CI: four doors

Ordered from cheapest to most expensive — if an earlier gate fails, later ones need not run:

Minimal GitHub Actions setup:

复制- run: npm ci
- run: npm run lint:schemas        # Ajv2020 lint
- run: npm run test:schema         # fixture trio
- run: npx jest --ci --testPathPattern=schema   # snapshot drift guard

The breaking-change gate deserves special emphasis: tools like json-schema-diff-validator flag "removed required field" and "enum lost a value" at review time — exactly the two changes plain API-diff tools often classify as non-breaking. Removing a response field is harmless to tolerant consumers and fatal to consumers with exhaustive switches.

6. Assert on error objects, not just booleans

Negative tests that only assert validate(data) === false are not enough. For the same schema you should confirm it rejects for the right reason, or an unrelated constraint may be passing by coincidence. Ajv's error object has a famous asymmetry: when required fails, instancePath points at the parent object, and the missing field name hides in params.missingProperty. In our test, missing the id field produced:

复制{ "instancePath": "", "schemaPath": "#/required",
  "keyword": "required", "params": { "missingProperty": "id" } }

Assert "an error exists with keyword=required and missingProperty=id", not a vague errors.length > 0. For a full breakdown of error-field semantics, see JSON Schema Error Messages: Locating Ajv Errors.

7. Common pitfalls at a glance

8. Rollout checklist

1. Schema files live in version control; node scripts/lint-schemas.mjs runs in lint;
2. Every schema gets at least 1 positive fixture and 1 negative fixture per constraint; filenames are assertions;
3. Error objects are snapshotted; CI passes --ci so snapshots cannot drift automatically;
4. Validator instances are reused at module level — never new inside handlers;
5. Breaking-change diffs gate PRs; required/enum changes force review;
6. Your first PR deliberately proves the gate can fail: remove a required field, watch it go red, revert.

Choosing how strict validation should be (when to set additionalProperties: false) is a separate question — see additionalProperties Left Open Makes Schema Validation Useless. This article is about keeping written rules trustworthy over time. For fundamentals, start with JSON Schema from Basics to Engineering.

FAQ

How does schema validation relate to Pact-style contract testing?

They divide the work. Schema validation answers "is this document structurally correct" — a one-directional assertion. Consumer-driven contract testing (Pact) answers "does the provider still satisfy what consumer X specifically needs" and requires two-way verification. Small teams should start with schema gates and adopt Pact as service count grows.

How many negative tests are enough?

One negative per constraint is the floor: if the schema has constraints (required, type, minimum, enum, pattern, additionalProperties…), there should be at least that many negative fixtures, plus boundary values tested reject-and-accept. Fewer negatives than constraints means some constraint is unwatched.

What counts as a "breaking" schema change?

Judge from the consumer's viewpoint: removing a response field, removing or narrowing an enum, tightening a pattern, and adding a required request field are all breaking; adding an optional response field is usually safe — provided consumer decoders tolerate unknown keys. Automate the comparison with json-schema-diff-validator at PR time and adjudicate gray areas by hand.

Should tests hit the real API or mocks?

Both, in layers: fixture tests touch no network, run in milliseconds, and run on every PR; live validation hits staging, takes seconds, and belongs in smoke pipelines, skipped via skipif when credentials are absent. Teams that run only real APIs in CI get worn down by flaky networks and rate limits, and eventually switch the tests off.