Why Auto-Generated JSON Schema Keeps Rejecting Real Data

Bottom line: converge the required/optional boundary by hand first — feed 5–20 multi-shaped records with "all required" off — then generate the schema and add format, enum, and additionalProperties manually; a single full-field sample will always be too strict.

The generator can only see the single record you pasted in. It reads "this record has an email" as "every record must have an email", so the more complete your samples get, the more it demands.

1. A generator does structural mapping, not business understanding

Hand a chunk of JSON to a generator and it does exactly one thing: walk the structure of the values and translate each node into schema keywords. The whole mapping table is fixed:

All the trouble is in the last row. required is not derived from types; it is derived from "this key exists in this sample". When the sample is a single record, that statement is equivalent to "every key is mandatory".

And data returned by real APIs almost always has half its fields optional. That is where the conflict begins.

2. Five same-shaped records, five acceptance rates

Start with measurements. Anyone looking at the five records below would say "these are the same shape":

复制[
  {
    "id": 1,
    "name": "Ada",
    "email": "ada@example.com",
    "tags": ["a"],
    "score": 10
  },
  { "id": 2, "name": "Bob", "tags": [], "score": 7 },
  {
    "id": 3,
    "name": "Cy",
    "email": "cy@example.com",
    "tags": ["a", "b"],
    "score": 8.5
  },
  { "id": 4, "name": "Dee", "email": null, "tags": ["c"], "score": 9 },
  {
    "id": 5,
    "name": "Eve",
    "email": "eve@example.com",
    "tags": ["d"],
    "score": 6,
    "admin": true
  }
]

email appears in 3 of the 5 records, admin appears only once, and record 2 has an empty tags array.

Now generate a schema from each record individually, then use each result to validate all five records (validator: ajv 8):

None of the five schemas is correct, but the only one that accepts all the data comes from record 2 — the record that is missing email. It wins for a plain reason: keys that never appear are never written into required.

Record 4 is even more instructive: its email value is null, so the generator hard-codes {"type": "null"}, and the other four records with string emails are all rejected.

3. Why "paste the most complete record" is the worst choice

The intuition is "paste the sample with the most fields and the schema will be complete". The measurements show the exact opposite — and a third party ran a larger controlled test.

absurdtools used 3000 random "data families", 8 records per family, with each optional field present roughly half the time, and measured how much data a schema generated from samples with different key counts could accept:

The conclusion is not "a slight decline" but a complete reversal: the minimal sample produced the loosest schema in 95% of families, while the most complete sample managed it only 0.1% of the time.

The reason lies in the semantics of required, which means "all"; optionality can only be observed through absence. Pasting a record with every field tells the generator "not one of these keys may be omitted"; pasting a record with a missing field hands it the evidence that "this key may be absent".

Remember it in one sentence: the value of a sample is not completeness, but which absences it covers.

How many records are enough

The same batch of experiments also measured convergence speed: accumulating records one by one under the rule "a key stays required only if every record so far has it", the median number of records for required to converge to the true required set is 3; 6 records cover about ninety percent of families, and 12 cover all of them.

So sampling 5–10 records day to day is the best value for effort; forcing twenty or more yields very little extra.

But there is a precondition: these records must come from genuinely different situations. Five records copied from the same template stop providing any new information from the second one onward — their field sets are identical, and required cannot narrow by a single key.

4. The fix: feed several records at once and let the absences speak

Since single-record sampling is necessarily too strict, the right approach is to let the generator look at multiple records at once and converge required with the rule "required only if every record has it".

The generator on this site accepts an array directly and has an "all required" toggle. Turn it off and it decides by occurrence count: only keys carried by every record are written into required.

Still those five records — this time combine them into an array and infer together, then take items as the element contract:

required converges as soon as the fourth record is added, and does not change afterwards. email and admin are correctly placed in properties but not in required. And because record 4's email is null, multi-record inference combines it into {"type": ["string", "null"]}, so both strings and null pass — something single-record sampling can never do.

This is the same order of magnitude as absurdtools' finding that "required converges after a median of 3 records".

Multi-record inference has blind spots of its own, though: if the email in these records happens to be null in every one (say they all come from accounts with no profile filled in), the inference result becomes {"type": "null"}, and string emails are still rejected. Likewise, if every tags value is an empty array, the element type cannot be inferred at all.

So sampling has one more hard requirement: every form of every field must appear at least once. Bring one string, one number, one null, and one empty array; don't let any form be entirely absent.

Concretely:

1. Pull 5–20 responses from logs, mocks, or real requests, and be sure to include the abnormal rows (the one with empty fields, the one with null, the one with an empty array);
2. Assemble them into a JSON array and paste it into the generator;
3. Turn off "all required" and pick draft 2020-12;
4. The result is type: array plus an items; take the items section as the contract for a single record;
5. Immediately validate the original samples with this schema — 100% must pass. If even one fails, your sampling is not enough; go back and add records.

Step 5 is the cheapest self-check. If the generator cannot even accommodate the data you fed it, your sampling method is wrong, and there is no point going further.

5. Four things a generator cannot fill in — you must write them by hand

Even after feeding multiple records, the following still cannot be inferred and must be added manually:

The third row deserves separate mention: the default value of additionalProperties is "allow". That means a config file that spells timeout as timeuot passes validation and then silently does nothing at runtime. For config-style schemas, this is an item you must explicitly turn off.

format has a trap too: since draft 2019-09, format belongs to the annotation vocabulary, and by default it only annotates rather than enforces. If you added format: "email" and illegal values still pass, in eight cases out of ten the validator has not enabled validateFormats or the corresponding format plugin — it is not that the schema is wrong.

A note on draft choice while we're here: if the schema is going into OpenAPI 3.1, it must be draft 2020-12, because 3.1 embeds that version directly; older toolchains (the Swagger 2.0 ecosystem and some $ref implementations) still center on draft-07. The two behave differently for $ref sibling keywords and for array-related keywords, so don't mix them within one set of schemas.

6. For undocumented third-party APIs, reverse-engineer the contract from response samples

Integrating a third-party callback, working with a partner who writes no documentation, or maintaining a ten-year-old legacy system — all of these mean "the data is in hand, the spec is not". Here a generator is not merely a convenience; it is the fastest path to a contract.

The method is the same as above, with two extra musts:

One: sample across time and across states. Third-party fields often change with business state — a promotion adds a discount, a refund order adds a refundAt, and an error response has no data at all. Grabbing a single snapshot means seeing only one state.

Feed a few successes, a few failures, and a few empty results in together: error and data often appear mutually exclusively, and multi-record inference puts both into properties while leaving them out of required — exactly matching reality.

Two: store the schema in the repository and attach a daily check. Validate today's real responses against it, and you will learn about fields quietly disappearing or changing type before your users do.

复制// Run once a day: contract-drift alert
const drifted = records.filter((record) => !validate(record));

if (drifted.length > 0) {
  console.warn(
    `${drifted.length} responses do not match the contract; first difference:`,
    validate.errors,
  );
}

The value of this practice is not "saving the effort of writing it by hand", but turning a verbal agreement into an executable alert. A third party never tells you in advance that it changed a field, but the change will certainly show up in your logs.

One boundary to draw clearly: the reverse-engineered schema describes what the other party returns now, not what it promises to return. The former is for monitoring; only the latter belongs in a contract or API documentation.

7. After completion, make validation errors point back to specific fields

Once the schema is usable, the next concern is readable errors. An ajv error object has five fields, and the first four are the useful ones:

One asymmetry is easiest to trip over: when required fails, instancePath points at the parent object (usually the empty string, i.e. the root), not at the missing field. The name of the missing key is in params.missingProperty. Likewise, when additionalProperties fails, the extra key name is in params.additionalProperty.

So when mapping errors to "field → message", handle these two special cases first:

复制import Ajv from "ajv";

const ajv = new Ajv({ allErrors: true });
const validate = ajv.compile(schema);

/** Convert the ajv error array into { fieldPath: [message] } for form highlighting */
function groupByField(errors) {
  const grouped = {};

  for (const error of errors) {
    // For required and additionalProperties, the offending field lives in params, not in instancePath
    const field =
      error.keyword === "required"
        ? error.params.missingProperty
        : error.keyword === "additionalProperties"
          ? error.params.additionalProperty
          : error.instancePath.replace(/^\//u, "").split("/").join(".") ||
            "(root)";

    (grouped[field] ??= []).push(error.message);
  }

  return grouped;
}

if (!validate(data)) console.log(groupByField(validate.errors));

allErrors: true is not optional. Under the default configuration ajv stops at the first error, so the user fixes one field, resubmits, gets the next error, and needs three round trips just to fill in one form. If you also use ajv-errors for custom messages, without this flag the plugin will silently replace only some of the messages.

8. A reusable rollout process

String the steps above into a fixed routine and follow it every time you integrate a new API:

1. Sample: grab 5–20 real responses, deliberately picking the ones with empty fields, nulls, empty arrays, and extra fields;
2. Infer: paste them together as an array into the generator, turn off "all required", and take items;
3. Complete the semantics: add format, enum, minimum/maximum, minItems, and additionalProperties one item at a time;
4. Self-check: the original samples must pass 100%, otherwise go back to step 1 and add samples;
5. Negative proof: prepare a "should fail" input for every constraint and confirm it really fails — testing only the passing path is the same as not testing;
6. Commit: put the schema in version control, keep the failing samples as test cases, and run the tests first whenever the API changes a field.

Step 5's negative testing is the easiest to skip. Adding a key to required without ever verifying that a missing key errors means betting that the validator agrees with your mental model.

复制import { describe, expect, it } from "vitest";

describe("user schema", () => {
  it("accepts a complete record", () => {
    expect(validate({ id: 1, name: "Ada", tags: [], score: 10 })).toBe(true);
  });

  it("rejects a record without name", () => {
    expect(validate({ id: 2, tags: [], score: 7 })).toBe(false);
    // Also assert the reason, so the test doesn't pass because some other field failed
    expect(
      validate.errors?.some((e) => e.params?.missingProperty === "name"),
    ).toBe(true);
  });
});

That last assertion matters. Asserting only false can stay green even after the schema has been broken — the failure may come from an entirely different cause by then.

9. FAQ

Can the schema a generator produces go straight to production

No. It is only a snapshot of the structure. You must at minimum add formats, ranges, enums, and an unknown-field policy, and then run both the original samples and the negative samples through it. The generator solves "typing out properties by hand is tedious", not "it decides the business rules".

For the keyword system of schema authoring, $ref splitting, and composition keywords, see JSON Schema: From Basics to Production.

Why does the record with the most fields reject more data

Because required is collected from "what appeared in the sample". A full-field sample turns every optional key into a mandatory one, while most of those keys are absent in real data. Feeding multiple records, or a record that deliberately omits fields, gets you closer to the truth than the most complete sample.

Will an integer be inferred as integer

The generator on this site does not do that: both integers and floats are output as number, so a schema generated from {"price": 5} accepts 5.99. That avoids accidents like "the sample happened to be an integer, so production rejects prices with decimals". Some third-party tools do infer integer from the value, so be especially careful with money fields.

For details on number precision and type boundaries, see The six JSON data types and number precision pitfalls.

I added format but it has no effect

Since draft 2019-09, format is annotation-only by default. Either explicitly enable format assertions in the validator (ajv needs ajv-formats), or use pattern for a hard constraint. Don't assume that adding it will block anything.

required reports an error, but the field is clearly present in the data

Look at instancePath first. A required error points at the parent object, which means the missing key is not at that level — common causes are a wrong nesting level (/profile/name is not the same as name at the root), differing case (userId versus userID), or the upstream serialization dropping undefined entirely.

Before investigating, confirm that the document itself is valid JSON: comments, trailing commas, and similar issues fail at the parsing stage, before the schema ever gets a chance. JSON Parse & Format points directly at the offending line.

How many records should I sample, and which draft should I use?

Sampling 5–10 records day to day gives the best value; required converges at a median of 3 records, and 12 records cover every family. Each record must come from a genuinely different situation, and every form of every field must appear at least once. Pin the draft to 2020-12, because OpenAPI 3.1 accepts only that version.

Why is a permissive additionalProperties default more dangerous?

The default emits no keyword at all, so a misspelled key is silently accepted, passes validation, and then fails at runtime — harder to diagnose than a direct error. Config-style schemas must set "additionalProperties": false explicitly, while API schemas should first weigh the compatibility cost.

Should I use a reverse-engineered schema as a hard gate?

No. A reverse-engineered schema describes what the other party returns now, not what it promises to return, so it belongs in monitoring and drift alerts. Whatever goes into a contract or API documentation must be the promised contract.