How to Build a Parser Regression Test Suite with Synthetic Documents

Start with failure modes, not file counts

A regression suite is valuable when every case protects a behaviour. Begin with a small set of parser failures: a folded header, a reordered section, a wrapped table, a different date format, or a missing optional field.

Synthetic documents make it safe to keep those cases in a repository and rerun them after a parser or prompt change.

Give each case a stable identity

Store a document ID, category, language, format, expected fields, and a short testing purpose beside the input. Keep the expected output narrow enough to survive harmless wording changes, but strict enough to catch a real regression.

• Use a fixed dataset pack for a stable local baseline.
• Use the API for controlled unseen cases and automated sampling.
• Use idempotency keys when a retry must not charge twice.
• Record credits, parser version, and result status with every run.

Add CI only after local assertions are clear

A GitHub Action can download a small licensed sample, run the parser, compare structured fields, and publish a failure summary. Keep credentials in repository secrets and never print API keys or document bodies in logs.

The same workflow can call the Deniable MCP server from an agent-assisted evaluation job, provided the key, credit limit, and output handling are explicit.

Keep the suite explainable

When a test fails, a developer should be able to see which case changed, which expected behaviour was violated, and whether the fault belongs to parsing, retrieval, classification, or business validation. A smaller explainable suite beats a large folder of opaque fixtures.