RAG vs Fine-Tuning: Which Needs Better Test Data?

The test question changes with the architecture

RAG asks whether a system retrieves the right evidence and uses it faithfully. Fine-tuning asks whether a model learns a desired behaviour without losing important generality. Both need high-quality cases, but the failure signals are different.

RAG needs evidence boundaries

RAG cases should include distractors, conflicts, missing evidence, long context, and questions whose answer depends on a precise field or section.

• Retrieval recall and ranking among similar documents.
• Citation correctness and answer grounding.
• Abstention when the evidence is missing or contradictory.
• Chunking behaviour across headings, tables, and long-form documents.

Fine-tuning needs behaviour and diversity

Fine-tuning data should represent the behaviour you want while varying wording, tone, context, and edge conditions. Keep evaluation cases separate so the model is not judged on examples it has effectively memorised.

For both workflows, document the licence, source status, labels, and release version used.

Use one catalogue across the lifecycle

A prepared case can begin as an evaluation input, become a regression fixture, and later support a carefully scoped training or fine-tuning experiment. Deniable licences its catalogue for internal development, evaluation, training, and fine-tuning within the stated product licence.