Building a Gold Dataset for AI Evaluation: Best Practices

A gold dataset is the foundation of meaningful AI evaluation. Without a representative, well-structured dataset, your scores won't reflect real-world performance — no matter how sophisticated your judge is.

This guide covers best practices for building and maintaining gold datasets that produce actionable evaluation results.

What is a Gold Dataset?

A gold dataset is a collection of input samples paired with known expected outputs or evaluation criteria. Unlike test sets used in traditional software, gold datasets for AI evaluation include:

• InputText — The source content (transcript, document, user query)
• IdealOutput — Reference output for comparison (optional but recommended)
• ExpectedEntities — Key entities that should appear in the output
• Metadata — Quality classification tags (length, complexity, speakers, etc.)

GoldSample Structure

In LLM Eval Suite, each sample follows this structure:

{
  "id": "sample-001",
  "inputText": "Dr. Smith: Good morning, how are you feeling? Patient:...",
  "idealOutput": "Doctor visit with Dr. Smith. Patient reports...",
  "expectedEntities": ["Dr. Smith", "blood pressure", "Friday"],
  "metadata": {
    "length": "medium",
    "wordCount": 245,
    "exchangeCount": 12,
    "clarity": "high",
    "speakers": 2,
    "complexity": "moderate",
    "specialty": "general",
    "synthetic": true
  }
}

Best Practices for Sample Collection

Synthetic vs. Real Data

Both synthetic and real data have their place:

For AI Doctor Notes, we started with 15 synthetic samples for rapid iteration, then planned to grow with real user samples after de-identification and consent.

Generating Synthetic Samples

When generating synthetic data, control these parameters:

• Domain — Medical, legal, casual, technical, etc.
• Task — Summarization, extraction, classification, etc.
• Speaker count — Single, two, multi-party
• Complexity — Simple, moderate, complex
• Noise level — Clear, moderate noise, high noise
• Question density — Few questions vs. many

Dataset File Format

Import datasets as JSON files with this structure:

{
  "name": "my-dataset",
  "description": "Voice journal samples for evaluation",
  "samples": [
    { "id": "sample-001", "inputText": "...", ... },
    { "id": "sample-002", "inputText": "...", ... }
  ]
}

Maintaining Your Dataset

Datasets need ongoing maintenance:

• Add new samples — As your feature evolves, add representative edge cases
• Remove stale samples — Outdated inputs reduce relevance
• Update metadata — Keep tags current with feature changes
• Version control — Track dataset changes alongside code

Importing into LLM Eval Suite

LLM Eval Suite supports JSON dataset import:

1. Open the Datasets view in LLM Eval Suite
2. Click "Import Dataset"
3. Select your JSON file
4. Review the samples and metadata
5. Confirm import

The app includes two bundled datasets to get started: voice journaling and meeting notes.

Common Pitfalls

Get Started

Ready to build your gold dataset? Create your first test using LLM Eval Suite and import your dataset to start evaluating. Then learn about LLM evaluation metrics to understand how outputs are scored.