How to use LLM Eval Suite

LLM Eval Suite evaluates Apple Intelligence Foundation Models, which require Apple Intelligence to be available on your device.

LLM Eval Suite is a macOS app that runs structured evaluations on Apple Intelligence Foundation Models. On first launch, the app seeds two example tests — voice journaling and meeting notes — so you can see how evaluation works immediately.

The core workflow involves three concepts:

• Datasets — Collections of input samples (transcripts) with expected outputs or entities to check
• Prompt Variants — Different prompt templates and generation configurations you want to compare
• Tests — Combine a dataset + variants + judge configuration for repeatable evaluation runs

Configure your API keys in the app's Settings view. You'll need a key from at least one supported provider to use cloud-based judge services.

When you run an evaluation, LLM Eval Suite performs these steps for each sample in your dataset:

1. Merge the prompt template with the sample's input text using the {input} placeholder
2. Send the rendered prompt to your chosen generation provider
3. Send both the input and generated output to the judge provider
4. Parse the judge's JSON response for four metric scores
5. Calculate the overall score and store results

The judge evaluates your outputs. You can choose from three cloud providers or use on-device Apple Foundation Models for generation only.

Not all features are available with all providers. This table shows which features require an API key.

For a deeper look at how LLM-as-a-judge evaluation works, read our LLM-as-a-judge framework guide.

1. Select a dataset — Choose from bundled datasets (voice journaling, meeting notes) or import your own JSON dataset.
2. Choose or create prompt variants — Each variant has a prompt template and generation configuration (provider, model, temperature, max tokens).
3. Configure the judge — Select your judge provider and scoring guide. The scoring guide determines how strictly the judge evaluates outputs.
4. Run the evaluation — The app processes each sample, shows real-time progress, and stores results when complete.

Tests are designed to be repeatable. Once created, you can run the same test multiple times to track performance over iterations.

Prompt variants define how inputs are transformed into outputs. Each variant contains:

• Prompt template — A string with an {input} placeholder that gets replaced with your sample's input text
• Generation provider — Which LLM generates outputs (MiniMax, Anthropic, OpenAI, or Apple Foundation Models)
• Generation config — Temperature, max tokens, sampling mode (greedy, random top-k, or probability threshold)

Summarize this voice journal entry in 2-3 sentences:
{input}
Keep the summary factual and capture the key emotions.

Datasets are collections of samples used for evaluation. Each sample (GoldSample) contains:

• inputText — The source content (e.g., a voice transcript)
• idealOutput — Optional expected output for reference
• expectedEntities — Named entities the output should mention
• metadata — Quality classification (length, word count, clarity, speakers, complexity, specialty)

Import datasets by loading a JSON file. The app includes two bundled datasets:

• Voice Journaling — Voice journal transcripts with varying clarity and speakers
• Meeting Notes — Meeting transcripts with multiple speakers and technical content

Scoring guides define the rubric the judge uses to evaluate outputs. They contain domain-specific instructions that shape how strictly each metric is scored.

Strictness levels (lenient, balanced, strict) further adjust the judge's scoring boundaries. Choose based on how much latitude you want in scores. Choose the right scoring guide for your use case.

When building your evaluation dataset, follow our guide to building a gold dataset with representative samples and proper metadata.

The four scoring dimensions — faithfulness, actionability, completeness, and hallucination — are designed to catch the most common AI output failure modes.

After an evaluation completes, results are stored and aggregated in several ways:

• Per-sample results — Each sample shows its four individual metric scores and overall score
• Run summary — Average scores across all samples, plus progress and status
• Leaderboard — Variants ranked by overall score, tracked across all runs
• A/B comparison — Compare baseline vs champion variants to see what improved

Evaluation runs are queued. If you start a new run while one is active, it goes into the queue and runs automatically when the current one finishes. Interrupted runs are recovered on app restart.