macOS · Apple Intelligence
Measure Foundation Model outputs with structured scoring. Run evals, compare variants, and know what actually improved.
$19.99 · One-time purchase
Evaluations
Execute evals against Foundation Models and get structured scores on every output.
Scoring
Every output is judged on four scores. Together they tell you what's working and what needs attention.
Does the output stay true to the source? Deduct for invented events, claims, or interpretations. 1.0 = perfectly grounded.
Is it useful when you need to act later? Reward clear next steps and commitments. 1.0 = immediately useful.
Are the key points and context captured? 1.0 = nothing important left out.
How much is fabricated or unsupported? 0.0 = nothing, 1.0 = everything.
Leaderboard
Every eval run is logged. Compare prompt variants and generation configs side-by-side. See which changes moved the needle and which ones didn't — no more guessing.
Foundation Models run on your Mac. No data leaves your device.
Use OpenAI, Claude, or MiniMax. Your keys, your account.
Quality, usefulness, completeness, hallucination risk.
Log every run. Track what changed. Know what improved.
Pick your dataset, choose a judge provider, set your scoring guide.
Execute evals against Apple Intelligence Foundation Models on macOS.
Review scores, inspect individual outputs, and iterate on your prompts.
Use Cases
We used LLM Eval Suite to iterate on the AI summarization feature in AI Doctor Notes. By testing prompt variants and generation configs against Foundation Models, we found that a higher max tokens setting combined with Top K sampling produced a meaningful improvement in both completeness and hallucination control.
Read the full storyThe same workflow works for any app using Apple Intelligence Foundation Models. Text generation, classification, extraction, summarization — if the output quality matters, you can evaluate it systematically. Define your metrics, run your evals, and make decisions backed by structured scores instead of gut feel.
Judge Providers
Use any of these as your judge. Your keys stay on your machine.
Claude 
MiniMax 