---
title: "Outbound AI eval set: score drafts on won and lost threads"
description: "Build an outbound AI eval set from your own won and lost threads. Rubric, scoring, and pipeline for testing AI drafts before they ship."
date: "2026-08-22"
tags: "ai outbound, ai sdr, cold email, quality control"
readTime: "16 min read"
slug: "outbound-ai-eval-set"
canonical: "https://firstsales.io/blog/outbound-ai-eval-set/"
---

# Build an eval set for outbound AI

**TL;DR:** Most teams judge AI drafts by reading a handful and going with a gut feeling. That does not scale and it does not catch regressions. Pull 100 to 200 of your own won and lost threads, score them against a fixed rubric, and you have a real eval set that tells you exactly when a prompt or model change made your outbound worse.

---

## Table of contents

- [What an eval set actually is for cold email](#what-an-eval-set-actually-is-for-cold-email)
- [Why reading a few drafts is not evidence](#why-reading-a-few-drafts-is-not-evidence)
- [Start with the corpus you already have](#start-with-the-corpus-you-already-have)
- [Sorting won and lost threads into the eval set](#sorting-won-and-lost-threads-into-the-eval-set)
- [A scoring rubric that turns judgment into numbers](#a-scoring-rubric-that-turns-judgment-into-numbers)
- [Gut check vs scored eval set](#gut-check-vs-scored-eval-set)
- [The eval pipeline, end to end](#the-eval-pipeline-end-to-end)
- [What to eval on: the failure modes that matter](#what-to-eval-on-the-failure-modes-that-matter)
- [Running the eval before every prompt or model change](#running-the-eval-before-every-prompt-or-model-change)
- [How often to refresh the eval set](#how-often-to-refresh-the-eval-set)
- [What is overrated in AI email evals](#what-is-overrated-in-ai-email-evals)
- [Where a review step fits without slowing the team down](#where-a-review-step-fits-without-slowing-the-team-down)
- [FAQ](#faq)
- [Conclusion](#conclusion)

Cold email reply rates fell from about 5.1% in 2024 to roughly 3.43% in 2026 across the platform-wide average.

That decline lines up with the year AI drafting tools went mainstream.

Correlation is not proof, but the pattern is consistent with what teams report: AI drafts more emails, faster, and a chunk of them are worse than what a trained rep would have written by hand.

The fix is not banning AI drafting.

It is testing it the way engineering teams test code, against a fixed set of real cases, before every change ships.

## What an eval set actually is for cold email

An eval set is a fixed batch of real inputs and expected outputs that you run a model or prompt against every time you change something.

In software, that means unit tests.

In outbound, it means a curated set of prospect profiles paired with the emails your best reps actually sent them, scored against a rubric.

The point is not to grade every email that goes out.

It is to catch regressions before they hit a live campaign, the same way a test suite catches a broken function before it reaches production.

Without one, every prompt tweak is a guess validated by whoever happens to skim the next day's sends.

## Why reading a few drafts is not evidence

Most teams "test" a new AI prompt by generating ten emails and reading them.

If they sound fine, the prompt ships.

That method has an obvious problem: five people can read the same ten emails and reach five different verdicts, because there is no shared rubric behind the read.

It also has a subtler problem. Ten emails is not a sample size, it is an anecdote, and anecdotes miss the failure modes that only show up on the hard 20% of prospects: the ones with thin research signal, unusual titles, or a company that looks like three different companies depending on which page you crawl.

An eval set with 100+ cases surfaces those edge cases because they are already sitting in your CRM from real campaigns.

You do not need to imagine them.

## Start with the corpus you already have

You do not need synthetic data to start.

Every team running outbound already has the two things that matter most: threads that closed and threads that went nowhere.

Pull the last 90 to 180 days of outbound from your CRM or your cold email sending tool.

Filter for threads with a clear outcome: booked meeting, closed deal, explicit "not interested," or full silence after 4+ touches with no bounce.

Discard threads with ambiguous outcomes, a reply that went unanswered internally, a meeting that never happened, a deal still open six months later.

Ambiguity in the label poisons the eval set faster than noise in the email text does.

## Sorting won and lost threads into the eval set

Once you have a clean pool of outcomes, split it into three buckets.

Won: the first outreach email in a thread that led to a booked meeting or a closed deal.

Lost: the first outreach email in a thread that got an explicit no, a rude reply, or was flagged as spam by the prospect.

No signal: a thread that got silence, which teaches you less about email quality and more about targeting or timing.

Weight your eval set toward won and lost, roughly 40% won, 40% lost, 20% no signal for context.

That ratio keeps the rubric anchored to outcomes you can actually explain, instead of drowning in the ambiguous middle where most emails land.

![Won, lost, and no-signal buckets sorting outbound threads for an eval set](/images/blog/outbound-ai-eval-set/inline-1.webp)

A word of caution here. A won thread is not automatically a great email.

Some deals close despite a mediocre opener because the timing or the account was already primed to buy, covered in our piece on [buying signals for cold email](/blog/buying-signals-for-cold-email).

Read every "won" case before trusting it as a positive example, and drop the ones where the win clearly came from something other than the email itself.

## A scoring rubric that turns judgment into numbers

A rubric converts "this feels off" into a repeatable score. Build it around five to seven dimensions, each scored 0 to 2.

**Relevance.** Does the email reference something specific and true about this prospect, not a generic industry line that could apply to 500 other companies.

**Claim accuracy.** Every fact in the email traces back to a real source. This is the dimension where [AI slop](/blog/ai-slop-cold-email) shows up first, a fabricated detail about funding or headcount that sounds plausible and is wrong.

**Tone match.** Does the email read like a human wrote it to this specific person, or does it read like a template with a name variable swapped in.

**Ask clarity.** Is there one clear next step, not three vague options buried in the last paragraph.

**Length discipline.** Does the email respect the reader's time, generally under 120 words for a cold opener.

**Compliance.** Does the email avoid claims that would fail a legal review, covered in more depth in our [compliance review for AI email](/blog/ai-generated-email-compliance-review) piece.

Score every email in your eval set on each dimension. A composite score below a threshold, say 8 out of 12, gets flagged as a fail.

Track the fail rate as your headline eval metric, not the average score, because averages hide the worst 10% that actually damage domain reputation.

## Gut check vs scored eval set

| Dimension | ✓ Scored eval set | ✗ Reading a few drafts |
|---|---|---|
| Repeatable | Same rubric, same score every run | Depends who reads it that day |
| Catches regressions | Flags a drop in fail rate immediately | Regression ships, gets noticed weeks later |
| Sample size | 100 to 200 real cases | Usually 5 to 15 |
| Covers edge cases | Includes hard prospects by design | Skews toward easy, generic cases |
| Traceable | Score tied to a specific email and rubric line | Verdict is a vibe, hard to defend or dispute |
| Cost to run | One script, minutes per prompt change | A person's attention every single time |
| Improves over time | New failures get added as new test cases | Same blind spots persist indefinitely |

## The eval pipeline, end to end

```mermaid
graph TD
    A[Pull 90-180 days of outbound from CRM] --> B[Filter to clear-outcome threads]
    B --> C[Label: won, lost, no signal]
    C --> D[Score each email on the rubric, 0-2 per dimension]
    D --> E[Set the fail threshold, e.g. below 8/12]
    E --> F[Store as fixed eval set, 100-200 cases]
    F --> G[Run new prompt or model against the same cases]
    G --> H{Fail rate vs baseline?}
    H -->|Higher| I[Block the change, investigate]
    H -->|Same or lower| J[Ship the change]
    I --> K[Add the failing case as a new test]
    J --> K
```

The loop closes on itself. Every failure you catch becomes a permanent test case, so the eval set gets harder to fool over time instead of staying static.

## What to eval on: the failure modes that matter

Not every quality problem is worth a dimension in your rubric. Focus on the failures that actually cost meetings or burn domains.

Fabricated personalization is the most damaging one. An AI draft that invents a "recent funding round" that never happened does not just fail to land, it tells the prospect your entire research process is fake.

Generic openers that could go to anyone are the second most common failure, and they are why signal-based emails post reply rates of 5-18% while generic sends sit at 1-3%.

Misclassified replies are a third category worth its own eval, since an AI agent that reads "not now, check back in Q2" as a hard no burns a lead that was actually still warm, a problem we cover directly in our piece on [AI reply misclassification](/blog/ai-reply-misclassification).

Over-long emails are a fourth failure mode. AI models left unconstrained tend to over-explain, and a 300-word cold email gets skimmed for three seconds and archived.

Build your rubric around these four before adding anything more exotic. A tight rubric that catches real failures beats a sprawling one that scores things nobody acts on.

![Scorecard showing six rubric dimensions scored against a pass threshold](/images/blog/outbound-ai-eval-set/inline-2.webp)

## Running the eval before every prompt or model change

The eval set only pays off if you actually run it before shipping a change, not after a campaign already went out.

Treat every prompt edit, every model swap, and every new data source the same way you would treat a code change: run the eval, compare the fail rate to your last known baseline, and only ship if it holds or improves.

This is the same discipline behind [training AI on won deals](/blog/train-ai-on-won-deals), except the eval set is the mechanism that proves the training actually worked instead of assuming it did.

Teams that skip this step find out their prompt regressed only after a week of live sends underperformed, by which point the damage is already in the campaign metrics and possibly in domain reputation.

A five-minute eval run before shipping catches what a week of live data would have told you anyway, at a fraction of the cost.

## How often to refresh the eval set

Static eval sets go stale. A rubric built on threads from January will not catch failure patterns that show up after your ICP shifts or after a new competitor changes how prospects respond to your angle.

Refresh the corpus every 60 to 90 days with new won and lost threads.

Keep the old cases in the set rather than replacing them wholesale, since a regression on last quarter's hard cases is just as real as one on this quarter's.

Add a case immediately whenever a live send fails in a new way, a hallucinated fact, a tone mismatch, a broken personalization token. That failure becomes tomorrow's automated check instead of tomorrow's repeat mistake.

## What is overrated in AI email evals

Automated "quality scores" from a second LLM judging the first LLM's output get pitched constantly, and they are overrated as a primary signal.

An LLM judge is fast and cheap, but it has no access to whether the prospect actually replied, and it can be fooled by the same fluent, confident, wrong output that fools a skimming human reviewer.

Use an LLM judge as a first-pass filter to triage volume, never as the final word on whether a draft ships.

The rubric anchored to real outcomes, won or lost, is the ground truth. The LLM judge is a shortcut for pre-screening obvious failures before a human looks at the harder cases.

Sample size worship is the second overrated idea. Teams chase 1,000-case eval sets when 150 well-labeled cases, evenly split across won, lost, and hard edge cases, catch the same regressions.

Bigger sets take longer to score and refresh. A smaller, sharply labeled set that gets maintained beats a bloated one that goes stale because nobody wants to touch it.

## Where a review step fits without slowing the team down

An eval set answers "did this prompt get worse." It does not replace a human checking the specific email before it hits a specific prospect's inbox, which is the argument at the center of [human in the loop cold email](/blog/human-in-the-loop-cold-email).

The two are complementary, not redundant. The eval set catches systemic regressions across hundreds of cases in minutes. The human review catches the one-off mistake in a single high-value send that a rubric average would smooth over.

FirstSales runs AI drafts through this exact pattern: the model writes, the eval-style checks run automatically against known failure modes, and a human still approves before anything sends.

![FirstSales AI draft approval screen showing a queued draft awaiting human review](/images/blog/shared/app-ai-draft-approval.webp)

That approval screen is also where teams catch the gap between an AI draft that reads fine and one that would actually get a reply, a distinction we cover in more depth in [human review rate for AI email](/blog/human-review-rate-ai-email) and in [how prospects spot AI-written emails](/blog/how-prospects-spot-ai-written-emails).

Teams that skip human approval entirely tend to be the same teams showing up in the 40-60% of AI SDR pilots that get paused or shut down within 90 days. An eval set reduces how often that approval step catches something bad, it does not eliminate the need for the step.

The hybrid model, AI drafting with a human checkpoint, is also the one behind the reported 2.8x pipeline gap over manual-only teams, and an eval set is the piece that keeps that model's output quality from drifting as volume scales.

## FAQ

### What is an eval set for outbound AI

An eval set is a fixed collection of real prospect cases, paired with scored outcomes, that you run every AI prompt or model change against before shipping it to live campaigns.

### How many emails do I need to build a useful eval set

100 to 200 cases is enough if they are well labeled with clear won, lost, or no-signal outcomes. Quality of labeling matters more than raw volume.

### Where do I get the emails for an eval set

Pull them from your own CRM or sending tool. Filter for threads with a clear outcome over the last 90 to 180 days, rather than writing synthetic examples.

### Should I include emails that got silence as "lost"

No. Treat silence as a separate no-signal bucket. Silence usually reflects targeting or timing, not email quality, and mixing it into "lost" muddies the rubric.

### What dimensions should the scoring rubric cover

Relevance, claim accuracy, tone match, ask clarity, length discipline, and compliance are the six that catch most real-world failures.

### How do I score each dimension

A simple 0 to 2 scale per dimension works well. Zero means the email fails that dimension outright, one means it is borderline, two means it clearly meets the bar.

### What counts as a failing score

Set a composite threshold, for example 8 out of 12 across six dimensions, and flag anything below it as a fail worth investigating before shipping.

### Can an LLM score the eval set automatically

An LLM can do a first-pass triage to flag likely failures faster than a human reading every case. It should not be the only judge, since it cannot see whether the prospect actually replied.

### How often should I refresh the eval set

Every 60 to 90 days with new won and lost threads. Keep older cases rather than discarding them so regressions on past patterns still get caught.

### What is the single biggest failure mode to test for

Fabricated personalization, an AI draft inventing a detail about the prospect that is not true. It is the fastest way to lose trust and burn domain reputation.

### Does a good eval score guarantee the email will get a reply

No. An eval score measures whether the email meets your quality bar, not whether the prospect will respond. Reply rate depends on timing, targeting, and offer as much as email quality.

### How is this different from A/B testing subject lines

A/B testing compares two live variants against real send data over days or weeks. An eval set is a pre-send gate that runs in minutes against a fixed historical set, before anything reaches a real inbox.

### Should every team member score the eval set the same way

Yes, that is the point of the rubric. If two reviewers score the same email differently, the rubric definitions need tightening, not the reviewers' judgment.

### Does this replace human review before sending

No. The eval set catches systemic regressions across many cases. A human review still catches the specific mistake in an individual send that a rubric average can miss.

### What tools do I need to build this

A spreadsheet or a simple script is enough to start. You do not need a dedicated eval platform until the set grows past a few hundred cases and manual scoring becomes the bottleneck.

### How do I know if my current AI drafting prompt is actually good

Run it against your eval set and check the fail rate. Compare that fail rate to your previous prompt version's fail rate on the same cases.

### What if I do not have enough won threads to build a rubric

Start with lost and no-signal threads to define what does not work, then add won cases as they accumulate. A rubric anchored to failures alone is still more useful than no rubric.

### Can I use a competitor's public cold emails in my eval set

Only as reference material for calibrating your rubric, not as scored cases, since you have no outcome data tied to their sends.

### How does an eval set help with AI SDR pilots specifically

It gives a pilot team an objective way to answer "did the AI get better or worse this week," instead of relying on a manager's impression of a handful of emails.

### Does eval scoring slow down how fast I can ship prompt changes

It adds minutes, not days, once the pipeline is scripted. The time it saves by catching a bad prompt before a live campaign far outweighs the setup cost.

## Conclusion

An eval set turns "does this AI draft feel right" into a number you can track, compare, and defend.

Build it from threads you already have, score it against a rubric anchored to real outcomes, and run it before every prompt or model change ships to a live campaign.

That single habit is the difference between catching a regression in five minutes and finding out about it three weeks into a underperforming campaign.