# AI Workflow Evaluation & Optimization Playbook

> Public edition v0.1 · 2026-07  
> Purpose: turn one AI workflow optimization into a comparable, attributable, and reproducible product experiment.

## Three principles

1. Keep the optimization order **Contract → Workflow → Prompt**.
2. Give deterministic concerns to code and structure; give semantic judgment to models.
3. Judges provide signals. A product owner remains responsible for trade-offs and the release decision.

---

## One complete campaign

### Step 0｜Define the problem

- Who is the user and what are they trying to do?
- Which outcome needs to improve?
- What must not regress?
- What is the single primary hypothesis?

### Step 1｜Freeze the contract and baseline

Record:

- input variables, types, and required state;
- output schema, empty-result semantics, and error structure;
- primary / fallback parity;
- routing, aggregation, and end nodes;
- model, parameters, plugin versions, and baseline checksum.

Only then can the team answer: “What exactly did the candidate change?”

### Step 2｜Freeze two dataset tiers

#### Tier-1 · Core Regression

- small, slow-changing, and strictly frozen;
- covers primary scenarios and high-risk boundaries;
- never used for prompt tuning;
- used before release to detect new P0 failures.

#### Tier-2 · Campaign Set

- built around the weakness targeted by this campaign;
- permits focused diagnosis and iteration;
- archived after acceptance and never presented as a universal conclusion.

Each case should contain at least:

```yaml
sample_id: ""
selection_reason: ""
scenario: ""
must_include: []
must_not_include: []
evidence_refs: []
severity_if_failed: P0 | P1 | P2
```

### Step 3｜Build the smallest candidate

Classify failures by layer:

- Contract: parsing, schema, IDs, error semantics, primary/fallback mismatch;
- Workflow: routing, variable transfer, retries, fallback, aggregation;
- Prompt: omissions, false positives, granularity, emphasis, language.

Change one primary variable per round. Write this before editing:

```markdown
## Hypothesis
If we change ______,
then ______ will improve for ______ scenarios,
without regressing ______.

## Change boundary
- Will change:
- Will not change:
- Known risks:
```

### Step 4｜Evaluate with three layers of evidence

#### Layer 1 · Contract / Hard Gate

Deterministic pass or fail:

- parseable output;
- valid schema;
- valid and relevant IDs / evidence;
- stable language, empty-result, and error semantics;
- correct primary/fallback behavior and routing.

A P0 cannot be averaged away.

#### Layer 2 · Semantic Quality

Build a 1–5 rubric around:

- faithfulness and evidence;
- key information coverage;
- structure and granularity;
- emphasis and specificity;
- readability and language.

When there is no unique reference answer, evaluate constraints and evidence—not exact wording.

#### Layer 3 · Runtime / Trace

Track separately:

- technical success and fallback rate;
- P50 / P95 latency;
- tokens and cost;
- repeated-run consistency;
- failing nodes and causes.

Do not collapse runtime and semantic quality into one score.

### Step 5｜Blind review and verdict

Anonymous A/B should:

- hide version, model, and optimization notes;
- randomize A/B order;
- delay reveal until all scores are saved;
- record A wins / B wins / ties, dimension scores, and reasons;
- actively search for cases where the baseline is better.

Final verdict:

```yaml
status: ship | iterate | reject
candidate: ""
dataset_versions: []
release_gates:
  contract_p0: pass | fail
  semantic_review: pass | fail
  runtime_regression: pass | fail
known_limits: []
deployment_status: not_deployed | deployed
approved_by: ""
approved_at: ""
```

---

## Six-SOP index

1. Contract audit and baseline freeze
2. Two-tier dataset freeze
3. Single-variable hypothesis and candidate iteration
4. Batch runs, three-layer evaluation, and regression
5. Readable reporting and evidence chain
6. Anonymous A/B and blind reveal

## Claim-boundary checklist

Every report should state:

- paths that were not tested;
- slices too small to support conclusions;
- improvements that only hold within this campaign;
- whether the candidate is test-passed, internally cleared, or production-deployed;
- accepted trade-offs and items deferred to the next campaign.

## Replace before reuse

This public edition provides structure only. Redefine:

- your domain contract;
- your P0 / P1 / P2 boundaries;
- your real samples and data permissions;
- your semantic rubric;
- your latency, cost, and release gates;
- your human owner and sign-off flow.

---

The point is not to make evaluation look complicated. It is to help a team answer one simple question:

**Why do we believe this is actually better?**