ab-test-setup

SKILL.md

A/B Test Setup

You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.

Initial Assessment

Check for product marketing context first:
If .agents/product-marketing-context.md exists (or .claude/product-marketing-context.md in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.

Before designing a test, understand:

Test Context - What are you trying to improve? What change are you considering?

Current State - Baseline conversion rate? Current traffic volume?

Constraints - Technical complexity? Timeline? Tools available?

Core Principles
1. Start with a Hypothesis

Not just "let's see what happens"

Specific prediction of outcome

Based on reasoning or data

2. Test One Thing

Single variable per test

Otherwise you don't know what worked

3. Statistical Rigor

Pre-determine sample size

Don't peek and stop early

Commit to the methodology

4. Measure What Matters

Primary metric tied to business value

Secondary metrics for context

Guardrail metrics to prevent harm

Hypothesis Framework

Structure

Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].

Example

Weak: "Changing the button color might increase clicks."

Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."

Test Types

Type
Description
Traffic Needed

A/B
Two versions, single change
Moderate

A/B/n
Multiple variants
Higher

MVT
Multiple changes in combinations
Very high

Split URL
Different URLs for variants
Moderate

Sample Size

Quick Reference

Baseline
10% Lift
20% Lift
50% Lift

1%
150k/variant
39k/variant
6k/variant

3%
47k/variant
12k/variant
2k/variant

5%
27k/variant
7k/variant
1.2k/variant

10%
12k/variant
3k/variant
550/variant

Calculators:

Evan Miller's

Optimizely's

For detailed sample size tables and duration calculations: See references/sample-size-guide.md

Metrics Selection

Primary Metric

Single metric that matters most

Directly tied to hypothesis

What you'll use to call the test

Secondary Metrics

Support primary metric interpretation

Explain why/how the change worked

Guardrail Metrics

Things that shouldn't get worse

Stop test if significantly negative

Example: Pricing Page Test

Primary: Plan selection rate

Secondary: Time on page, plan distribution

Guardrail: Support tickets, refund rate

Designing Variants

What to Vary

Category
Examples

Headlines/Copy
Message angle, value prop, specificity, tone

Visual Design
Layout, color, images, hierarchy

CTA
Button copy, size, placement, number

Content
Information included, order, amount, social proof

Best Practices

Single, meaningful change

Bold enough to make a difference

True to the hypothesis

Traffic Allocation

Approach
Split
When to Use

Standard
50/50
Default for A/B

Conservative
90/10, 80/20
Limit risk of bad variant

Ramping
Start small, increase
Technical risk mitigation

Considerations:

Consistency: Users see same variant on return

Balanced exposure across time of day/week

Implementation

Client-Side

JavaScript modifies page after load

Quick to implement, can cause flicker

Tools: PostHog, Optimizely, VWO

Server-Side

Variant determined before render

No flicker, requires dev work

Tools: PostHog, LaunchDarkly, Split

Running the Test

Pre-Launch Checklist

 Hypothesis documented

 Primary metric defined

 Sample size calculated

 Variants implemented correctly

 Tracking verified

 QA completed on all variants

During the Test

DO:

Monitor for technical issues

Check segment quality

Document external factors

Avoid:

Peek at results and stop early

Make changes to variants

Add traffic from new sources

The Peeking Problem

Looking at results before reaching sample size and stopping early leads to false positives and wrong decisions. Pre-commit to sample size and trust the process.

Analyzing Results

Statistical Significance

95% confidence = p-value < 0.05

Means <5% chance result is random

Not a guarantee—just a threshold

Analysis Checklist

Reach sample size? If not, result is preliminary

Statistically significant? Check confidence intervals

Effect size meaningful? Compare to MDE, project impact

Secondary metrics consistent? Support the primary?

Guardrail concerns? Anything get worse?

Segment differences? Mobile vs. desktop? New vs. returning?

Interpreting Results

Result
Conclusion

Significant winner
Implement variant

Significant loser
Keep control, learn why

No significant difference
Need more traffic or bolder test

Mixed signals
Dig deeper, maybe segment

Documentation

Document every test with:

Hypothesis

Variants (with screenshots)

Results (sample, metrics, significance)

Decision and learnings

For templates: See references/test-templates.md

Growth Experimentation Program

Individual tests are valuable. A continuous experimentation program is a compounding asset. This section covers how to run experiments as an ongoing growth engine, not just one-off tests.

The Experiment Loop

1. Generate hypotheses (from data, research, competitors, customer feedback)
2. Prioritize with ICE scoring
3. Design and run the test
4. Analyze results with statistical rigor
5. Promote winners to a playbook
6. Generate new hypotheses from learnings
→ Repeat

Hypothesis Generation

Feed your experiment backlog from multiple sources:

Source
What to Look For

Analytics
Drop-off points, low-converting pages, underperforming segments

Customer research
Pain points, confusion, unmet expectations

Competitor analysis
Features, messaging, or UX patterns they use that you don't

Support tickets
Recurring questions or complaints about conversion flows

Heatmaps/recordings
Where users hesitate, rage-click, or abandon

Past experiments
"Significant loser" tests often reveal new angles to try

ICE Prioritization

Score each hypothesis 1-10 on three dimensions:

Dimension
Question

Impact
If this works, how much will it move the primary metric?

Confidence
How sure are we this will work? (Based on data, not gut.)

Ease
How fast and cheap can we ship and measure this?

ICE Score = (Impact + Confidence + Ease) / 3

Run highest-scoring experiments first. Re-score monthly as context changes.

Experiment Velocity

Track your experimentation rate as a leading indicator of growth:

Metric
Target

Experiments launched per month
4-8 for most teams

Win rate
20-30% is common for mature programs (sustained higher rates may indicate conservative hypotheses)

Average test duration
2-4 weeks

Backlog depth
20+ hypotheses queued

Cumulative lift
Compound gains from all winners

The Experiment Playbook

When a test wins, don't just implement it — document the pattern:

## [Experiment Name]
**Date**: [date]
**Hypothesis**: [the hypothesis]
**Sample size**: [n per variant]
**Result**: [winner/loser/inconclusive] — [primary metric] changed by [X%] (95% CI: [range], p=[value])
**Guardrails**: [any guardrail metrics and their outcomes]
**Segment deltas**: [notable differences by device, segment, or cohort]
**Why it worked/failed**: [analysis]
**Pattern**: [the reusable insight — e.g., "social proof near pricing CTAs increases plan selection"]
**Apply to**: [other pages/flows where this pattern might work]
**Status**: [implemented / parked / needs follow-up test]

Over time, your playbook becomes a library of proven growth patterns specific to your product and audience.

Experiment Cadence

Weekly (30 min): Review running experiments for technical issues and guardrail metrics. Don't call winners early — but do stop tests where guardrails are significantly negative.

Bi-weekly: Conclude completed experiments. Analyze results, update playbook, launch next experiment from backlog.

Monthly (1 hour): Review experiment velocity, win rate, cumulative lift. Replenish hypothesis backlog. Re-prioritize with ICE.

Quarterly: Audit the playbook. Which patterns have been applied broadly? Which winning patterns haven't been scaled yet? What areas of the funnel are under-tested?

Common Mistakes

Test Design

Testing too small a change (undetectable)

Testing too many things (can't isolate)

No clear hypothesis

Execution

Stopping early

Changing things mid-test

Not checking implementation

Analysis

Ignoring confidence intervals

Cherry-picking segments

Over-interpreting inconclusive results

Task-Specific Questions

What's your current conversion rate?

How much traffic does this page get?

What change are you considering and why?

What's the smallest improvement worth detecting?

What tools do you have for testing?

Have you tested this area before?

Related Skills

page-cro: For generating test ideas based on CRO principles

analytics-tracking: For setting up test measurement

copywriting: For creating variant copy