Setting the file. One moment.
Skill 01 · Content Experimentation Best Practices
Subchapter 1.2
references/common-pitfalls.mdMarkdown7 KBView on GitHub
Avoid these mistakes that invalidate results or lead to wrong conclusions.
The problem: Checking results daily and stopping when you see significance.
Why it’s wrong: Statistical significance fluctuates. At any point during a test, you might see “significance” that disappears with more data. This is called the “peeking problem” or “repeated significance testing.”
The fix:
The problem: Running tests without enough traffic to detect realistic effect sizes.
Why it’s wrong: You’ll conclude “no difference” when there actually is one—you just couldn’t detect it.
The fix:
The problem: Testing many variants or metrics and celebrating any that reach significance.
Why it’s wrong: With 20 metrics, you expect 1 false positive at 95% confidence—by chance alone.
The fix:
The problem: Only looking at aggregate results.
Why it’s wrong: Simpson’s Paradox—overall winner might be loser for your key segments.
The fix:
The problem: Changing headline, image, CTA, and layout simultaneously.
Why it’s wrong: You won’t know which change caused the result. And each variable multiplies required sample size.
The fix:
The problem: “Let’s see if this new design is better.”
Why it’s wrong: Without a hypothesis, you can’t learn WHY something worked (or didn’t).
The fix:
The problem: Changing the control during the test, or not having one.
Why it’s wrong: You need a stable baseline to compare against.
The fix:
The problem: Running a test during a sale, holiday, or major event.
Why it’s wrong: External factors affect both variants differently, contaminating results.
The fix:
The problem: Testing on a non-representative sample (e.g., only logged-in users).
Why it’s wrong: Results won’t generalize to your full audience.
The fix:
The problem: Variants don’t render correctly, tracking fires incorrectly, assignment is biased.
Why it’s wrong: You’re not testing what you think you’re testing.
The fix:
The problem: Implementing a change because it was “statistically significant” even though the effect was tiny.
Why it’s wrong: Statistical significance ≠ practical significance. A 0.01% improvement isn’t worth the complexity.
The fix:
The problem: Only reporting point estimates (“5% improvement!”).
Why it’s wrong: The true effect could be anywhere in the confidence interval.
The fix:
The problem: Running tests but not recording what you learned.
Why it’s wrong: You’ll repeat mistakes, forget context, lose institutional knowledge.
The fix:
The problem: Running experiments but ignoring results when leadership disagrees.
Why it’s wrong: Defeats the purpose of data-driven decision making.
The fix:
The problem: Running experiments on trivial changes that don’t matter.
Why it’s wrong: Wastes resources, creates testing fatigue, delays important experiments.
The fix:
The problem: The actual traffic split doesn’t match the intended split (e.g., you expect 50/50 but observe 52/48).
Why it’s wrong: SRM is a strong signal of an implementation bug — broken randomization, bot contamination, or redirect issues. Results from experiments with SRM cannot be trusted.
The fix:
The problem: Users react differently to new designs initially, and the effect fades over time.
Why it’s wrong: Short experiments may show inflated effects that don’t persist. Returning users may click more simply because something looks new.
The fix: