Setting the file. One moment.
Skill 01 · Content Experimentation Best Practices
Subchapter 1.3
references/experiment-design.mdMarkdown3 KBView on GitHub
Well-designed experiments produce actionable insights. Poorly designed ones waste time and can mislead.
State what you believe and why.
Bad: “Let’s test a new headline” Good: “We believe a benefit-focused headline will increase signup rate by 10% because users are currently confused about our value proposition”
Structure: “We believe [change] will [impact metric] because [reasoning]”
Define primary and guardrail metrics.
Primary metric: The main thing you’re trying to improve (conversion rate, engagement time) Guardrail metrics: Things that shouldn’t get worse (bounce rate, page load time)
Calculate required sample size before starting.
Factors:
Use calculators like Evan Miller’s (opens in a new tab).
Run tests for full business cycles.
| Impact | Effort | Priority |
|---|---|---|
| High | Low | Do first |
| High | High | Plan carefully |
| Low | Low | Quick wins |
| Low | High | Avoid |
// Experiment variant schema
defineType({
name: 'experimentVariant',
type: 'object',
fields: [
defineField({ name: 'name', type: 'string' }),
defineField({ name: 'weight', type: 'number', description: 'Traffic allocation (0-100)' }),
defineField({ name: 'content', type: 'reference', to: [{ type: 'page' }] }),
]
})
// Experiment document
defineType({
name: 'experiment',
type: 'document',
fields: [
defineField({ name: 'name', type: 'string' }),
defineField({ name: 'hypothesis', type: 'text' }),
defineField({ name: 'status', type: 'string', options: {
list: ['draft', 'running', 'concluded']
}}),
defineField({ name: 'variants', type: 'array', of: [{ type: 'experimentVariant' }] }),
defineField({ name: 'startDate', type: 'datetime' }),
defineField({ name: 'endDate', type: 'datetime' }),
defineField({ name: 'winner', type: 'string' }),
defineField({ name: 'learnings', type: 'text' }),
]
})Statistical significance can fluctuate. Commit to your sample size.
Each variable multiplies required sample size.
Winners may differ by device, traffic source, or user type.
Future you (and your team) will thank you.