Interpretation

• C1 — Peeking / early stopping
• C2 — Low-volume variance (looks broken but isn’t)
• C3 — A/A test showing significance
• C4 — Multiple comparisons (no correction across variants or metrics)
• C5 — Bayesian interpretation traps
• C6 — Frequentist interpretation traps
• C7 — Bayesian vs Frequentist confusion (overlapping intervals, p-values)
• C8 — Inconclusive but trending — when is it ok to ship?
• C9 — “Significance reached” notification is not a green light to ship
• C10 — Ship-variant default does not consider any metric result
• C11 — External calculator disagrees with PostHog

C1 — Peeking / early stopping [HIGH]

Watching results live and ending the experiment the moment it looks significant inflates false positives — you’re giving randomness more chances to look significant.

In Bayesian: PostHog applies a minimum-sample-size guard before analysis proceeds — a low per-variant floor plus a proportion-validity rule of np > 5 and n(1-p) > 5 for funnel/proportion metrics (legacy stats module: 100 exposures per variant via FF_DISTRIBUTION_THRESHOLD). Early swings within that band are still noise — in the early days of the experiment, significance can flip back and forth a lot.

Recommend:

• Predetermine duration before launching. Use the running-time calculator on the experiment.
• For the duration calculator: shows “Pending” until at least 1 day and 100 exposures.
• Frequentist: PostHog uses α=0.05 by default → a single metric has ~5% chance of false-positive significance even when nothing changed.
• Don’t treat 0.05 as a hard cliff. It’s a convention, not a meaningful threshold by itself — results just below and just above are close to equivalent in evidence.

C2 — Low-volume variance (looks broken but isn’t) [MEDIUM]

Symptom: few hundred or fewer exposures per variant; the visible split looks badly off (a roughly 2-to-1 skew at a few dozen exposures is well within normal noise).

Mechanism: With low samples per variant (rule of thumb: under a thousand), the visible split can swing widely from the configured ratio — deterministic-hash variance is large at small samples. PostHog’s calculations account for this; the visible ratio is not a bug.

Funnel/proportion-specific validity gates. Beyond the per-variant exposure floor, funnel metrics also need the normal approximation to hold:

• At least 5 conversions per variant.
• n * p ≥ 5 and n * (1 - p) ≥ 5, where p is the conversion rate.

If a variant has very few converters (or, symmetrically, almost everyone converted), the test will refuse to report — not a bug. The fix is the same: more exposures, or accept that the result isn’t ready.

Recommend: wait. Run longer or increase rollout. Don’t read estimates before the running-time calculator threshold (≥1 day and ≥100 exposures).

C3 — A/A test showing significance [MEDIUM]

A/A tests should almost never show significance. If the user reports their A/A test is showing a significant difference, work through:

1. Which stats module is the experiment on? Experiments created before January 2025 may be on the legacy Bayesian module. The new module (rolled out January 2025) corrected several methodological issues that produced over-significant A/A tests in the legacy module — its A/A false-positive rate is much closer to the expected α.
2. Is it actually random chance? Even with a correct methodology, a small share of metric-variant pairs in an A/A test will look significant by chance (this is the false-positive rate, around α). With multiple metrics × multiple variant pairs, expect some to flicker significant. C4 below.
3. Is it actually different exposure handling? If multiple_variant_handling = "exclude" and the A/A flag is producing $multiple users (from identity fragmentation, A3 in bias-and-skew.md), the asymmetric exclusion can produce real differences between two arms that should be identical.
4. Is the implementation correct? A large multiple-fold gap between equal-sized variants is extremely unlikely to be random — instrumentation is the more likely cause. A specific shape worth checking: data-warehouse-source metrics where per-user exposures are joined to a per-group warehouse table (ExperimentDataWarehouseNode with events_join_key: $group_<n> on the exposure side and data_warehouse_join_key on a group-keyed metric table). The LEFT JOIN duplicates each per-group row by the number of exposed users in that group, so a sum metric over-counts proportional to per-group user count. If user counts are balanced but per-group user counts aren’t, the sum can swing 5–30% even on a true A/A — and Bayesian reads that as significant under the i.i.d. assumption. Detect: read the generated clickhouse_sql from experiment-results-get, look for an exposures CTE joined per-user to a metric table where the metric is group-aggregated upstream. Sanity check: re-aggregate the warehouse table by org/group once (deduped) and compare to the per-user sum; a large gap confirms repeated-row inflation.

Recommend: if conditions 1–3 don’t explain the result, investigate instrumentation rather than assuming the methodology is wrong.

C4 — Multiple comparisons (no correction across variants or metrics) [HIGH]

PostHog does not apply multiple-comparisons correction:

• Across variants — each test variant is compared to control independently
• Across metrics — each metric is tested independently

So with many metrics or many variants, the chance of some spurious significance grows. Concrete math at α=0.05 (the default): with 5 independent metrics, the chance of at least one false-positive is ~23%; with 10 metrics, ~40%. (Confidence level is configurable — see C6.)

Recommend:

• Define a small set of planned, hypothesis-driven metrics up front.
• Treat results as a pattern across planned metrics, not a single “gotcha” significant metric.
• Add guardrail metrics as secondary, not primary.
• Be especially wary of metrics added after seeing data — that’s p-hacking. See mid-run-changes.md.

C5 — Bayesian interpretation traps [HIGH]

PostHog defaults to Bayesian. Common misreads:

• “96% chance to win” is about direction (test is better than control), not the magnitude of the lift. Read the credible interval alongside it.
• Don’t ship the moment chance-to-win flips green — the minimum-sample guard means early flips are within the noise band.
• Non-informative priors. PostHog uses non-informative priors (mean 0, large variance). Early swings aren’t the prior pushing things around — they’re the data being sparse.
• Legacy methodology (pre-2025 experiments). Experiments created before January 2025 may use the legacy methodology (different multivariate semantics, different significance gates). If a user is reading results from an experiment in that window and the numbers look different than expected, see PostHog’s legacy-methodology docs page (opens in a new tab).

C6 — Frequentist interpretation traps [HIGH]

PostHog has Frequentist support (rolled out June 2025). Set in stats_config. Quick rules:

• 95% CI doesn’t cross 0 → significant vs control. CI crosses 0 → not significant.
• PostHog uses Welch’s t-test as the default — it handles unequal variance between groups, unlike Student’s t-test (which assumes equal variance).
• α = 0.05 by default → ~5% chance of false-positive on a single metric.
• Confidence level is configurable per team (and per experiment). Valid values are 0.90, 0.95, 0.99 — set via default_experiment_confidence_level on the team or confidence_level on the experiment’s stats_config. If a user reports a p-value of 0.07 as “significant”, they’re likely on the 90% setting; check before debugging the math.
• Significance is per-metric. With many metrics, expect some to flicker in/out as the sample grows.

C7 — Bayesian vs Frequentist confusion [MEDIUM]

A frequent source of confusion:

• Overlapping confidence intervals do not imply non-significance in Bayesian. Overlapping intervals are a frequentist heuristic. In Bayesian, significance is determined by win probability, so overlapping credible intervals can still indicate a clear winner.
• p-values don’t apply in Bayesian. A question about “p < 0.05” is a frequentist frame. If the experiment is on Bayesian (default), redirect to win probability + credible interval.
• Frequentist is opt-in. Most experiments are Bayesian unless stats_config explicitly selects Frequentist.

C8 — Inconclusive but trending — when is it ok to ship? [MEDIUM]

Shipping an inconclusive result can be defensible when all of these hold:

• A clear primary metric improvement without a guardrail regression
• Strong qualitative conviction (replays, user feedback, intuition)
• The cost of being wrong is low (e.g. easy to roll back via the flag)

Do not ship if the timeseries chart shows a sustained regression — point-in-time significance can flip, but a sustained downward trend on the timeseries is a stronger signal than a snapshot reading.

Recommend the user open the experiment’s timeseries view (per metric) — point-in-time significance can flip, but a sustained trend is a stronger signal than a snapshot reading. The agent can also pull this directly via experiment-timeseries-results.

For the qualitative part (replays / intuition), invoke the posthog:analyzing-experiment-session-replays skill — it surfaces variant-level replay patterns and is the right tool when the call is “primary metric is up, no guardrail regression, do we ship?”

C9 — “Significance reached” notification is not a green light to ship [HIGH]

PostHog can mark a metric as significant and send a notification well before the experiment has accumulated enough data for the result to be stable. The verdict can revert as the sample grows. Treat the notification as a prompt to review, not an instruction to ship.

Before acting on a significance notification, check all of:

• Participants per variant. A minimum-sample guard runs before analysis — a low floor plus np > 5 / n(1-p) > 5 for proportions (legacy stats module: 100 exposures per variant). That’s a floor for analysis, not a sufficiency bar for shipping. Aim for the number the running-time calculator produced when the experiment was set up.
• Days running. For high-stakes ships, wait at least a full week before acting on a significance flag — shorter windows can swing as the sample grows. This is a working norm, not a product-enforced threshold.
• Pre-planned duration. If the experiment hasn’t reached its planned end date, the significance is “current best estimate”, not “settled”.
• Variant balance and $multiple %. If A/B/skew (bias-and-skew.md) is in play, the significance verdict is suspect regardless of how large the gap looks.
• Secondary metrics. See C10.

When a previously-significant banner reverts to not-significant, that’s not a bug — it’s the same analysis updated with more exposures. Explain the difference between signal seen so far and result confirmed.

C10 — Ship-variant default does not consider any metric result [HIGH]

The End-experiment modal pre-fills the “Variant to keep” selector with the first non-control variant (feature_flag_variants[1].key) every time it opens. There is no significance check, no primary-metric direction check, and no guardrail check feeding that default. The “End experiment” button is gated by selecting a conclusion (won / lost / inconclusive / stopped early), not by touching the variant selector — so a user who picks a conclusion and clicks through without re-examining the variant ships the position-default variant. The only way to end without rewriting the flag is to manually clear the variant selector before clicking; the modal does not prompt for this.

The modal also asks how to release the chosen variant, with two radio options:

• Roll out to the experiment population (default, recommended) — variant distribution flips to 100/0 for the chosen variant; the flag’s existing release conditions and per-user variant overrides are preserved. Only users already in the experiment’s population see the variant.
• Roll out to all users — additionally prepends a catch-all release condition that overrides existing release conditions and per-user overrides. Anyone hitting the flag gets the chosen variant.

The release-mode choice doesn’t read metrics either; the safer “experiment population” option is the default. If the user clicks through without re-examining, they get the safer behavior on release mode but still the position-default variant — those are independent risks.

Recommend: before clicking “End experiment”, do three things:

1. Manually review every metric — primary direction and significance, plus every secondary / guardrail metric. The position-default is not a “winner”.
2. Explicitly choose the variant to keep — either re-pick from the dropdown (after reviewing metrics) or clear it to end without shipping. Don’t accept the pre-fill silently.
3. Confirm the release mode matches intent — “experiment population” keeps the variant scoped to current targeting; “all users” overrides existing release conditions and per-user overrides. The default is the safer choice; flag any non-default selection back to the user explicitly.

If any guardrail is trending negative, or the primary isn’t actually significant, the safe move is to keep control rather than ship the position-default. This matters most for sophisticated users who set guardrails for a reason — they are exactly the population the default will mislead.

C11 — External calculator disagrees with PostHog [MEDIUM]

A common case: conversion counts from the experiment page get pasted into an online A/B calculator, which returns a different verdict (“not significant” vs PostHog’s “significant”, or vice versa).

Two questions to ask before debugging stats:

1. Which methodology? PostHog is Bayesian by default. Most online calculators are Frequentist. The two answer different questions; they will not agree on borderline cases. If the user wants a Frequentist comparison, flip stats_config and re-read (see C6).
2. Are the inputs actually the same? The numbers on the experiment page are post-scope — $multiple excluded, test accounts filtered, exposure-bounded date range, per-user aggregation for trends, conversion-window applied for funnels. An online calculator gets none of that — if the user typed in raw event counts they grabbed from SQL, the calculator and PostHog are computing on different populations, and disagreement is expected.

After confirming both methodology and inputs match, if the disagreement persists, treat it as a real anomaly worth investigating with the experiment URL.