Setting the file. One moment.
Skill 25 · Configuring Experiment Analytics
Subchapter 25.3
references/metric-templates.mdMarkdown8 KBView on GitHub
Most metric requests match one of these shapes. Pick the template, fill in the project’s events (confirmed with read-data-schema), and set its defaults.
Check for a shared metric first (Step 1 in SKILL.md): a matching shared metric beats any template.
Every conversion window needs a unit: set conversion_window and conversion_window_unit together, or neither.
A window without a unit is ignored and the metric counts events until the experiment ends.
| Template | Use when the user asks | Shape | Defaults to set | Say in the summary |
|---|---|---|---|---|
| Conversion | “more people do X” | funnel, one step (exposure is step 0) | goal: "increase". A window with a unit when X usually happens within a known delay (a signup within a day) | X must be captured for the exposed person, with the same identity |
| Journey | “more people get from A through B to C” | funnel, 2 to 4 ordered steps | Same as conversion | Numbers won’t match a product analytics funnel: exposure comes first and test accounts are removed |
| Count per user | “people do X more often” | mean, math: "total" | upper_bound_percentile: 0.99 | Heavy users can dominate the mean; the upper bound caps them |
| Revenue per user | “more revenue”, “people spend more” | mean, math: "sum", math_property = the revenue property | upper_bound_percentile: 0.99 | A few large orders make revenue noisy. If revenue arrives on several events or properties, the metric must cover all of them. Server-side purchase events need the same identity as the exposure |
| Average value | “X gets faster”, “sessions get longer” | mean, math: "avg" with math_property = the value property | goal: "decrease" when lower is better (latency, load time), otherwise "increase" | Server-side events have no session |
| Unique-user rate | “a larger share of users do X” relative to another group | ratio, math: "dau" (unique users) on both numerator and denominator | None | A few heavy users can move a total-over-total ratio; unique users avoids that |
| Average order value | “bigger orders” | ratio, numerator math: "sum" of the revenue property, denominator math: "total" of the same event | numerator_outlier_handling: {"upper_bound_percentile": 0.99} (ratio metrics take no top-level upper_bound_percentile) | Read it together with conversion and revenue per user: fewer, larger orders can raise it while revenue falls |
| Return rate | “people come back”, “retention” | retention, start_handling: "first_seen", retention_window_start: 1, retention_window_end: 7, retention_window_unit: "day" | Completed window only (see below) | Users exposed near the end have had less time to return and pull the rate down |
| Trial to paid | “more trials convert to paid” | funnel with a fixed window | A window with its unit, longer than the trial: it starts at exposure, not at the trial start, so it must cover the time to start the trial plus the trial. Completed window only (see below) | One window must cover both the trial start and the payment |
The experiment setting “Require completed conversion or retention window” (only_count_matured_users) counts only people whose whole window has passed.
Creation turns it on when the team’s default is on (team_defaults.only_count_matured_users in experiment-setup-context).
The agent cannot set it: experiment-create and experiment-update don’t expose it.
For return rate and trial to paid, when the team default is off, recommend it in the summary and tell the user to turn it on in the experiment’s Settings tab.
The primary metric decides the ship call, so it should be the most frequent event that still expresses the hypothesis. If the requested event is rare on this page (the running time comes out at months), make it secondary and propose an earlier, more frequent event as primary.