Setting the file. One moment.
Skill 36 · Creating Experiments
Subchapter 36.1
references/setup-decisions.mdMarkdown23 KBView on GitHub
Use this file after experiment-setup-context returned. Each section maps the tool’s facts to one configuration choice and to a tier for the summary:
Every section has the shape {status, data}. The field paths below are inside data: target_surface.anonymous_share means target_surface.data.anonymous_share.
A [] in a path means a list: previous_experiments.experiments[].primary_metric_types is a field on each row.
Write every dotted path from its section name, never from below it: previous_experiments.summary.using_custom_exposure, not summary.using_custom_exposure. A test resolves the full paths against the response and cannot check a shorthand. A bare field name is fine in a sentence whose section is already named.
A section whose status is not ok gives no facts. Choices that depend on it drop to “best guess” at most, and the summary says which section was missing.
Read these facts:
target_surface.anonymous_share: among the distinct IDs that sent the target event and say whether they are identified, the share that were anonymous. Every SDK’s events count, not only the web SDK’s. A high share usually means logged-out visitors. It is null only when no target event carried $is_identified at all.target_surface.device_id_share: the share of target events that carry a device ID, across every SDK that sent them. Null when there were no target events.target_surface.libs[]: target_surface.libs[].anonymous_share and target_surface.libs[].device_id_share per SDK that reached the surface, plus a target_surface.libs[].category of web, mobile, server or other. A row’s device_id_share is always a number, because the row exists only where that SDK sent target events.sdk_profile.libs[]: per SDK, sdk_profile.libs[].device_id_share and sdk_profile.libs[].locally_evaluated_share on flag calls, plus a sdk_profile.libs[].category read the same way. It covers the whole project, not one page, so use target_surface.libs to pick the rows that apply to this page. It is empty when the project sent no multivariate flag calls in the last 7 days.sdk_profile.libs_on_any_event[]: which SDKs the project sends any event from. Set only when sdk_profile.libs is empty.| Facts | Choice | Tier |
|---|---|---|
anonymous_share above about 0.9, or below about 0.1: the page is seen mostly by one kind of visitor | User-id bucketing. Leave ensure_experience_continuity out, so the team’s persistence default applies | Best guess |
| Between: the same people likely see the page before and after they are identified, and user-id bucketing can switch their variant at that point | The steps below | Best guess, always listed for review |
anonymous_share is null | User-id bucketing, team persistence default | Not decided: say no target event reported whether its sender was identified, so the identity mix is unknown |
When anonymous_share is in the middle band, work through the steps below whatever the project has done before.
A summary showing previous_experiments.summary.using_persistence and previous_experiments.summary.using_device_id_bucketing both at 0 is not a reason to skip them.
A project whose surfaces cross identification and that has never used either has most likely been splitting the same person across two variants all along, and a run of experiments agreeing with each other does not make that right.
Name it in the report: say the project’s earlier experiments bucket on the user id and that this surface needs more, then follow the steps.
Precedent can confirm a choice these facts already allow. It never overrides this one.
Device-id bucketing is a web option. A mobile SDK puts no $device_id on its events, flag calls included, so both device-id shares step 1 reads sit near 0 on a mobile row and step 1 can never pass.
When every target_surface.libs[] row that reaches the surface has category: "mobile" and target_surface.libs_truncated is false, go straight to step 2: the choice is persistence or user-id bucketing, and reporting “not decided” over a share that can never arrive is wrong.
target_surface.device_id_share is near 1, at least one sdk_profile.libs row has a lib that appears in target_surface.libs, and every matching row has sdk_profile.libs[].device_id_share near 1. A flag call without a device ID gets no variant, so a server SDK must forward the browser’s device ID; local evaluation works when it does. target_surface.device_id_share is read from the target events; a sdk_profile.libs row’s device_id_share is read from flag-call events. Neither is read from the flag requests, so neither proves that the request carried a device ID. The web SDK sends the device ID on flag requests from posthog-js 1.307.1. An older version puts a device ID on events and still gets no variant, so tell the user to check the SDK version. Set it up with the device-id recipe in configuring-experiment-rollout. experiment-create cannot set it.ensure_experience_continuity: true), if no matching sdk_profile.libs row evaluates flags locally: every row that matches a lib in target_surface.libs has sdk_profile.libs[].locally_evaluated_share near 0, or null on a web or mobile row. Null on a server row is unknown: go to step 3. Persistence also needs person profiles for anonymous users, no bootstrapping, and $anon_distinct_id on server flag calls. The tool cannot see those three, so list them for the user to check.When sdk_profile.libs holds rows, steps 1 and 2 read only the ones that match, so a target_surface.libs[] row with no sdk_profile.libs row of the same lib is unchecked rather than safe.
sdk_profile.libs reads multivariate flag calls over 7 days while target_surface.libs reads target events over 14, so a missing row can mean that SDK evaluates no multivariate flag, or only that it made none in the shorter window.
Name every unmatched SDK in the report.
When an unmatched row has category: "server", take step 3: an unread server SDK is the case that breaks both steps, because it may send its flag calls without a device ID and may evaluate them locally.
Otherwise keep the step’s choice and drop the tier to “best guess”, naming the SDK that was not checked.
Both lists are capped, so read the truncation flag before you trust a match.
target_surface.libs holds at most the 5 SDKs with the most people on the surface, and target_surface.libs_truncated is true when the cap dropped a further SDK.
When it is false, the list is complete and the check above is conclusive; when it is true, an unseen SDK can reach the surface, a server one included.
When target_surface.libs_truncated is true, say the list is at its cap and drop the tier to “best guess” even when every row matched, because “no unmatched SDK” is then unproven rather than true.
sdk_profile.libs is capped at 10 and does report it, in sdk_profile.libs_truncated.
When that is true, an SDK can read as unmatched only because its own row was dropped, so name the cap alongside it. The cap cannot add a bad matching row, so steps 1 and 2 stay safe on the rows that did come back.
If sdk_profile.libs is empty, read sdk_profile.libs_on_any_event.
The endpoint fills that field only on an empty profile, so a profile that holds rows and matches none of them to target_surface.libs is not this branch: the unmatched-SDK rules above decide that case, and the field reads null there.
It is null on an empty profile too when its own query timed out. Nothing then names the platforms, so none of the reads below apply: say so and stay on user-id bucketing, marked “not decided”.
It names the platforms the project sends from, which settles one case and no other: a project that sends only from mobile SDKs can never reach the share step 1 needs, so the choice there is persistence or user-id bucketing.
Trust that mobile-only read only when sdk_profile.libs_on_any_event_truncated is false. When it is true, the cap dropped the SDKs that sent the fewest events, so the project can also send from a web or server SDK that the list does not show.
Every other platform mix stays open, a server-only project included. A server SDK that forwards the browser’s device ID puts one on every flag call, and this fallback cannot see whether it does.
It says nothing about device IDs on flag calls or about local evaluation, so steps 1 and 2 still do not pass. Keep user-id bucketing and mark it “not decided”.
Say which platforms the project sends from. When target_surface.device_id_share is near 1, target_surface.libs lists no server SDK, and target_surface.libs_truncated is false, say device-id bucketing is the likely fit: with no server SDK in the mix, nothing has to forward the device ID for the flag call to carry one. A mobile-only project never reaches that share, so it is settled above rather than here.
That share counts target events, and this branch read no flag call at all, so step 1’s version caveat applies here with nothing to offset it: posthog-js sends the device ID on flag requests only from 1.307.1, and an older version puts one on the events while the request still gets no variant.
Give device-id bucketing as a lead to check, with the version named as the thing to confirm, and leave the choice at user-id bucketing.
Leave ensure_experience_continuity out unless you are choosing persistence. When omitted, experiment-create applies the team’s default (team_defaults.flags_persistence_default). Set it to false only when the team default is true and local evaluation rules persistence out. The device-id recipe needs no such step: create-feature-flag leaves persistence off unless you set it, and experiment-create rejects a feature_flag object for a flag that already exists.
Never change bucketing or persistence on a flag that is already live.
If sdk_profile.evaluated_on_server_and_web is true, or target_surface.libs lists a server SDK next to web:
$feature/<flag> from that event.Tier: not decided. The facts show the risk, not whether the code handles it. Tell the user what to check.
Keep the team’s default exposure event (team_defaults.default_exposure_event). Use a custom exposure only when the user names the moment the person sees the change and the flag is evaluated well before it: a prefetch, a server render, or a flag read across the whole app while the change sits on one surface. Tier: best guess.
Keep the test-account filter on and say so. team_defaults.new_experiments_filter_test_accounts says what a new experiment does when its own exposure criteria say nothing, and it does not follow the project’s insight default. If team_defaults.test_account_filter_count is 0, say the project has no test-account filters, so the setting changes nothing. Tier: best guess.
read-data-schema.shared_metrics.metrics[] for one to reuse instead of building an inline metric.
shared_metrics.metrics[].matches_metric_event set to true says the event appears in the metric somewhere. shared_metrics.metrics[].metric_event_roles says where: funnel_step and funnel_final_step for a funnel, mean_source for a mean, ratio_numerator or ratio_denominator for a ratio, retention_start or retention_completion for a retention metric.shared_metrics.metrics[].metric_event_roles with a true match is not a rejection. The metric does count the event, but it is stored in a shape the role reader does not parse, which its shared_metrics.metrics[].metric_type shows as null or as a type not listed above. Retrieve it and read the query.experiment-saved-metrics-retrieve and check its metric_type and math against what the user asked for. The role alone does not prove those, so the tier stays best guess until you have read the query.experiment-update with saved_metrics_ids. Tier: confident. When the user asked for no questions, link it and report it rather than asking first.shared_metrics.metric_event_match_truncated is true, the match read only the newest shared metrics, so an older metric that counts the event shows matches_metric_event false. Call experiment-saved-metrics-list with event set to the metric event before you build an inline metric. That filter reads every shared metric.configuring-experiment-analytics (references/metric-templates.md). Tier: best guess.If the metric counts only some occurrences of its event, for example a purchase with one payment method, pass those filters as metric_properties on the setup-context call. Use the same filters on the metric you build.
Without them, candidate_metric reports a baseline for every occurrence of the event, and the running time below is too optimistic. candidate_metric.metric_properties echoes the filters the tool read.
Every conversion window carries a unit (conversion_window_unit). A window without a unit is ignored.
Read previous_experiments.experiments[].primary_metric_types. It carries one entry per primary metric, so an experiment with two primary metrics of different types carries both.
A row counts only when its list is not empty and every entry in it is the same type, and that type is the row’s shape. An empty row ran no primary metric, and a mixed row made two different choices, so neither votes.
Count the rows that qualify. Fewer than three is not a precedent: say the project has too few experiments to read one.
From three qualifying rows up, propose the type held by at least 80% of them - 3 of 3, 4 of 4, 4 of 5 - unless the request asks for something else. Say you followed precedent and out of how many experiments. A project that measures every test as a funnel gets a funnel.
Then read previous_experiments.experiments[].primary_metric_events. When the event you picked already appears on a row, say which experiment measured it and as what, reading primary_metric_types on the same row. Tier: best guess.
When candidate_metric.status is ok and it has baseline stats:
experiment-calculate-running-time:
metric_type: "funnel", baseline_stats from candidate_metric.funnel_baseline_statsmetric_type: "mean_count", baseline_stats from candidate_metric.mean_count_baseline_statsminimum_detectable_effect: team_defaults.minimum_detectable_effect, else team_defaults.product_default_minimum_detectable_effectexposure_rate_per_day: target_surface.exposures_per_day_estimate, times the rollout share when the experiment includes less than 100% of trafficnumber_of_variants when there are more than two variantsexperiment-create as running_time_calculation, so the plan is stored on the experiment:
minimum_detectable_effect: the value you sent (the calculator does not return it)recommended_sample_size: the calculator’s recommended_sample_sizerecommended_running_time: the calculator’s recommended_running_time_daysThe estimate assumes people are exposed on the target surface. If the flag is evaluated more widely (across the whole app, or on a page before the one that changes), the exposed population is larger and converts less. The baseline and the running time are then too optimistic. Say so, and consider a custom exposure on the surface event. This is the one case where a custom exposure is right in a project that has never used one, and “Precedent” says so too.
previous_experiments.experiments[].outcome.control_baseline_value shows what that dilution looked like: a control baseline far below the rate the surface itself converts at is an experiment that was exposed too widely.
Check how the call scoped the target surface before you trust these numbers. target_surface.target_properties and target_surface.target_url_contains echo the filters the tool read.
target_url_contains is a substring match on $current_url, so a bare domain matches any host that contains it, and a homepage path matches every page under it. Both overstate the page’s traffic and the exposure rate.
If the echo came back wider than the surface under test, the numbers here are too optimistic.
Correct the scope and call the tool once more: an exact $host and an exact $pathname in target_properties for one page, or an exact $host with only the path fragment in target_url_contains for a wider surface.
Read the numbers from the second response.
A candidate_metric.persons_reached of 0 straight after an exact filter usually means the value is not the one the project records, such as /pricing against /pricing/.
Read the shape back with read-data-schema (event_property_values, a sample of the values) and correct the filter, rather than widening it.
previous_experiments.summary says how the project’s earlier experiments fared for size. When previous_experiments.summary.launched_with_zero_analyzed_exposures or launched_with_under_100_analyzed_exposures is a large share of launched, the project has been launching experiments that could not measure anything. Say so, and treat the running time as the number to get right rather than a formality.
candidate_metric.persons_reached is 0: no one sent the target event in the window under target_properties and target_url_contains. Fix the target before reading the two counts below. Only people who reached the target can convert, so candidate_metric.persons_converted is 0 whatever the metric does, and candidate_metric.event_volume counts the metric event without the target and says nothing about the pair.candidate_metric.event_volume is 0: the metric event never occurred in the window under metric_properties. Check the event name with read-data-schema, then check the filters. A candidate_metric.conversion_rate of 0 says nothing until this is above 0.candidate_metric.event_volume is above 0 but candidate_metric.persons_converted is 0: the event happens, but never after the target event. Either the metric measures something people do elsewhere in the product, or the target is wrong. candidate_metric.unique_persons says how many people send the event at all, which separates a rare event from a misplaced one.Tier: confident on the arithmetic, best guess on the inputs (the baseline is an estimate over candidate_metric.window_days).
Read previous_experiments.summary. Its using_* counts cover every listed experiment, drafts included, so read them as what the project sets up, not what it launches.
When previous_experiments.experiments is empty, the project is creating its first experiment. There is no precedent, and nothing in this section applies. Say that rather than reporting an absent precedent as agreement.
If most listed experiments use one setting, follow it unless the facts above contradict it, and say you followed precedent. Tier: best guess. Two limits:
previous_experiments.summary.using_custom_exposure and using_activation are 0 has given you no reason to narrow it.Read previous_experiments.experiments[].serving_single_variant before you trust previous_experiments.summary.using_uneven_split. Shipping a variant rewrites the flag to serve that variant to everyone it matches, so an ended experiment’s flag no longer carries the split it ran with. The summary leaves those experiments out of using_uneven_split, and previous_experiments.experiments[].split_even is null on any row where serving_single_variant is set.
previous_experiments.summary.using_exposure_property_filters counts experiments whose exposure is narrowed by properties on whichever event it counts. A narrowed default event is not a custom exposure, so using_custom_exposure does not count it, but it is still a deliberate choice about what an exposure means. Read previous_experiments.experiments[].exposure_property_filters: an experiment that counted exposure only where $pathname was /pricing is the precedent for a new test on that page.
For a holdout, look at previous_experiments.experiments[].has_holdout.
Leave stats_config out, so the experiment follows the team’s defaults (stats_method, confidence_level, CUPED, sequential testing). Set it only when the user asks for a different method. Tier: confident.