Setting the file. One moment.
Skill 07 · Auditing Experiments Flags
Subchapter 7.1
references/experiment-checks.mdMarkdown9 KBView on GitHub
Run these checks against each experiment fetched via experiment-get or experiment-list.
For each check, the “Look at” section tells you which fields to inspect on the experiment object. The “Findings” section lists what to report and at what severity.
Verifies the experiment has a valid primary metric configuration.
Look at: metrics, metrics_secondary
Findings:
No metrics at all: Both metrics and metrics_secondary are empty or missing.
Secondary metrics only: metrics is empty but metrics_secondary has entries.
Verifies the experiment’s linked feature flag is valid and correctly configured.
Look at: feature_flag (the linked flag object or ID), and fetch the flag via feature-flag-get-definition if only an ID is available.
Findings:
Missing flag: feature_flag is null or missing.
Inactive flag: The linked flag exists but active is false.
Deleted flag: The linked flag has deleted set to true.
Incomplete variant split: The rollout percentages in the flag’s filters.multivariate.variants don’t add up to 100.
Too few variants: The flag’s filters.multivariate.variants holds fewer than two variants.
Checks for contradictions between an experiment’s conclusion and its current flag state.
Look at: end_date (non-null means concluded), archived, parameters.recommended_variant, and the linked flag’s active state and variant configuration.
Findings:
Conclusion contradicts shipped variant: The experiment concluded with a recommended variant (in parameters.recommended_variant), but the flag is rolled out to a different variant at 100%.
Concluded but still splitting: The experiment has an end_date (it’s concluded) but the linked flag still has multiple variants with non-zero rollout (traffic is still being split).
Checks for experiments stuck in unproductive states.
Look at: created_at, start_date, end_date, description (for hypothesis)
Findings:
Stale draft: start_date is null (never launched) and created_at is more than 7 days ago.
No hypothesis: description is empty or missing, and the experiment has been launched (start_date is set).
Checks for experiments that have ended but whose flags are still active and splitting.
Look at: end_date, archived, and the linked flag’s active status and variant rollout.
Findings:
end_date is set (experiment ended), but the linked flag is still active: true and has multiple variants with non-zero rollout percentages.
Note: This is related to but distinct from “concluded but still splitting” in check 3. Check 3 focuses on the contradiction with the conclusion; this check focuses on the resource waste of an ended experiment still consuming flag evaluations.
Checks whether a running experiment has collected enough data.
Look at: start_date, end_date
Findings:
Very short run: start_date is set, end_date is set, and the duration is less than 7 days.
Short run: Duration is between 7 and 14 days.
Checks for unusual statistical configuration.
Look at: start_date, end_date (or current date if still running), parameters.stats_config
Findings:
start_date to end_date or today if still running).
Checks for flag modifications that may have affected experiment integrity. These checks require activity logs. If activity logs are not available, skip this entire check and note it was skipped.
Look at: Activity log entries for the linked feature flag, filtered by the experiment’s run period (start_date to end_date or today).
Findings:
Pre-run flag changes: The flag was modified between experiment creation and launch.
Mid-run rollout changes: The flag’s rollout percentages were changed while the experiment was running.
Mid-run variant changes: Variants were added or removed from the flag while the experiment was running.
Mid-run flag toggles: The flag was toggled on/off during the experiment run.
Mid-run targeting changes: The flag’s targeting conditions (properties, groups) were modified during the run.