Setting the file. One moment.
Skill 181 · Working With Autoresearch
Subchapter 181.2
references/reading-model-quality.mdMarkdown6 KBView on GitHub
Quality has two stages. The model makes a promise at training time (holdout) and keeps it, or not, in production (realized). Lead with the realized numbers when they exist, and say plainly when they do not exist yet.
| Tool | What it gives |
|---|---|
posthog:autoresearch-retrieve | The champion’s champion_holdout_auc, champion_realized_auc, champion_lift_at_10, champion_is_preliminary, champion_realized_auc_trend, people_scored, likely_count, first_check_expected_at and coverage. Start here. |
posthog:autoresearch-online-performance-retrieve | One row per model per validated date: realized AUC with a 95% interval, Brier score, calibration error and bins, mean prediction against base rate, lift at 10% and 20%. |
posthog:autoresearch-models-list | Every champion, challenger and archived model with its scores. Use it to see whether a retrain promoted a new champion. |
posthog:autoresearch-models-retrieve | One model’s recipe (feature SQL, transforms, model class) and its feature importances. |
posthog:autoresearch-training-runs-history | Each run’s distilled summary: what won, what was tried and dropped. |
The UI gives each champion a label from AUC, and states its basis:
| Label | AUC |
|---|---|
| Strong | 0.8 and up |
| Fair | 0.7 to 0.8 |
| Weak | below 0.7 |
champion_is_preliminary is false. Otherwise it is testing only and comes from holdout AUC.first_check_expected_at says when the first check is due.A validated date where every scored person had the same outcome has no AUC.
Its online performance row carries the warning single_class_no_auc.
This is common on a rare target with a short horizon.
Tell the user that realized AUC needs dates where some people did the target and some did not, and that more dates will fill it in.
AUC measures ranking. Calibration measures whether a 30% score means 30% of those people do the target.
mean_p_y with base_rate on each realized row. If mean_p_y is clearly above, the model over-predicts. If it is clearly below, it under-predicts.$autoresearch_p_y is the corrected probability, and $autoresearch_p_y_raw keeps the model’s own score. If calibration is still poor, tell the user to rely on the order and the segments, not on the absolute number.Lift at 10% is the clearest sentence for a non-technical reader: “people in the top 10% are 3× more likely than average to do X”. Lift at 20% helps when the user plans to act on a larger group.
The Predictions tab splits scored people into Likely, Possible and Unlikely.
Large populations score in rolling batches, so “scored today” and “everyone has a fresh score” are different claims.
coverage on posthog:autoresearch-retrieve has population, with_score, never_scored and the score age in days (average, median, 90th percentile, oldest).never_scored above zero, or a median score age above the cadence, means a rolling population.When the user asks “is my model any good?”, answer in this order: