Setting the file. One moment.
Skill 181 · Working With Autoresearch
Subchapter 181.3
references/troubleshooting-and-steering.mdMarkdown10 KBView on GitHub
For any “something is wrong with my model” question, read these three, in this order:
posthog:autoresearch-retrieve: status, champion, last scored time, coverage.posthog:autoresearch-training-runs-list: the newest training runs, their status, , and .iteration_countbest_holdout_scoreerrorposthog:autoresearch-runs-list: the newest scoring and validation runs, their status, rows_scored and error.| Symptom | Likely cause | What to do |
|---|---|---|
Training run shows 0 iterations but is running | Normal. Counters are written only when the run completes. | Wait. Check again later. |
Training run failed with 0 iterations | Usually the environment, not the question: access, quota or a sandbox problem. | Read error. Confirm the user has PostHog Desktop access. Retry once from the model page. |
posthog:autoresearch-train-create returns 403 from a sandbox | The call came from inside a sandbox, such as a PostHog AI cloud task. | Ask the user to start training from the model’s page. Do not retry. |
posthog:autoresearch-train-create returns 403 about Desktop access | The user has no PostHog Desktop access, which cloud training needs. | The model page refuses too. The user needs Desktop access first. |
posthog:autoresearch-train-create returns 400 | A run is already live, the model is paused, its creator or target is no longer valid, or an action target and a key without action:read. | Read the message. Wait for the live run, resume in the UI, add the action:read scope, or recreate the model only when the target or creator is gone. |
| Training completed but no champion | No experiment produced a usable model, or none beat the current champion. | Read the run’s summary with posthog:autoresearch-training-runs-history. Check the base rate and positive count. |
| Scores barely vary at all | An identifier mismatch between the features and the people being scored. | Check the spread with the quantile query in using-predictions.md first: on a rare target, healthy scores can all be small. If the quantiles are the same value, read the champion’s feature SQL with posthog:autoresearch-models-retrieve. Every row must be keyed on the person id. |
| Holdout AUC above about 0.97 | The features likely see the outcome, or a proxy that happens because of it. | Read the feature importances. If the top feature is the target or a step after it, suggest removing it. |
Scoring run failed | The feature query was too heavy, or a transient error. | Read error. After two scheduled days in a row of repeatable failures, the product starts a small rescue training run to find a champion that can score. It can fail too, so check the next training runs rather than waiting. A repeated memory error means the features scan too much: steer the next run toward shorter windows. |
Fewer people in a cohort than rows_scored | Some scored people could not be resolved to a current distinct ID. They get an event but no property. | Expected for a small share. Use the events for analysis that must include everyone. |
| Population only partly scored | Rolling batches on a large population. | Expected. Read coverage on posthog:autoresearch-retrieve. |
| No realized AUC before the horizon | The horizon has not passed since the champion’s first scoring. | Wait. first_check_expected_at on posthog:autoresearch-retrieve says when. |
| No realized AUC after the horizon | Each validated date had only one outcome class (warning single_class_no_auc). | Explain that AUC needs dates with both outcomes. More dates fill it in. On a very rare target, consider a longer horizon. |
| Realized AUC drops over several dates | Behavior shifted, or the training population no longer matches who is scored. | Compare training and inference populations. Retrain, or steer with a suggestion. |
A suggestion is a note to the training agent. It reads queued suggestions at the start of the next training run.
posthog:autoresearch-suggestions-create with a free-text prompt.
Good suggestions name a hypothesis: “account for day-of-week, the target spikes on Mondays”, “try features from the billing page events”, “drop the recency feature, I suspect it leaks”.priority: "try_next" makes the agent act on it before its own ideas. The default consider is advisory.posthog:autoresearch-suggestions-list shows how the agent handled each one: acted_on, picked_up or dismissed, with a one-line agent_response.posthog:autoresearch-train-create after the user confirms.The agent decides whether a suggestion helps, and the server still picks the champion by score. A suggestion that makes the model worse is recorded and dropped.
Create a new model instead of retraining when:
Leave the old model in place until the new one has realized results, then pause or delete the old one in the UI.