A six-step audit for finding where typed classification replaces a generative call, and for proving it was worth doing. The framing of steps 1–2 is adapted from jevify, a prompt for investigating what a decision model makes possible in an existing project.
(opens in a new tab)
Keep exploration separate from production changes. Do not edit the agent until step 6.
Which primitives apply — Noul, Choice, or Score, and why.
Which questions share one request versus genuinely depend on a prior answer. Questions in one request are answered independently; if B depends on A’s answer, they are two requests and two nodes.
How plain code consumes the result — the routing function, in full.
What still needs generation or retrieval. Most workflows keep an LLM for the part that produces text; the decision model only picks which items get there.
Do not bury hard reasoning in a fuzzy question. “Is this code correct?” is not a classification. If a question needs multi-step reasoning to answer, it needs a reasoning model.
Estimate the whole workflow, not one call: retries, fallbacks, the escalation path, and the generative work that remains.
Separate per-request latency from end-to-end. Fan-out changes the second and not the first.
Compare against the cheaper alternatives you skipped: deterministic code, caching, embeddings, a smaller model. Sometimes a regex really is the right answer.
If you have no measurements yet, state the assumption and the break-even point rather than a number you cannot defend.
When benchmarking against an LLM baseline, make the baseline fair — native structured output (method="json_schema"), no tool-schema injection, caching enabled on any stable prompt prefix. An unfair baseline makes the result useless for deciding anything.
A decision model returns probabilities. Probabilities are signals whose calibration must be tested on your workload; vendor benchmarks do not transfer.
Baselines — current behavior, and the cheapest non-model alternative.
Asymmetric costs — a false negative on a privilege check is not a false positive on a spam check. Weight them.
Latency distribution — p50 and p99, not the mean.
Adversarial and ambiguous inputs — items that should land mid-rubric. These are where thresholds get decided.
Threshold and fallback validation — sweep thresholds against labeled data. Do not hand-pick them from a handful of examples.
Go/no-go criteria, written before you run it.
langsmith-skills covers dataset construction and evaluators for this step.
Prompted classifier → one classifier node. The direct swap. The prompt’s enumerated options become Choice.criteria; its rating scale becomes Score.criteria; its yes/no becomes a Noul. Delete the parsing code — that is the point.
Chain of LLM guards → one request, many questions. Sequential guardrail calls that each ask one yes/no collapse into a single request with several Nouls, then a routing function. Only do this where the guards are genuinely independent.
Sampling → full coverage. Where cost forced you to check 1 in 100, check all of them and route the uncertain ones to the expensive path. This usually improves the product, not just the bill.
Keyword rules → semantic questions. Replace regex allowlists with a Noul plus a threshold. Keep the regex as a fast path if it is precise; use the model for the tail it cannot cover.