Iterative Training

Rule: Change ONE variable per experiment.

Experiment Tracking

What to Try (Priority Order)

Priority 1: Data Quality (highest leverage)

• Fix inconsistencies: Contradicting examples confuse the model
• Add diversity: Add examples for input types the model fails on
• Reduce noise: Remove “correct but not ideal” outputs

Priority 2: Hyperparameters

See references/hyperparameters.md for full guide.

Quick sweep strategy:

1. Baseline: epochs=2, lr=1.0
2. Overfitting → lr=0.5 or epochs=1
3. Underfitting → lr=1.5 or epochs=3
4. Good LR found → try batch_size=16 or 32

Priority 3: Base Model

Priority 4: Training Type

• SFT plateaued + need better reasoning → RFT (if model supports it)
• Need style alignment → DPO
• See references/training-types.md before switching

Diagnostic Decision Tree

When to Stop

1. Beaten baseline by meaningful margin (>5%) and last 3 experiments didn’t improve
2. Diminishing returns: each experiment improves < 0.1 points
3. Model is “good enough” for production
4. Budget exhausted (time or money)

Multi-Model Strategy

Run the same dataset through 2-3 base models:

1. gpt-4.1-mini — primary candidate
2. gpt-oss-20b — large-dataset specialist (500+ examples)
3. gpt-4.1-nano — fast inference option

Common Mistakes

1. Not establishing a baseline first
2. Changing multiple variables at once
3. Overfitting to the eval set (keep a separate final test set)
4. Ignoring training curves (they tell you what to change next)
5. More data without quality check (lower-quality data often makes things worse)
6. Not cleaning up old deployments (wastes quota and money)