Reward Hacking

The model optimizes for the grader’s scoring function rather than the actual task. The training grader becomes a proxy reward that diverges from true quality — the model games the proxy instead of improving.

Core rule: Your training grader MUST produce the same ranking as your evaluation methodology.

Misaligned graders are the #1 cause of reward hacking.

Train-Val Gap Thresholds

Pre-Training Checklist

1. Baseline the grader: Run training grader on base model outputs. Record scores as your floor.
2. Cross-validate graders: If training grader ≠ eval grader, generate 50 outputs, score with both, compute Spearman ρ. Proceed only if ρ ≥ 0.8. If ρ < 0.6, fix alignment first.
3. Test hackability: Generate 5 intentionally bad outputs that might score well. If grader scores any > 5/10, redesign it.
4. Set gap threshold: Monitor train-val gap every eval_interval. Stop if > 0.10.

Grader Iteration Loop

When reward hacking is detected:

Red Flags Checklist

Investigate immediately if any are true:

• Not done: Train-val gap > 0.10
• Not done: Training reward increasing but eval quality stable or declining
• Not done: Model outputs are longer/more verbose than base model
• Not done: Outputs structurally match references but are semantically wrong
• Not done: Different LLM judges disagree on quality
• Not done: Conciseness/style scores dropping while correctness climbs
• Not done: Model produces “template” responses

Key Principles