Setting the file. One moment.
Subchapter 31.2
references/dataset-formats.mdMarkdown4 KBView on GitHub
Standard chat-completion JSONL. Each line: JSON object with messages array.
Scripts
scripts/11 files{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is 2+2?"}, {"role": "assistant", "content": "4"}]}Rules:
messages must contain at least one user and one assistant messagesystem message is optional but recommendeduser/assistantassistant (that’s what the model learns)Validation checklist: .jsonl extension, valid JSON per line, every example has messages, every message has role and content, no empty content.
Three top-level fields: input, preferred_output, non_preferred_output.
{"input": {"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain gravity."}]}, "preferred_output": [{"role": "assistant", "content": "Gravity is a fundamental force that attracts objects with mass toward each other."}], "non_preferred_output": [{"role": "assistant", "content": "Gravity is when stuff falls down."}]}Rules:
input: Object with messages array (system + user turns). May include tools and parallel_tool_calls.preferred_output / non_preferred_output: Array of messages (assistant or tool role only)assistant messageDPO REST API example:
{
"model": "gpt-4.1-mini-2025-04-14",
"training_file": "file-abc123",
"method": {
"type": "dpo",
"dpo": { "beta": 0.1, "l2_multiplier": 0.1 }
}
}Chat-completion format with key differences from SFT:
{"messages": [{"role": "user", "content": "Write a Python function to reverse a string."}], "reference_code": "def reverse_string(s):\n return s[::-1]", "expected_output": "olleh"}Rules:
user role (model generates its own response)messages are accessible to grader via item.*assistant as last message — unlike SFT, RFT generates its own outputsAPI version: Python graders require api-version=2025-04-01-preview or later.
Grader types: string_check (exact match), text_similarity (fuzzy/BLEU/ROUGE), python (custom function), score_model (LLM judge), multi (weighted combination).
Python grader template:
def grade(sample, item):
"""
sample: dict with 'output_text' (model's generation)
item: dict with extra fields from JSONL
Returns: float 0.0–1.0
"""
output = sample.get("output_text", "")
reference = item.get("reference_code", "")
return scorePython grader constraints: 256KB code max, no network, 2GB memory, 1GB disk, 2min timeout.
Grader field access:
sample.output_text → model’s generationsample.output_json → structured output (if using response_format)item.* → extra JSONL fields{{item.field_name}} — no spaces inside braces, no array indexinguser), add grader reference fields. Use scripts/convert_dataset.py --format rft.