1[2 {3 "input": {4 "llmo_project_name": "cmd-i-skill-evals",5 "baseline": "a4b9d02e-203c-4d62-b8ec-ae986e83cb06",6 "candidate": "a05b32f1-351b-44f0-bf4a-efa85c910de7",7 "question": "Are there specific scenarios where the notebook skill was loaded in one experiment but not the other?"8 },9 "labels": {10 "answer": "Yes, there are 4 scenarios where the candidate loaded the skill and the baseline didn't, and 3 where the baseline loaded the skill and the candidate didn't."
30 "question": "Which judge gives answers that are better aligned with human preferences, as represented in the eval set, the candidate judge or the baseline judge?"
31 },
32 "labels": {
33 "answer": "4.1 (the candidate) is better aligned. I think it's because we're giving it pretty heavy and specific instructions, which the 4.1 series is built to follow, rather than more open ended or \"goal oriented\" prompting that reasoning models are better at.",
53 "question": "Does the tool \"code inspection\" help getting better results?"
54 },
55 "labels": {
56 "answer": "Yes, but the tool is not called, leading to a suspicion that the better results are due to informing the model that it CAN call code.",
57 "key_points": [
58 "Overall results slightly improve, though not significantly, across all categories. Most gains, suspiciously, occurred in non-sandbox scenarios, suggesting the improvement is due to simply informing the model it can investigate code.",
59 "Sandbox use is primarily for code investigation, but results vary significantly from previous experiments, indicating high run-to-run variance. These improvements aren't correlated with sandbox usage."