The @SnorkelAI team did an audit of these findings using our internal QC platform. Tl/dr: We agree with many of the findings, but ~55% of the fixes from @preferencemodel had their own issues with reward hacking or over-prescriptiveness. Additionally: ~40% of the issues found were already tracked in the upstream repo - including 21 tasks with outstanding fix PRs! Building datasets like this - and maintaining ambitious open benchmarks around them - is hard. We believe it requires open, collaborative review with community maintainers and relevant domain experts to get right. @terminalbench was one of the first community efforts to build an open, continuous benchmark, and drive a vision of how OSS contributes to the rapidly advancing frontier of AI. We hope that the community will invest in working collaboratively with benchmark maintainers and task authors to keep this vision alive. --- Some takeaways + details of our preliminary audit: (1) We confirm 100% of the reward hacking / verification gap findings using our QC system. However, we find that 17/44 (39%) of the corrected tasks are just as or more reward hackable. Additionally, we don’t yet see evidence that these verification gaps meaningfully affected benchmark rollouts within CIs, though our analysis here is still ongoing. (2) Our preliminary review raises questions about the underspecification findings for 22/41 (54%) of the flagged tasks, including 7 explicitly marked by task authors as intentional design choices. These cases warrant expert peer review and adjudication: balancing over/under-specification is a critical and tricky design decision - much of human intelligence concerns dealing correctly with reasonable and realistic underspecification - and we believe these corrections made to the tasks are likely too over-specified, reducing realism and difficulty relative to spec. (3) Around 40% of the issues found were already tracked in the upstream repo (including 21 tasks with outstanding fix PRs)! OSS projects require community efforts to correct identified open issues, and ideally audits like these are done collaboratively with task authors and maintainers directly, in the traditional community spirit that has kept OSS alive for decades.

As the online hype around AI data brokering hits a fever pitch, a lot are missing the basic distinction between - (A) Unrefined data for pre-training: I.e. raw company records (potentially de-identified) - (B) Refined data for post-training/eval: I.e. well-formed, high-quality tasks, envs, rubrics based on private data or human traces (A) is low margin brokerage - and will commoditize rapidly, esp as pre-training data curation techniques mature and increase discriminative capabilities for buyers. (B) is an incremental (but exciting) evolution of the sophisticated frontier data/envs that AI data labs/vendors already build. Margin/differentiation here will only increase as the frontier continues to advance, raising the quality and complexity bar.

Thank you to all of our speakers!! Some great chats with @fchollet, @dawnsongtweets , and many others at the @SnorkelAI Frontier Data Summit about the future of AI capabilities; benchmarking / evals; and AI data / environment development. Check it out!

Very cool result from the @SnorkelAI research team on how to adjust GRPO to effectively improve recall on RL-tuned models for long horizon tasks (esp impactful in domains with available verifiers!)

Congrats @modal @bernhardsson !! Well deserved (and thanks for the kind @SnorkelAI feature!)