understanding and benchmarking vulnerability detection capabilities is an interesting challenge and I can recommend @niklas2484's recently published PhD thesis on that topic. it includes "Top Score on the Wrong Exam", one of my favorite (pre-LLM) ml-for-vuln-detection papers.

it's great that with decbench, we have a high-quality public benchmark. yet, decompiler evaluation remains hard, and I see two primary challenges right now: 1. high-quality, real-world-like, fully private data. Frontier LLMs have seen virtually every open source function in existence and leakage is not fully avoidable, something we also cannot rule out for our model. 2. finding a better proxy for semantic fidelity. A program can recompile, but do something entirely different. A program can also recompile, have a totally different byte-level structure from the groundtruth, and still do the same job. Dynamic-execution based attestation is tough to get correct at scale, but might be a better signal in the long run.

we submitted and verified our results via the public site: decbench.com/leaderboard/?da… (`normalize` as there is no arch-specific view right now) it's cool to see that our {static context collection -> one-shot decompilation} approach can compete with fully agentic frontier LLMs

today, we @RevEng_AI released the next generation of our decompiler: Ventris the new model comes with improved struct layout recovery, better PE support, and more. on decbench, we beat all other available decompilers in structure recovery, and achieve #2 rank overall