Adit Abraham of @reductoai on why everything in AI is "state-of-the-art all the time," and how to find what actually works for you: "One of the unfortunate things about the AI industry... is benchmarks have become marketing over time." "There's a public benchmark out there that we tested where if you take the same documents and you add a little bit of noise, like you tilt it one degree, you add a little Gaussian blur, the provider that published the benchmark gets way, way worse than everyone else, but everyone else is kind of consistent." "Our recommendation is actually LLM as a judge is quite strong today if you're thoughtful about how you think about a rubric and how to approach it." "We want to see the industry move closer to people testing on their own data, with tools maybe to make that evaluation easier." "Instead of trusting a generic benchmark, you really should be at a point where you know what works best for you." @aditabrm

Introducing Exa 3.0: our largest ever improvement to search quality. Exa is now #1 on every benchmark we tested, from public evals to verticals like coding and GTM intelligence. It's the result of multiple research breakthroughs 🧵