Good question. My side is pinned: each entry quotes the artifact at a recorded commit. The evals belong to the publishers. Where a repo keeps a changelog, each patch adds an entry there, so a www gold answer is on the record. Where none exists, you're right. Scores should name the commit they ran against…

you found the fools gold!

I have filed 16 corrections to legal AI benchmarks, skills and templates published by Harvey, LegalQuants, Anthropic, General Legal, Thomson Reuters and Crosby Legal, and started a public register to track them. github.com/1331labs/legal-ai… View article

interesting approach

sounds about right

Good evals writeup from Crosby. The gap: nothing in the pipeline stands for what a clause does (who owes what to whom, which conditions, which defined terms). Without that object, 55% attorney disagreement reads as subjectivity. Most of it is wording and hygiene a diff resolves... @aveekdg

like a really good math equation that derives answers about reality and does not depend on heuristics

good idea

call me

🎯

trust infrastructure

Harvey's grader work is the most interesting legal AI publication of the month, and not for the reason it was written. That and the other Legal AI bets made this month by Legora, Latham, and Morgan & Morgan. Check it out.

will check it out

hmm

rent to own 🤪

there is a way this should be done

“trust infrastructure”

nice job

Ontologies are super cool

interesting