m

ALT On Goodhart’s law, with an application to value alignment El-Mahdi El-Mhamdi ∗1,2 and Lˆe-Nguyˆen Hoang2,3 1 ´ Ecole Polytechnique, France 2Calicarpa, Switzerland 3Tournesol Association, Switzerland October 15, 2024 Abstract “When a measure becomes a target, it ceases to be a good measure”, this adage is known as Goodhart’s law. In this paper, we investigate formally this law and prove that it critically depends on the tail distribution of the discrepancy between the true goal and the measure that is optimized. Discrepancies with long-tail distributions favor a Goodhart’s law, that is, the optimization of the measure can have a counter-productive effect on the goal. We provide a formal setting to assess Goodhart’s law by studying the asymptotic behavior of the correlation between the goal and the measure, as the measure is optimized. Moreover, we introduce a distinction between a weak Goodhart’s law, when over-optimizing the metric is useless for the true goal, and a strong Goodhart’s