Tag: goodhart’s overhang

Not, I should clarify right away, my own coinage. The term is due to Anthony Lee Zhang (@AnthonyLeeZhang on Twitter), and what a magnificent term it is.

We’re big fans of Goodhart’s Law in these parts, as you may have noticed. For the uninitiated, Goodhart’s Law says that once any measure becomes a target, it ceases to be a measure.

Say, for example, you want to show that the library in your college is being effectively used by all of the faculty members. That is, you want to show that faculty members visit the library regularly. Well, how about putting the attendance muster (it’s the 21st century, so let’s call it the fingerprint scanner) in the library? That way, we can “show” that all faculty members visit the library daily.

The measure (attendance) became the target, and so it has ceased to be a good measure, since it has been gamed. That’s Goodhart’s Law.

---

So what is Goodhart’s Overhang?

Anthony defines it as “sticking too long to old and familiar goal-metrics when the objective function has changed, and the metrics are no longer relevant”.

1. Grades matter in high school for getting in college. Some college students naively optimize for high GPA, assuming it's the same game as high school, in fields where networking/interview prep/github, etc matter more for job finding, and are surprised when this doesn't work

— alz – e/acc (@AnthonyLeeZhang) January 3, 2024

What should college students optimize for? Given the world they know in high school or junior college, students end up thinking that they should be optimizing for grades, when it is actually networking or preparing for interviews.

Goodhart’s overhang is twice as problematic as Goodhart’s Law in other words. Not only are you gaming a measure, which is bad enough, you’re not even gaming the relevant measure!

It’s not just students and grades, of course. Academicians churning out one meaningless paper after another, newly promoted managers working themselves too hard rather than delegating effectively, retail stores measuring success by counting the number of walk-ins might all be good examples of Goodhart’s Overhang.

One, what other examples can you think of in your line of work?

And two, your toolkit when it comes to Goodhart now consists of two questions, not just one:

1. Are you gaming the measure you’re trying to optimize for?
2. Is the measure still relevant in the first place?