Goodhart law

November 13th, 2023 (11 months ago) • 2 minutes

Goodhart law states that - When a measure becomes a target, it ceases to become a good measure.

In other words, when we make a metric a target, we will tend to do whatever it takes to acheive that target, even if it means sacrificing other factors that determines success.

There are many valid examples that we can see in the world. For example

  • For students, they are incentivized to get good grades no matter the cost, hence they resorted to rote memorization and cramming for exams
  • For universities, they are incentivized to bump their rankings to attract more students, hence they resorted to gaming the system by optimizing certain metrics that US News looks at, e.g increasing number of class while lowering sizes, pushing lower GPA students abroad, encourage graduates to publish papers although quality is low
  • For companies, people are incentivized by how many tickets they close, hence they tend to choose easy tickets, and leave the hards ones behind

How do we then prevent this? A simple way is to have multiple metrics and keep them accountable for each other. For example, have one quantitative metrics and another qualitative metrics to balance the results. Being aware of the existence on this law is also good to prevent us from falling too deeep into it.