Goodhart’s Law is the idea that when a measure becomes a target, it ceases to be a good measure.
Maybe this sounds familiar: a company has a metric that they want to improve. So they decide to use that metric as the target for what they’re trying to improve. But without realizing it, they’ve just fallen prey to Goodhart’s law.
In this article, we will discuss Goodhart’s law in different fields, especially in survey research, and how you can avoid it.
What is Goodhart’s Law?
Goodhart’s law is a principle in economics that states: “When a measure becomes a target, it ceases to be a good measure.” The principle, proposed by economist Charles Goodhart in 1975, arises when an economic policy variable (e.g., the money supply) that was initially used as an indicator of the economic situation (e.g., to signal appropriate policy action) becomes a target for direct policy action itself.
So, direct policy action is then taken on what was previously only an indicator of the economic situation. This can make the indicator lose its informational value and thus its usefulness as an indicator of the economic situation.
Let’s say you have a measurement of customer satisfaction, and you want those scores to go up. You then decide that you’ll reward the people who get high customer satisfaction scores. However, with this, your employees will do whatever they can to get high customer satisfaction scores, even if they have to manipulate the customer in some way. Once this happens, customer satisfaction scores may cease to be a good measure of happiness.
Another example is if you tell employees in your company that you’ll be measuring their success by how many hours they spend at their desks. While they’ll definitely spend more time at their desks, does it actually indicate they’re doing a good job? Maybe not.
Read: Nominal, Ordinal, Interval & Ratio Variable + [Examples]
Goodhart’s law says that what you measure might not be a direct representation of how well things are actually going. So when you’re trying to improve performance in an organization, it’s important to regularly check what your metrics are measuring so you don’t end up confusing progress with actual improvement.
Implications of Goodhart’s Law
Let’s see an example – Let’s say you’re a manager and you want your employees to work together better as a team. You know this is an important part of their success in meeting company goals, so you decide to make it an official KPI.
You show them how they’re doing on this KPI, and tell them that if they don’t improve they’ll be fired. They start working together better as a team.
To make sure they look like they’re working together better as a team, they start covering for each other’s mistakes, so no one ever looks like they’re slacking off. The problem is that because covering for each other means ignoring problems instead of solving them, the company ends up with bigger problems. Employees can’t also trust each other enough to be honest about where their work is falling short.
The implications of Goodhart’s Law are numerous and potentially quite serious. When something like this happens in one area, we often don’t notice or care but if it begins happening all around us in many different places at once, it could lead to some pretty major problems for society as a whole.
Instead of using metrics as an end goal or a stick to beat your subordinates with, use them as feedback that you can take action on. You want to see what numbers are improving and what numbers aren’t doing so well so you can see where there’s room for improvement on both sides of the equation.
This will help you cultivate a culture where employees are encouraged to do their best work and grow, rather than just meeting their goals for fear of termination.
Effects of Goodhart’s Law on Education
Goodhart’s Law states that when a measure becomes a target, it ceases to be a good measure. In other words, as soon as you start using measures to try and affect outcomes, those measures become less useful.
This is particularly true in education, where there are several measures of student and school performance that are used by the government. This can cause teachers and administrators to focus on these measures rather than on the actual education of their students.
Students can also feel pressure to perform on particular metrics that doesn’t reflect their knowledge or skills. Goodhart’s Law is a problem because these measures aren’t perfect.
Some students do much better on standardized tests than they would do in real-life situations, while others struggle with them despite having in-depth knowledge of the material being tested. Also, schools that have large numbers of students from low-income families tend to perform poorly on standardized tests than schools with students from more affluent backgrounds.
For example, let’s say you are using test scores to evaluate how well a student or a teacher is performing. Test scores can only tell you so much about what students know, but they’re still easy to measure.
Test scores are also easy to compare over time and against benchmarks set by outside organizations. They’re useful for evaluating the quality of education at a school or within the community as a whole.
But if you decide that test scores are the most important way of measuring student performance, then students will focus more on simply passing tests than on learning the material. Teachers will find ways to gamify the system which may improve their students’ test scores but not actually help them learn. And schools might start teaching to test instead of teaching students how to think or solve problems.
Effects of Goodhart’s Law on Marketing
Marketers frequently run into the effects of Goodhart’s law when they focus on metrics that are easy to track and which look like they might be good indicators of success, without looking at how these metrics relate to the end goal. For example, if you’re running an e-commerce store and you want to increase your sales, you might decide that increasing your average order value is a good way to accomplish this goal.
If you start offering discounts or freebies with every purchase when the customer spends more than $100, it might work and you could achieve your target. But did pushing for that number actually increase your overall sales? Maybe not.
It could be because there was no real demand for those purchases among your customers. Or, you could have been pushing them too hard for the sale and lost some potential business.
In other words, if you use something as a metric to determine how well your marketing is doing, then it will stop being an indicator of success. Another example is if you have an e-commerce website where customers can purchase goods using credit cards, and you use revenue as one of your KPIs to determine how well your business is doing, then you might make the mistake of minimizing the friction involved in making a purchase.
You may remove security checks that make sure someone isn’t trying to use stolen credit card information or reduce the number of steps required in order to process a transaction. This can lead to a short-term increase in sales (and revenue) but at the same time, it could be enabling fraud.
Effects of Goodhart’s Law on Survey Research
Goodhart’s law is a measure of the reliability of quantitative data. It states that once a measure becomes a target, it ceases to be a good measure.
In survey research, this can lead you to collect data with the goal of proving a hypothesis, even when you have reason to believe that your hypothesis could be wrong. In addition to potentially misleading yourself and your readers, Goodhart’s Law could also lead you to a negative view of survey research in general, particularly from non-researchers.
In survey research for a company, once an aspect of a company is measured and used as a target for improvement, the validity of that measure goes down. For example, if a company measures employee satisfaction and then uses those results to improve or change things, the results will no longer reflect true employee satisfaction.
This is because employees will change their answers in order to provide feedback that reflects them in the best possible light. They don’t want to seem ungrateful or unreasonable by giving answers that lead to negative changes in the workplace, so they hide their true feelings and give answers they think will result in more positive changes.
Examples of Goodhart’s Law
Example 1
If you are working on developing software for an e-commerce site and a quality assurance team comes in to test your code, their metric might be the number of bugs they identify. However, if they make that the goal, they stop measuring how well the code performs its intended function and instead focus on how many bugs they can find.
A secondary effect of this is that new bugs will inevitably appear, and they will only be found once they’re already affecting customers because no one is looking for them anymore.
Example 2
Let us say a university president wants to increase the number of students who graduate from the school. The president might then decide that graduation rates should be measured and used as an indicator of success.
If these changes were made, faculty may then focus more on making sure students graduate rather than teaching them general skills they will need after graduation. So, faculty members may become more focused on boosting the graduation rate rather than on improving students’ education.
If this happens, then the metric has lost its effectiveness because it is no longer measuring genuine improvement in education it’s only measuring graduation rates.
Example 3
Let us assume a company decides it wants to increase customer retention rates, so it starts measuring how long customers have been with them and how many new customers they get in each month. While this could be effective at first, eventually that metric will stop being useful as employees start focusing solely on keeping old customers and not acquiring new ones (because this is what is being measured).
How to Avoid Goodhart’s Law
Here are some ways to avoid Goodhart’s law:
- Try to identify the variables. If you can observe them, then you’ll be able to see if they’re changing. If you can’t observe them, try to find other ways to measure them. This will allow you to see if they’re changing.
- If you still can’t observe them or measure them in any way, then try to figure out what’s going on by looking at the data itself and looking for patterns that might indicate there’s a problem.
- Measure outcomes, rather than actions and make use of multiple measures
- To create metrics that are actually useful for evaluating your policy, try using variables that are as specific and detailed as possible.
- Emphasize long-term goals and be wary of incentives that are too powerful
Conclusion
The concept behind Goodhart’s law is simple enough. When you are focusing your efforts on meeting a particular goal, whether that goal is traffic to your website, sales of your product, or any other metric, you begin to change the way you do things in order to meet the goal.
It is, however, important that while you focus on a goal, you take note of the other aspects of your survey or research so that your whole research does not suffer setbacks