Quantifying the damage of the rush to quantify

By David Callahan
February 2, 2012

It was unsurprising to hear, as we did Tuesday, that Claremont McKenna College had lied about its students’ SAT scores to boost its position in the U.S. News & World Report annual ranking of colleges. University officials are famously obsessed with these rankings, and this is not the first time that a school has admitted fudging data. Just last year, Villanova Law School said that it had given false information to U.S. News.

Today we quantify and rank the performance of people and institutions as never before – all in pursuit, supposedly, of better outcomes and greater efficiency. Yet this obsession with metrics, a hallmark of the free-market ideology, invariably creates more incentives to cheat.

University presidents fret endlessly about the U.S. News rankings because they can have dramatic effects on everything from the quality of student applicants to the ability of schools to attract faculty and raise money. In an earlier era, one free of U.S. News, schools would not have had much reason to lie about SAT scores or admission rates. But now, with these numbers seen as hugely important, you can understand the temptation to monkey around with the reported data.

It’s not just colleges. Metrics have also taken over our K-12 schools, and they’re worse off for it. The education world has been roiled by major scandals recently in which administrators and teachers have been found to have manipulated student scores on standardized tests. These have not been isolated incidents, with revelations of false reporting in Atlanta, Chicago, Washington, Birmingham, New York City and Los Angeles.

It’s no secret why so many teachers and administrators are fudging test numbers: The pressure surrounding high-stakes testing is evident enough in the phrase itself. Tests now determine how schools are funded and, in some places, which teachers get bonuses. Incentives to cheat are baked into this system, and those incentives will only grow if more localities link teacher performance to compensation.

Similar temptations are elsewhere, with more disastrous consequences. The ever-greater focus on quarterly earnings by public companies has partially led to a rising tempo of corporate scandals since the 1990s. Executives, fearing shareholder backlash, have cooked the books to prop up stock values – most sensationally in the gigantic Enron and WorldCom frauds. (The Japanese manufacturing company Olympus is now embroiled in a major accounting scandal of its own.)

Executives didn’t used to obsess much about quarterly earnings, according to former CEOs I interviewed for a book I wrote about the Harvard Business School class of 1949. Now these numbers dominate corporate culture and pervert both business decisions and the ethics of executives.

Other examples abound of how a growing focus on bottom-line results can bring out the worst in people. Perhaps no police department in the U.S. has relied more heavily on metrics than New York’s, with its famed Compstat program. And sure enough, testimony last year by a former New York City narcotics detective revealed that it was common practice among NYPD officers to fabricate drug charges against innocent people to meet arrest quotas.

The number-crunching crowd argues that stronger metrics lead to better outcomes, and certainly there are places where this is true. The Compstat program did help make New York a safer city by using data in new ways and holding police precincts more accountable for their performance. Better data on school performance has empowered parents and elected leaders seeking to choose the best schools and improve public education.

But, as many critics have pointed out, trying to quantify everything is questionable given the subtleties of the human experience. And the unethical things that happen when numbers rule should reinforce these concerns. Rankings and quotas aren’t the only way to motivate people to do better or measure what is happening. Softer qualitative indicators can also tell a lot about performance, and there is growing movement in various spheres to embrace these indicators as a complement to hard numbers.

Colleges have already begun to migrate away from the quantified life. More than 850 schools no longer require SAT or ACT scores as part of admission applications, and that number is growing. There have also been recurrent efforts to boycott or undercut the U.S. News & World Report rankings – most recently with a group of law professors urging deans at law schools to withhold LSAT data from U.S. News.

Meanwhile, there is greater awareness of how the tyranny of the bottom line leads corporations to behave badly. Reformers are working to change the laws around corporate charters to allow public companies to pay heed to a wider array of stakeholders and move beyond a fixation with short-term profits at any cost.

Finally, the Obama Administration is moving to dismantle the No Child Left Behind regime – which some have joked should be called “No Teacher Left Honest” – and create a more flexible way to measure schools that hinges less on standardized tests. While the administration’s “Race to the Top” program does stress rigorous quantitative assessments, it blends an emphasis on data with other factors like better preparation for teachers. Obama himself has pointedly criticized too much standardized testing.

The obsession with metrics rolled into U.S. society amid the triumphalism of free-market economics starting in the 1980s. It won’t go away easily, and, to be sure, there are still areas where better measurements are needed – like gauging happiness and quality of life.

But it seems as if more leaders finally understand that measuring everything is not such a hot idea. Let’s hope that the scandal at Claremont McKenna serves to reinforce this point.

7 comments

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/

The police department brings up another primary issue with metrics, which is if they will be used, significant thought needs to go into creating a metric that incentivizes the behavior you’re actually aiming for.

IE: a metric that purely rewards arrests is useless, as it can easily be gamed, and doesn’t actually achieve what you’re trying to reach – less crime.

Corporate culture is rife with this problem as well, particularly in areas where groups can create their own metrics to try and illustrate how wonderful they are. Most are fluff, like widgets produced, or quarterly earnings, that don’t actually achieve the important goal – a healthy, profitable company or a satisfied customer.

Posted by Araes | Report as abusive

Remember the old management mantra:
“Performance only improves where it is measured”?
An entire school of management has been built around this purported panacea. If only I had a dollar for every meaningless statistic I’ve been cajoled into reporting to supervisors when the time would have been better spent discussing observations and recommendations instead.

It’s only a few years since Léo Apotheker wrote a Reuters blog article explaining how SAP could help big business keep track of carbon consumption, for reporting under the new carbon credit markets. I pointed out at the time that no software would accomplish that goal effectively if businesses could populate their databases with any data they liked (or deliberately focus on aspects of their local carbon cycle most likely to be favourable) so as to “prove” the predetermined point they wanted to make. There’s also not much point if it just turns into an exercise in diving down a rabbit-warren of ever-increasing detail regarding geological/ biological aspects of the carbon cycle (the only people who would benefit from these approaches are those at SAP).

Sometimes we’re better off with an inexact approach, like IR spectroscopy via satellite digital imagery to detect and quantify large-scale carbon emissions. That would take a broad measurement that would be extremely difficult to falsify. My feeling is that the best possible approach is to combine these sorts of aggregate statistics with personal impressions and fully personalised communication.

Posted by matthewslyman | Report as abusive

Metrics are much of what we have. What we lack is the ability to evaluate these measures. We live in a culture that’s disinclined to question the numbers: how were they produced, what precisely do they reveal?

Rational folks have always been extremely suspicious of anecdotal evidence. This same level of inquiry should be applied to statistics, and the methods used to produce same.

Please forgive me, for stating the obvious.

Posted by RMoS | Report as abusive

I have seldom seen a conclusion more at odds with presented facts. In American philosophy, who is more revered than Horatio Alger?

Success is the Holy Grail of capitalism. It is deemed so desirable, like our Medal of Honor, that people incapable of achieving it will claim or steal it if they think they can get away with it. While it is impossible to change fundamental human nature, it is not only possible but necessary and desirable to study them so as reduce their effects upon our society.

In the 1980s America had the inspiration of Ronald Reagan after the despair of Jimmy Carter. Once more “we, the people” bought into the idea that we were exceptional and could do anything. You can’t be “exceptional” without comparison. You can’t be “successful” without comparison. Comparison is the inseparable companion of achievement. Comparison is also the inseparable companion of accountability.

Unfortunately America is a country run by bureaucracy. There is the Academic, or Educational bureaucracy, largely unionized. There are government bureaucracies at local, state and the federal level, largely unionized. Accountants are “private bureaucrats” of considerable influence. Executives “run” companies administered, by and large, according to bureaucratic principles.

The “tyranny of the bottom line” is but a euphemism for accountability. Notice how quickly the subject went from a negative to a positive? Words are important, but proper understanding and comprehension are even more so.

The last thing in the world the average bureaucrat wants is to be accountable in any manner. No average institution, no average professor, no average teacher, no average union worker, no average executive, no average police chief, no average military leader and no average politician will willingly subject themselves to the bright light of accountability so long as there is an alternative. But the excellent in any line of endeavor always welcome comparison.

The single weakness of comparison is in how what is compared and what is the goal are diddled. Too often the dull start with the answer they desire and contrive a “results-oriented” process backward from there. Lazy judges often pervert our legal process this way.

Conversely, it is devilishly difficulty for incompetents to muddy the water when a method is agreed in advance with which to equitably and accurately measure progress once the process begins towards a clearly defined goal. It’s like spray painting, where the most important and time consuming work takes place before you fill and pick up the spray gun.

So yes, each of us is solely responsible for “doing our homework” before reaching a decision or venturing an opinion. We each must be intellectually “accountable” or no one will take us seriously. Is not being “taken seriously” the pinnacle of credibility AND accountability? A good “bottom line” is a reward for good performance. It is “tyranny” only for those who would pass off bad performance for good.

A society that would be a meritocracy must think, speculate, innovate, compare, evaluate, decide, implement, observe, and repeat, ad infinitum. We need not more “metrics” but more understanding. We are no different economically than we are politically. Too many forget that the very survival of democracy is utterly dependent on “informed” voters. And no, I’m not about to try to measure THAT!

Posted by OneOfTheSheep | Report as abusive

its not the measuring, its the method that matters.
metrics are unavoidable. any institution that selects people must select based on some criteria. the criteria can be debated, but the fact that a selection based on some critera must occur cant be.

Posted by excrement | Report as abusive

Mr Callahan, it’s not a matter of too much metrics in schools, but TOO LITTLE ETHICS, which flows all the way down from the various “leaders” in this country. Other countries use metrics to gauge the success of their government programs without widespread cheating and self-dealing. We need to fire and prosecute all the corrupt individuals in this country starting at the top.

Posted by FredF99 | Report as abusive

[Counterpoint]: In favour of metrics: if you can set them up so that you can cross-check different people’s figures, and set up a system of incentivisation that works against collusion; you can sometimes get accurate metrics reported. In this case, metrics can be VERY useful, and you can do all kinds of useful analysis on them… Otherwise, my previous comments apply, and metrics should be regarded very skeptically.

Posted by matthewslyman | Report as abusive