Why official statistics are like corporate earnings

By Felix Salmon
August 30, 2010
Mark Gimein uncovers some statistical legerdemain at the Harlem Village Academies, a pair of middle schools in New York. Let's see how good you are at middle-school math, by seeing whether you can answer this question correctly.

" data-share-img="" data-share="twitter,facebook,linkedin,reddit,google" data-share-count="true">

Mark Gimein uncovers some statistical legerdemain at the Harlem Village Academies, a pair of middle schools in New York. Let’s see how good you are at middle school math, by seeing whether you can answer this question correctly.

A middle school runs from 5th grade to 8th grade. In the 2006-7 year, it had 66 students in its fifth-grade class. In the 2009-10 year, 100% of its eighth-graders were proficient in math. If all the eighth-graders who passed their math test were members of the fifth grade in 2006-7, how many eighth-graders passed their math test this year?

If you said 66, you get an F. If you said “I don’t know, probably a few of those fifth-graders moved away, maybe 60?” you get a D. If you said 19, you get an A.

Of course, the public announcements from the Academies are all about that 100% math-proficiency rate, and make no mention of the fact that clearly you’re only allowed to even enter eighth grade in the first place if you’re going to pass that test.

Gimein puts his finger on the broader syndrome:

Steve Koss, a former math teacher at Manhattan’s Lab School, paints the city’s relationship with stats as an example of Campbell’s Law at work. Named for sociologist Donald T. Campbell, the precept holds, essentially, that the more that numbers are used for political purposes, the more they will be manipulated—and distort the decisions they were supposed to inform.

So whenever you see a politician citing some datapoint you never much heard about before, be very suspicious. And the same goes double when money’s on the line: if inflation-linked or GDP-linked bonds ever become a large part of a government’s liabilities, expect shenanigans surrounding the official inflation and GDP figures. Statistics can be useful, but only if nobody much cares about them. In that sense, they’re a bit like headline corporate earnings.

More From Felix Salmon
Post Felix
The Piketty pessimist
The most expensive lottery ticket in the world
The problems of HFT, Joe Stiglitz edition
Private equity math, Nuveen edition
Five explanations for Greece’s bond yield
Comments
9 comments so far

I’ll bite, how does anyone know how many 5th graders of one sample group made it into the 8th grade sample group? It doesn’t seem like enough info to make that determination. Is this a case of lies, damn lies, and statistics?

Posted by MikeStover | Report as abusive

Yes, exactly. The statistics given (100% passing rate) are misleading, and you aren’t given the other necessary information (dropout rate) to see why.

Posted by TFF | Report as abusive

How on earth do you arrive at 19?

Posted by STORYBURNthere | Report as abusive

Charter schools like HVA have greatly added to the data cooking of educational statistics.

Posted by MarkPalko | Report as abusive

Government statistics are so bad that people have started entire business doing nothing but re-evaluating and correcting them: http://www.shadowstats.com/

Probably the biggest government statistics boondoggle over the past 30 years is the manipulation of the US CPI including imputed rents instead of house prices and substitution. Inflation was probably overestimed before 1995 and underestimated after 1995.

Posted by ErnieD | Report as abusive

Never allow yourself to get caught up in someone else’s mental maelstrom ..

Posted by Woltmann | Report as abusive

This is a great piece.
Sadly,in the UK and France every statistic is boiled, fried and schnitzeled before it makes it into the media. We have the ONS (an independent body) and so politicians just tell it to stop recording the bad data and look at other stuff which is better.
Our police have been caught falsifying crime stats seven times in nine years.
Nobody over 55 believes the inflation rate.
I certainly don’t believe the price of gold reflects a real market.
And there never was a ‘recovery’ on either side of the Pond: there was QE – but that’s something else.
http://nbyslog.blogspot.com/2010/08/bern anke-in-wyoming-blind-mans-bluff.html

Posted by nbywardslog | Report as abusive

“How on earth do you arrive at 19?”

It’s not a logic problem, they’re real numbers. Inbetween 07 and 10, 47 students left or were made to leave because they wouldn’t pass the test.

Posted by drewbie | Report as abusive

I think your question is a bait-and-switch also. You asked “…how many eighth-graders passed their math test this year?” The answer is 100%, as stated earlier; 100% were proficient. What you were trying to ask, and either didn’t state cleary or intentionally fuzzed the question, is how many of the fifth graders in the 2006-2007 class passed the 8th grade math test. That answer we can’t determine without knowing transfer/dropout rates.

Posted by amygdala17 | Report as abusive
Post Your Comment

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/