The limits of statistics

By Felix Salmon
November 16, 2011
post about statistics in Ontario, none more so than from Reihan Salam.

" data-share-img="" data-share="twitter,facebook,linkedin,reddit,google" data-share-count="true">

I got some very smart responses to my post about statistics in Ontario, none more so than from Reihan Salam. Picking up on my chart of median household income in Ontario and New York, Reihan responds by saying that, essentially, that particular game is rigged: Ontario has more married-couple households, even if it doesn’t have bigger households, and that will help explain at least some of the difference.

I think that Reihan’s right here, although I suspect that even if you adjust for household size, median incomes in Ontario are still going to be higher than in New York. But Reihan’s argument is essentially a subset of a bigger fact, which is that it’s never possible to make true apples-to-apples comparisons between two different states or countries — especially if you take into account things like “the particular historical challenges facing a post-slavery society”.

This is simply a basic fact of statistical analysis: making comparisons across time is a lot easier than making comparisons across space. So measuring GDP growth, for instance, is actually easier than measuring GDP, which is surprisingly difficult. As was demonstrated in a particularly startling manner last year:

To show that this is not an arcane point, consider the case of Ghana, which decided to update its GDP last year to the 1993 system. When they did so, they found that their GDP was 62 percent higher than previously thought. Ghana’s per capita GDP is now over $1,000, making it a middle-income country.

The fact is that when you’re comparing GDP per capita between two different states, there are just as many weird idiosyncrasies as there are when you’re comparing median household incomes. Ontario, for instance, has significantly lower GDP per capita than resource-rich states like Alberta, but that doesn’t mean that people in Alberta are better off than people in Ontario. And if Albertan GDP Is artificially raised by the energy industry, then New York’s GDP is artificially raised by Wall Street — something which does little good for poor families upstate or even people in New York City struggling with a cost of living which has been inflated by bankers’ bonuses.

And then when you’re comparing states with two different currencies, as we are here, you run into a whole other set of problems. In the comments to my post, “topofeatureAM” declared that “PPP is the correct way to compare stats cross border”, and in general that’s right. But PPP is really hard to measure, and I defy you to find a useful time series showing the purchasing power of the Canadian dollar over time. (The people at the conference I attended didn’t even try: they just decided that the purchasing power of a US dollar is 1.2 Canadian dollars now, and it basically always has been.)*

In the specific case of Ontario, there are actually good reasons to look at FX rates rather than PPP: the vast majority of Canadians live very close to the US border, and quite regularly buy their goods in the US. I work in Times Square, and there are lots of Canadians there on any given day, enjoying the purchasing power that the Loonie has over here. When the Canadian dollar appreciates, that really does result directly in a higher quality of life for millions of Canadians, even if Canadian exporters will predictably moan.

And if PPP makes sense — and it does, in many ways — then shouldn’t we be using it when we compare New York to, say, Illinois? The purchasing power of US dollar varies widely from state to state — is there some way of incorporating that into statistics?

The big point here is that if you’re comparing two different places, it’s silly to try to reduce them to a single datapoint like PPP GDP per capita. Even if that’s the best single datapoint — even if it’s better than median household income, or Rawlsian thought experiments, or anything else — it’s still going to be flawed in many ways. We should be looking at many more indicators, and we should be looking at distributions rather than medians or means. It’s the same point I was making about the online advertising industry: what we can measure and what’s important are not always the same thing.

Statistics can be illuminating, but they can also be misleading: we all know this. The trick is to try to get a feel for when they’re the former and when they’re the latter. And that kind of thing — often based on the old-fashioned smell test — is always going to be more of an art than a science.

*Update: For much more on Canadian PPP, how it’s measured, and how it changes, see this paper from Statistics Canada.

10 comments

Comments are closed.