The limits of statistics

By Felix Salmon
November 16, 2011
post about statistics in Ontario, none more so than from Reihan Salam.

" data-share-img="" data-share="twitter,facebook,linkedin,reddit,google" data-share-count="true">

I got some very smart responses to my post about statistics in Ontario, none more so than from Reihan Salam. Picking up on my chart of median household income in Ontario and New York, Reihan responds by saying that, essentially, that particular game is rigged: Ontario has more married-couple households, even if it doesn’t have bigger households, and that will help explain at least some of the difference.

I think that Reihan’s right here, although I suspect that even if you adjust for household size, median incomes in Ontario are still going to be higher than in New York. But Reihan’s argument is essentially a subset of a bigger fact, which is that it’s never possible to make true apples-to-apples comparisons between two different states or countries — especially if you take into account things like “the particular historical challenges facing a post-slavery society”.

This is simply a basic fact of statistical analysis: making comparisons across time is a lot easier than making comparisons across space. So measuring GDP growth, for instance, is actually easier than measuring GDP, which is surprisingly difficult. As was demonstrated in a particularly startling manner last year:

To show that this is not an arcane point, consider the case of Ghana, which decided to update its GDP last year to the 1993 system. When they did so, they found that their GDP was 62 percent higher than previously thought. Ghana’s per capita GDP is now over $1,000, making it a middle-income country.

The fact is that when you’re comparing GDP per capita between two different states, there are just as many weird idiosyncrasies as there are when you’re comparing median household incomes. Ontario, for instance, has significantly lower GDP per capita than resource-rich states like Alberta, but that doesn’t mean that people in Alberta are better off than people in Ontario. And if Albertan GDP Is artificially raised by the energy industry, then New York’s GDP is artificially raised by Wall Street — something which does little good for poor families upstate or even people in New York City struggling with a cost of living which has been inflated by bankers’ bonuses.

And then when you’re comparing states with two different currencies, as we are here, you run into a whole other set of problems. In the comments to my post, “topofeatureAM” declared that “PPP is the correct way to compare stats cross border”, and in general that’s right. But PPP is really hard to measure, and I defy you to find a useful time series showing the purchasing power of the Canadian dollar over time. (The people at the conference I attended didn’t even try: they just decided that the purchasing power of a US dollar is 1.2 Canadian dollars now, and it basically always has been.)*

In the specific case of Ontario, there are actually good reasons to look at FX rates rather than PPP: the vast majority of Canadians live very close to the US border, and quite regularly buy their goods in the US. I work in Times Square, and there are lots of Canadians there on any given day, enjoying the purchasing power that the Loonie has over here. When the Canadian dollar appreciates, that really does result directly in a higher quality of life for millions of Canadians, even if Canadian exporters will predictably moan.

And if PPP makes sense — and it does, in many ways — then shouldn’t we be using it when we compare New York to, say, Illinois? The purchasing power of US dollar varies widely from state to state — is there some way of incorporating that into statistics?

The big point here is that if you’re comparing two different places, it’s silly to try to reduce them to a single datapoint like PPP GDP per capita. Even if that’s the best single datapoint — even if it’s better than median household income, or Rawlsian thought experiments, or anything else — it’s still going to be flawed in many ways. We should be looking at many more indicators, and we should be looking at distributions rather than medians or means. It’s the same point I was making about the online advertising industry: what we can measure and what’s important are not always the same thing.

Statistics can be illuminating, but they can also be misleading: we all know this. The trick is to try to get a feel for when they’re the former and when they’re the latter. And that kind of thing — often based on the old-fashioned smell test — is always going to be more of an art than a science.

*Update: For much more on Canadian PPP, how it’s measured, and how it changes, see this paper from Statistics Canada.


We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see

Your second to last paragraph is something I absolutely agree with. But you didn’t do that. You dismissed the PPP chart because it disagreed with your hypothesis, and instead used an even more problematic analysis predicated on Nominal FX rates. Your “Rawlsian thought experiment” is actually probably the best way to think about your question anyway. And I pretty much agree with where you come out. that said I still think your analysis is sloppy.

I also think its a little bit of a reach to compare US and Canadian PPP/GDP calculations and their errors with issues estimating Ghanian GDP. Surely you must recognize that the level of statistical rigor in the analysis the Americans and the Canadians are doing is going to be leagues ahead of what the Ghanians were doing – simply because of the availability of data.

As for your points concerning the importance of nominal FX rates because of Canadian spending in US Dollars on a regular basis – lets look at the data wrt to day trips from Ontario 207/10464-eng.htm
a piece from ’07.
So lets figure there are 25 million day trips to the US by Canadians, and the paper says 80% of them are from Ontario, 20 million, and the paper lists spending as ~CDN$50/per trip or there abouts – so 1 bil CDN. There are 13 million people in Ontario, so on average each person in Ontario is spending roughly CDN$77/person in the US. The difference between the current FX rate and PPP is 20%. Is that really passing the materiality threshold?

And yes you should be using PPP when comparing across states. Just think what the lifestyle of someone making 150k in NYC is compared to making 150k in Atlanta.

Posted by topofeatureAM | Report as abusive

Sorry I misread the data – there were only ~13 million daytrips to the US from Ontario, not 20 million. Also that report shows the data for Internet Shopping from Canada to the US (300 Mil) and cross-border Car & Truck Purchases (1 Bil)

Posted by topofeatureAM | Report as abusive

Good points, Felix. Even among those of us who are sufficiently schooled to know that statistics are easily manipulable, too often we still take statistical findings at face value.

I, for one, am generally only skeptical (and therefore question) statistics when I believe there’s purposeful bias involved. This is a good reminder that even when obvious political points aren’t trying to be scored, there’s always the possibility of innocently overlooked lurking variables.

Posted by gmail3121 | Report as abusive

Felix’s topic is interesting; it is also a quite narrow way to look at “would you rather live here or there?”

@topofeatureAM: I agree that it does make sense to use PPP when comparing U.S. states – that what all of those “Cost of Living in SanFran vs. Reno” calculators are. I suspect most people that travel even occasionally think of PPP before FX rates. Intra-US, people know big cities are more expensive, and there’s not even FX to worry about.

However, as Felix has mentioned before, some people feel there is a great advantage to living in certain locations (NYC,SanFran) that makes up for the cramped expensive apartments, and that should go into a calculation about PPP. It’s not just about the prices of salaries, housing, groceries, movie tickets, etc. – it’s also about what intangibles that location has to offer.

It’s not how much you pay for equivalent purchases, it’s how much you pay for equivalent LIVES.

Would you rather live in the industrial part of town near the landfill, or by the river with parks, jogging trails, walkable to restaurants/retail, and an ampitheater that’s busy every weekend? After all, the former is cheaper. But I suspect more people would rather pay for the later.

How does one value, in a statistical sense, that cost-benefit? Manhattan can be expensive to live in, but it also provides public-good/intangible benefits that might not be available in other places, or might be more costly to obtain in other places.

But then you’re off of PPP-adjusted GDP, and onto other measures like Gross Happiness Indexes, trying to quantify human perceptions of subjective value, not just objective costs.

Posted by SteveHamlin | Report as abusive

Great post, with the tiniest of nit-picks: Alberta is a province, not a state.

Posted by Kosta0101 | Report as abusive

Alberta is an Independent and Autonomous Country, or at least it pretends to be so, even as it depends on the willingness of others to buy its unearned resources to support its lifestyle in a manner it will pretend is “in equilibrium.”

Posted by klhoughton | Report as abusive

“[T]he vast majority of Canadians live very close to the US border, and quite regularly buy their goods in the US.”

This seems, at best, a considerable overstatement. It all depends, I suppose, on what one means by “very” and “regularly.” There are only three metropolitan areas in Canada that are contiguous with the U.S, and two of them – Windsor and Niagara Falls/Welland – are quite small. Vancouver is the only major metropolitan area adjacent to the U.S. Working east, it is 200 miles to the U.S. border from Calgary, 65 miles from Winnipeg, 80 miles from Toronto, and 45 miles from Montreal. How often will people will drive a minimum of 100 miles round trip to save maybe 15 or 20% on purchases? Not very. While it is true that a great many Canadians live within 100 miles of the U.S. border, the number who live within 10 or 20 miles, a distance more conducive to frequent shopping trips, is actually very small.

Look at it also from the spending side. One of the previous comments quoted the amount of cross-border spending as about $1.3 billion. That works out to roughly $400 per capita. According to a 2002 survey by Statistics, Canada, average annual spending on goods by Canadians at that time was perhaps $15,000 per household per year (  /eng/ca02117.html#a91). Assuming an average household size of 3 (a generous estimate) and no increase in total spending since 2002, U.S. spending works out to be about 8% of the total.

Posted by MarkLI | Report as abusive

Is Mr Salam aware that slavery ended nearly 150 years ago?

Posted by Danny_Black | Report as abusive

“Is Mr Salam aware that slavery ended nearly 150 years ago?”

Institutionalized racism — the remnants of slavery — was with us into the 60s, and cultural momentum makes it difficult/slow to break entrenched patterns.

More African-American students are making it into (and through) college than ever before, but even today most of their parents are not college educated. You can’t overstate the importance of having role models to teach academic attitudes and success.

Posted by TFF | Report as abusive