Ranking economic forecasts

August 27, 2009

Financial journalists spend a lot of time surveying market economists ahead of macro-economic data releases to find out how they think the next CPI or GDP number is going to turn out. A poll 20 or 30 economists gives a market median forecast, which will determine how traders react when the data comes out. If the figure beats expectations and points to a strong economy and likely rate rises, the currency will jump, and vice versa.

But how good are these forecasts? Why react if there’s no track record for accuracy? Economists have a pretty good feel for how reliable forecasts are for different indicators, but it would easier to have a number that tells us how reliable forecasts are for data such as GDP, jobs data or the CPI?

Forecast accuracy is a live topic in academic journals. There’s the MAE and the MSE, the sMAPE and the MAD/Mean ratio among others. Some measures depend on scale so they can’t be used to compare different series of data, such as GDP and the jobless rate. Using percentage error — the MAPE — can overcome this but it gives whacky results with outcomes of zero or near zero. One possible solution is to use the mean absolute scaled error – or MASE – suggested by Professor Rob Hyndman at Australia’s Monash University and colleague Anne Koehler from Miami University, Ohio in 2006.

The MASE measures how forecasters have performed against a so-called naïve forecast — simply forecasting that next month’s result will be the same as last month’s. The lower the result, the better the forecast. So 0 is a perfect forecast, while a score above 1 means the forecast is worse than a naïve forecast.

Applying the test to some Japanese economic indicators, we can rank forecasts of the different data series according to how much better they are than a naïve forecast. So from best to worst:

Industrial output Score 0.25 – Economists are very good at forecasting industrial production, which measures the output of items such as flat-screen TVs, automobiles and electric machinery. Apart from manufacturer’s own forecasts, economists can monitor export data, electricity usage and steel and auto output figures for clues.

CPI Score 0.29 – Deflation has been accelerating due to falling oil costs and weak domestic demand. National CPI tends to track Tokyo CPI, which is released a month in advance and forms the basis for forecast numbers.

Machinery orders Score 0.43 – Core machinery orders is a highly volatile series, which is seen as an indicator of capital spending in the coming six to nine months. Analysts are actually pretty good at forecasting its ups and downs, if not at getting the exact levels.

Household spending Score 0.78 – Household spending is a measure of consumption, which has recently been affected by the government’s one-time payouts to households. Economists track retailers’ sales figure, but the result is fairly poor.

No comments so far

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/