The Great Debate

Successful investment managers and state-of-the-art forecasters recognize that the future is, in some sense, unknowable and any predictions are subject to a wide range of uncertainty.

If outcomes can be described as a probability distribution, often the range and shape of that distribution are more important than measures of central tendency such as the mean or the median.

Yet too many commodity predictions (for prices, supply, demand and inventories) are still expressed as a single number or point forecast. They give little idea of how forecasters see the distribution of risks around that central or “most likely outcome. But for most users, the distribution of risks is far more important than the single number.

FORECASTING ERRORS

Focusing on a single point can give the impression of spurious accuracy. There is no evidence anyone can successfully and consistently predict the future this way — no matter clever they are or how many resources are thrown at the problem. Evidence of substantial forecasting “errors” abounds:

* The Bank of England’s Medium-Term Macro Model (MTMM) was criticised in the late 1990s and early 2000s for repeatedly over-predicting inflation rates, causing the Bank to hold interest rates higher than necessary. In 2005, MTMM was replaced by a new model (BEQM), specified in terms of more than 300 equations. BEQM has been criticised for errors in the opposite direction, systematically under-estimating inflation outturns since its introduction.

* Errors made by the Federal Reserve Board and 12 Federal Reserve Banks for both inflation and growth forecasts are well known, despite the enormous personnel and statistical resources they can bring to bear on the problem. The Board of Governors in Washington alone employs 450 economists, half of whom have doctorates. But the Fed’s predictive track record is not particularly impressive.

* Investment banks and other forecasters have fared no better. Bloomberg reported earlier this month that seven of the nine top trades for 2010 recommended by Goldman Sachs, the most famous trading house of them all, had lost money in the first few months of the year.

No one could accuse these top-flight forecasters of being stupid. Nor do errors suggest modelling is a waste of time; it brings much needed rigour to explaining the past as well as thinking about the future. But the elegant mathematics in which many models are written is fuelling an illusion they are more accurate than is really the case.

Arguably it is also breeding a dangerous over-confidence about the ability to predict and manage risk. It was the intricacy and apparent rigour of models for slicing and dicing risk that contributed to the disastrous loosening of lending standards in the U.S. housing market and elsewhere, leading directly to the crisis in subprime mortgage securitisations.

EXOGENOUS SHOCKS
Forecasters tend to claim their models work well in “normal” conditions, but that predictions can be blown off course by “exogenous” shocks such as the eruption of the banking crisis in 2007-2009, and should not be judged by this standard.

But normal conditions are subjective. Shocks of one sort or another (hurricanes, wars, market corrections, liquidity crunches, oil spills and recessions) are commonplace. Forecasts that exclude them and simply project present trends into the future miss the point. For users, turning points and sources of volatility are as important as trends.

Rather than treating shocks as “external” to the model, the best forecasts bring them inside by recognising the future is inherently uncertain. It is the product of a complicated interaction among multiple shocks, from large (credit crunches, hurricanes) to small (missed deadlines, slower than expected growth, slightly warmer weather). Forecasting is inherently stochastic rather than determinative.

Each factor in the forecast can take a range of values with a different set of probabilities. The challenge is to identify probability distributions associated with each factor, then model and predict how they interact with one another to produce a range of possible outcomes for the forecast element, each with its own assigned probability.

State of the art forecasts are already produced in this way. Most medium and long-term weather and climate predictions are expressed in terms of probabilities. In monetary policy, the Bank of England has been publishing forecasts for inflation and output growth in the form of density functions (“fan charts”) for more than a decade, showing a range of possible outcomes and the probability that the central bank attaches to each of them.

The fan is not perfect. The Bank’s models are still throwing out large (and biased) estimates for inflation, but at least it draws attention to the range of outcomes as well as the central tendency, and highlights the degree of uncertainty about them, and any imbalance between upside and downside risks.

Encouraged by Federal Reserve Chairman Ben Bernanke, the rate-setting Federal Open Market Committee is increasingly presenting its own summary economic projections in terms of a distribution of the different forecasts made by the Board of Governors and each of the Federal Reserve Banks, recognising the inevitable element of uncertainty and different views among the participants.

AN OPTION WORLD

The stochastic nature of markets and probabilistic nature of forecasting is implicit in the valuation theories used to price options. Buyers of six-month oil options are not taking a deterministic view of where crude prices will go by year-end, but a probabilistic one of where they might go, based on assumptions about volatility.

Most end-users of physical commodities and derivatives already think this way. They may use a single number as a simplified planning assumption or baseline. But the essence of hedging strategies is uncertainty about the future and the need to manage risk associated with particular (adverse) outcomes which are possible but not certain.

An airline thinking about buying upside calls to hedge its exposure to a sudden rise in jet fuel prices needs to consider not only whether prices might increase and how far, but what is the likelihood of any particular rise, and how it compares with the probability of prices declining and the cost of the option itself.

Forecasts without probabilities are of little use.

Probabilistic thinking is hardwired into commodity markets. Yet too many forecasts are still expressed as a single point or “average” outcome with little discussion of upside and downside risks, the balance between them, and the shape of the underlying probability distribution. And tail risks can be significant, as any investor in subprime CDOs has learned at painful cost.

Presenting a point forecast (“oil will reach $200 by 2014″) in a probabilistic world is a bit like playing a game of pin the tail on the donkey. It might turn out to be right by accident or design. And hides a huge amount of (costly) uncertainty which is expensive to hedge (for corporate users) and eats into risk-adjusted returns (for investors).

The probabilistic nature of the future, and predictions about it, as well as the apparent increase in volatility will continue to drive the market to a more stochastic forecasting approach.

The next few years will see a gradual, perhaps reluctant, migration from point forecasts and simplistic scenarios (central, up, down) to a more nuanced approach focusing at least as much on the dispersion of outcomes and the factors that drive them as the “average” or most likely one.