Why didn’t people in finance pay attention to Benoit Mandelbrot?

October 18, 2010

Mathematician Benoit Mandelbrot was not one of those great thinkers who was ignored in his own time. He won lots of prestigious prizes. He wrote acclaimed books. He even gave two TED talks.

But it’s curious how little of the acclaim and attention Mandelbrot received over the years, and after his death last week, came from the world of finance. Mandelbrot was, believe it or not, one of the founding fathers of modern quantitative finance. In the early 1960s, he and scholars at Harvard, MIT, Chicago and a couple other places began to explore the meanings of random walks in stock prices. (I spent several years immersing myself in this history for a book; hence my obsessive interest. Here’s an excerpt from it related to Mandelbrot.)

In the early days, Mandelbrot was very much one of the random walk gang. He considered Eugene Fama, then a grad student at the University of Chicago, to be his student and protégé. A 1965 article by Mandelbrot in the Chicago B-school’s Journal of Business proved that a rational financial market would be an unpredictable one, providing an essential building block for what soon came to be known as the efficient market hypothesis.

Before long, though, Mandelbrot and the finance crowd drifted apart. It was partly just that Mandelbrot was a curious guy, and got interested in other sources of the fractal patterns that he saw in stock prices. But he also felt that “an ominous cloud” was developing in his relationship with the other random walkers. As long as quantitative finance was mostly exploratory in nature, Mandelbrot and the economists and finance professors got along fine. But as soon as the latter groups started trying to develop tools for understanding and managing risk in financial markets, there were tensions. The tool builders wanted to shoehorn market-price behavior into a bell-curve statistical model. That is, they wanted to believe that while price movements couldn’t be predicted, price volatility could. Mandelbrot thought volatility was far harder to capture than that. As Paul Cootner, a random walker from MIT’s Sloan School, wrote in 1964:

Mandelbrot, like Prime Minister Churchill before him, promises us not utopia but blood, sweat, toil and tears. If he is right, almost all of our statistical tools are obsolete . . . Surely, before consigning centuries of work to the ash pile, we should like to have some assurance that all our work is truly useless.

And so it went. For Mandelbrot, the crucial turning point came with the development—and widespread acceptance—of the Black-Scholes options pricing model in the early 1970s. Black-Scholes and the many financial risk models that have evolved from it (including Felix’s friend the Gaussian copula) are all about volatility being measurable and predictable. “When Black-Scholes came out, I said, ‘Well, it won’t last,’” he told me in 2005. “‘I’ll come back when it’s gone.’”

After the 1987 stock market crash, brought on in part by portfolio insurance strategies built upon Black-Scholes, Mandelbrot began paying attention to finance again, and some financial practitioners actually began paying attention to him. So he made a partial comeback. But the reliance on risk-management systems based on the belief that price volatility can be easily measured and predicted has continued, and it has continued to lead the financial system to the brink of disaster every few years.

So why haven’t finance academics and practitioners paid more attention to Mandelbrot’s warnings? I think it’s mainly that he didn’t provide them a handy alternative to Black-Scholes. I can’t pretend to fully understand the practical implications of his fractal view of markets (and yes, I’ve read his book for lay readers on the subject), but it does seem more useful as a critique than as a positive model of market behavior. You can’t haul in big consulting fees or create giant new securitization markets with a critique. So the natural tendency of both scholars and bankers has been to hold on for dear life to the Black-Scholes approach to modeling market risk. They get paid well for doing so, after all.

Finally, to switch topics entirely: Thanks to Felix for inviting Barbara and me to procrastinate blog at his place for the next couple of weeks. It should be fun. And it won’t normally be quite this wonky.

20 comments

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/

Great article, I also wonder what Nassim T. has to say about this subject.

Posted by Developer | Report as abusive

how in the world would you calibrate a fractal model to financial data series?

Posted by q_is_too_short | Report as abusive

“So why haven’t finance academics and practitioners paid more attention to Mandelbrot’s warnings? I think it’s mainly that he didn’t provide them a handy alternative to Black-Scholes.”

In some sense, he was saying that there was no alternative to Black-Scholes – that no model could exist that captured the behavior of prices, or even of their statistical derivatives such as volatility. As you point out, it’s hard to make consulting fees on that basis. Of course, if that’s true, it also means that the people who are pulling in those big fees are committing fraud, although mitigated since they believe that the models work.

Posted by KenInIL | Report as abusive

Traders have always, will always, engage in the straightforward game of, “Heads I win, Tails you lose”.

For those extended periods of time that their models seem to work, they reap big bonuses (customarily half the profits). When their models fail, they shrug their shoulders, blame “black swans”, and walk away from the back-breaking losses.

Why should they care if the models fail horribly once every 20 years? It isn’t THEIR money.

Posted by TFF | Report as abusive

Why should financiers and traders worry about “random” bad events when the federal goverenment stands ready to cover their losses if they are in danger of bankruptcy?

Unfortunately, few useful lessons will be learned from 2007-2009 because of this.

Posted by ErnieD | Report as abusive

I also read Mandelbrot’s book on misbehaviour and frankly he is talking about a different time. Alot quant models have jump-diffusions in them – particularly in the credit market where most of the issues have been – and having time as a random variable – his idea of having fast and slow trading times – is also a more or less standard technique. I don’t think anyone assumes constant vol anymore.

One forgets that in the midst of a crisis the like of what we saw in 2006-2008, value doesn’t matter. The prices are driven by supply and demand. The quant models also tended not to take into account operational issues such as collateral payments which bankrupted AIG.

Posted by Danny_Black | Report as abusive

In response to Danny_Black: Yes, most post-LTCM trading algorithms try to account for volatility. Problem is, it’s an impossible task. “Operational issues” such as those surrounding the Lehman collapse can’t be predicted by an algorithm and occur too frequently and with outsized effect on the market.

This time, the undetected “operational issues” revolved around the credit freeze. Before that, it was rising commodity prices, likely due to hoarding and speculation. And before that, it was 9-11, or accounting scandals, or panic in the currency markets. And that’s just in the twelve years these new models have been in use.

As Mandlebrot demonstrated, you can create Paretian distribution models that *simulate* these shocks in frequency and magnitude, but using proper assumptions they should have very little *predictive* value – even over the long run, since any Pareto-distributed fractal algorithm designed to predict large shocks every 5-10 years and massive shocks every half-century or so will also have gigantic margins of error. We’re talking unusable any-asset-could-reasonably-be-worth-zero -scale errors, with lumpy downside distributions and variable skew and kurtosis. Might as well use a dart board for trading strategies with an investment window of less than a few hundred years.

I don’t see how portfolio-theory-based algorithm trading can work its way out of this mess. In an apolitical world it would have been fatally exposed by this crisis. But in our world it will survive, at least until the next crisis, and probably beyond. Too many powerful people make fast millions doing it.

Posted by DaggaRoosta | Report as abusive

[...] –Mandelbrot’s great, and under-recognized, contribution to finance. [...]

I think the financial world did listen, but Mandelbrot didn’t have a silver bullet, or a way to make money, (what he was suggesting would’ve cost them profit) so they ignored/sidelined him.

Their loss, our gain, (or at least it was until they socialised the loses :)

The world has lost a truly brilliant mind.

Posted by praxis22 | Report as abusive

He wasn’t ignored — but you have hit the nail on the head. Follow the revenue stream, and you’ll find why the CAPM world, where risk is underpriced, pays the bankers/traders/marketers more.

Posted by quantacide | Report as abusive

Danny_Black has got it right. Mandelbrot was indeed a colossal figure, and something of an enthusiasm among mathematically inclined finance professionals. The reason that he had no influence on finance is that he never troubled to formulate any financially applicable theory. Lots of useful work has been done modeling volatility itself as a (not necessarily continuous) stochastic process; little of it owes anything to Mandelbrot.

The side rant about Black-Scholes is just from-outer-space weird and irrelevant. Nobody anywhere at any time has ever used B-S as a risk model, and nobody ever will. That is because it is not a model of risk. Nobody who uses B-S for relative pricing and hedging “believes” that the GBM of the model describes underlying asset behavior; that is self-evident from the presence of vol smiles/skews and surfaces. But B-S is not going anywhere, because when you can use it, it works really well. (When you can use it depends on the option you are pricing, not “fat tails” in the distribution of the underlying.) It has exactly the right number of free parameters, namely one. Even when guys like Dupire and Derman fit deterministic local volatility functions, they are not asserting that volatility is actually deterministic. These functions are best regarded as expectations.

Posted by Greycap | Report as abusive

One more point, the reason B-S also sticks around is because a lot of options get quoted in implied vol, which is the vol that if plugged into B-S gives the actual price.

There seems to be this narrative that somehow finance people have formed some sort of conspiracy to deny the True Financial Theory(TM) from replacing the guassian model whose sole purpose is to give traders money/extort money from the government. I can guarantee that if someone had anything that looked even vaguely like it could perform that he would have money throw at him.

Posted by Danny_Black | Report as abusive

DaggaRoosta, pre-LTCM people didn’t assume constant vol. At least in the early 90s people were using GARCH models for vol.

Not entirely sure what a Pareto distribution fractal algorithm means but I know of plenty of people who tried to apply different forms of chaos or complexity theory with very mixed results. Certainly they made less money less consistently than the toolkits you seem to be criticising.

I am obviously not saying that quant models today have reached complete perfection any more than say the Standard Model in particle physics has done but I was making a comment that what journalists seem to think banks use and what they actually use seem to have zero correlation. Far worse is this idea that somehow banks are **deliberately** looking to avoid having a good pricing model is simply insane.

Posted by Danny_Black | Report as abusive

praxis22, what was he suggesting that would have cost the banks profit? As I stated most of what he was suggesting is pretty much par for the course now.

quantacide, please name me one bank that uses CAPM in any significant way for interal risk management so i can put all of my wealth into shorting it.

Posted by Danny_Black | Report as abusive

Reply to Greycap and Danny_Black: I agree with just about everything you both have to say but I think it misses the point.

First, Mandlebrot didn’t give much aid to the field of practical quant finance because he doesn’t believe in it, not because he “wasn’t troubled” to do so.

And the problem lies with the point you’re both making: that B-S and similar strategies work very well when the market isn’t broken. Options, for example, are typically short-term instruments and the distribution of the underlying really doesn’t matter much in the context of a single trade since the distribution will be narrow regardless. And as long as the market behaves itself, it’s going to give you something close to a perfectly hedged investment strategy.

But in the medium run the market doesn’t behave itself. Correlations suddenly move toward 1 and the models break. And because most of the financial sector now relies on algorithmic trading strategies, there’s a serious risk that at least some systemically-important institutions will fall into sleepy optimism by fat profits and spreadsheets that say they’re perfectly hedged, and overextend themselves in the run-up.

So the problem isn’t that the models don’t work. It’s that they work too well most of the time. The drawback is that it’s impossible to quantify how much you can continue to rely on them during a large unexpected change in market behavior (that’s where the inflated variance of the underlying distribution matters – these events happen often and have huge effects) and their short term profit-maximizing features virtually guarantee over-reliance across the industry. That breeds both a tendency toward systemic weakness and a deeply inefficient first response.

Posted by DaggaRoosta | Report as abusive

And Danny: I agree with your last points. It should be perfectly understandable why traders use quant methods: they work well in the short run, and when they fail, it’s because everything else is failing too. There’s no need to project conspiratorial malice onto people.

But I do suspect that quants have spent a little too much time learning complex methods and not enough time understanding the fragile philosophy behind the methods. As a social scientist I have to constantly question my assumptions and anticipate all the ways in which my models could break. And they almost always break in practice. Markets are also a chaotic social phenomenon and in my opinion, deserve the same degree of statistical hesitance. But the quant strategies employed in markets seem a little too focused on firm-level results and miss the collective market-level impact of their behavior. Too many trees, not enough forest.

Posted by DaggaRoosta | Report as abusive

What DaggaRoosta said. I’m well aware that nobody out there is using 1965-vintage CAPM or 1973-vintage Black Scholes to manage risk, but I also get the impression most modern quantitative risk management models still rely on some of the same core assumptions as CAPM and Black-Scholes: that volatility can be predicted, that historical data helps a lot in these predictions, and that market participants are all price takers. Most of the time, these assumptions hold up. And then they don’t.

Posted by JustinFox | Report as abusive

DaggaRoosta, what you describe is just not how B-S is used. As a social scientist, you might want to consider observing how your subjects behave, rather than just imaging it and hoping for the best. Perhaps you should question your assumptions and consider how they might break.

The number one use of B-S is actually what Danny_Black said: converting a quoted vol into a dollar price, which kind of important to know. This quoting convention is not arbitrary; for some markets it is much more convenient (stable) to quote in moneyness/vol terms than strike/price. When it is used for pricing, it is to create consistency in a set of related quotes. Nobody is building up a huge gamma/vega position and just delta hedging it without thinking.

Posted by Greycap | Report as abusive

Greycap – I’m not a finance guy but I do understand that B-S is mostly used as a quoting convention and not as a means to perfectly hedge longshot high volatility bets. At least I’d certainly hope the latter is not the case. I’m concerned that the automated trading systems in use proceed from the same flawed assumptions, but I don’t have access.

In any case I don’t see how any of that affects my critique. Granted, I don’t think I voiced that critique very clearly or succinctly, and the point itself moved around a little, so let’s try again.

The complexity and opacity of quant methods as a whole, combined with a general track record of good short-term profitability, interacts with human nature to create a general tendency toward complacency about a financial firm’s capacity to handle potential problems in that market.

As a side note, if you want to convert volatility to price, you’ve got to use a Gaussian model to do it. It’s the only way. And I’m sure that’s fine, especially if you’re the only one doing it. But if everyone’s doing it in a particular market, using similar methods and the same volatility data, I’d be concerned with collective action problems. Because it seems to me you’d be inviting automatic feedback loops of artificial agreement (“creating consistency”?) where less agreement would exist otherwise, and and over time that would create distorted market data, worsening the problem. It may create actionable intelligence at the expense of unbiased intelligence, in other words. And when the bias is geared toward short-term contacts and profits, it seems to me that long-term bubble propensity in the markets should increase as a result. But that’s just remote pontification.

Ultimately I think the main cause of the crisis had more to do with behavioral factors related to quant models rather than the models themselves.

If you disagree, please let me know why, because that’s the alternative explanation of the crisis I’ve been using to defend traders from the critics who think they’re all bloodsucking dee-bags who sank the economy for the sake of a bonus bump. I prefer to think that financial types simply overestimated their predictive powers and failed to save money in the good times to protect them from the bad, just like the rest of us.

Posted by DaggaRoosta | Report as abusive

DaggaRoosta, actually i think the opposite of what you say is true. Over the medium term, arbitrage based pricing tends to work pretty well. The issue is short-term bursts, normally caused by some sort of liquidity squeeze which pulls market prices away from the underlying value.

The example I gave earlier was AIGFP notorious swaps which people seem to think started paying out because the reference obligations started to have credit events. Actually in most cases – possibly all – this simply wasn’t the case. What happened is that the market **PRICES** of the reference obligations declined triggering collateral payments for the swaps which then grew to a size that AIGFP got credit downgraded which triggered more collateral payments etc. Arguably the quant models were correct in valuing the swaps – and also irrelevent – and the issue was liquidity management.

Posted by Danny_Black | Report as abusive

I spent a weekend with Mandelbrot at a small group +10 people Finance retreat in Texas in 2001. He and his wife were polite, thoughtful and very interesting people. One of the things I respected about him was that he never allowed his name to be put on some strategy etc. He didn’t sell out, although lots of groups would have liked the marketing cache.

He was very dignified and helpful to some of the mathematicians working with us. Ultimately fractals and related processes such as volatility are nice ways of measuring process outcomes, but they tell one nothing about input processes for controlling risk.

Critical instability in systems and power law related extreme outcomes are common. If anyone is interested in speaking about these things drop me a line http://www.gogerty.com

The flash crash was a large event from the HFT process. there have been 20 smaller crashes since then indicating the process is still unfolding and likely to exhibit a large outcome at some point in time, similar or greater than the flash crash of may 6, 2010.

Posted by Nick_Gogerty | Report as abusive

[...] Sam Jones, FT Fat tails, tail dependence, and micro-correlations – Resources for the future paper Why didn’t people in finance pay attention to Mandelbrot? – Justin [...]