Modelling model risk

By Felix Salmon
July 2, 2009

Paul Wilmott has words of wisdom for anybody in the financial-services industry who’s putting a model together:

At every stage of valuation and model development you must be asking questions about risk and robustness. It is dangerous to come up with some fancy model and only afterwards start asking questions about model error. Anyone who has ever calibrated a model knows that the methods used to mitigate model risk almost come as an afterthought, and are totally inconsistent with the original model. This need not be the case.

The problem is that developing a model is the sort of thing which (a) quants are trained to do, and (b) can, eventually, make money. While mitigating model risk is a very recondite field which very few people have any expertise with; what’s more, it doesn’t really make money in and of itself. Where will all the model-risk modellers come from?


We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see

I love the talk of financial ‘modeling’. Consider transistor models. They are fairly complicated but based on fundamental solid-state physics. When you build a complicated circuit, you don’t verify its function by builiding a separate model, you put in the individual transistor models and then simulate.

I would like to see models of MBS’s that instead of whole-modelling, use individual mortgage level models that are simulatable under different conditions. But if you work off the assumption home prices never fall, and never use that as an input, your individual mortgage models aren’t worth much.

Posted by winstongator | Report as abusive

What about ‘Climate Change’ modelling? Who’s checking these models”

Posted by Eli Baker | Report as abusive

Gee, who’s checking climate change models? Gosh, is anybody motivated to question them? Has Peabody Energy or Exxon Mobile funded any studies to question those models? But gee, the industry funded studies don’t seem to get accepted by reputable scientists! It must be a worldwide conspiracy!

Or could it just be that with billions already spent on challenging the climate change models, they are already fully tested and there’s no longer any serious doubt?

No, we can’t accept that conclusion.

Posted by Dollared | Report as abusive

The real challenge in questioning models is that there is no career path for naysayers. Simply put, preservation of capital should be a top priority for OPM managers, and best RISK-ADJUSTED growth paths the goal for business managers, but in my experience, nobody gets bigger budgets and promotions for telling investors and/or senior executives that the best path is lower risk and more consistent returns.

There are many unemployed 52 year olds in the technology industry whose sole sin was consistently maximizing returns in segments with high single digit or low double digit growth.

The ones still employed are promising to hit home runs in greater growth areas where the company has a one in ten chance of success. You do the math.

Posted by Dollared | Report as abusive

I don’t have doubt about climate change modeling, but I do question the veracity of the bullsh*t we get to hear from the Al Gores of the world and yet he wants nothing to do when it comes to honest, open debate from Bjorn Lomberg’s of the world.

I call it the “bully pulpit” syndrome…hell the world is still fighting famine & disease on at least one major continent. Another major growing country, India, lacks routine infrastructure like sanitation & clean water. Which battles promote greater economic good for the least amount of economic input?

Posted by Griff | Report as abusive


Point well taken. near me we have the Bill Gates Foundation, which will dramatically increase the survival rate of the world’s poor children. Of course, what about food and jobs for them when they grow up? Somebody else’s problem….

However, on climate change, the impact on the poor should be one of our biggest concerns, and one of the biggest reasons why we should cut down on our consumption of fossil fuels. 2007-04-06-01.asp

Posted by Dollared | Report as abusive

Increase the survival rate, the villages can adapt clean water / sanitation techniques & the next progression is growing their own agriculture (feedstock, support livestock). I don’t see the progression as necessarily requiring more 3rd world sewing factories or sneaker production facilities. Becoming a more self-sustaining econ model, and perhaps growing into regional co-ops to balance supply/demand could work. They don’t need a wall street, just maybe an active commodity exchange.

agree, it must be a balancing act..the economics for renewables just don’t seem to fit, or if they do as with wind turbines the good old “nimby” factor kicks in. who wants a 150-ft turbine blocking the views on cape cod, right?

Posted by Griff | Report as abusive

Griff, dude…

Didn’t anyone ever mention “the division of labor” to you? Why do you paint market policy and climate policy as mutually exclusive? There have been just as many smart physicists working in the climate field as there had been on the Street. Sounds like you want to disqualify everybody else’s expertise just because you’re a hotshot in one field.

Posted by a.k.a. | Report as abusive

I aint no hotshot at all, just debating ideas…which seems to be how these forums typically work. If I can bend my head around others ideas, perhaps there’s can bend around a few of mine as well.

If market policy and climate policy can intersect without egregious and/or superficial economic cost I’m on board. But as we’ve seen with ethanol mandates (Grow corn. grow corn now!) there is / can be decidedly after-effects which aren’t always fully considered on the interim prior to implementation.

Posted by Griff | Report as abusive