The limits of the scientific method in economics and the world

By Roger Martin
November 10, 2011

By Roger Martin
The opinions expressed are his own.

This is part one of this essay. Read part two here.

As the economy teeters and the capital markets gyrate, I can’t get out of my mind the evening of May 19, 2009.  We were near the stock market nadir and fears were cresting that we were heading straight into the next Great Depression. I was invited to a dinner along with half a dozen tables of guests to hear a very prominent macroeconomist opine on the state of the economy and the path to recovery.

The economist held forth with a detailed, analytical account of what had caused the economic meltdown in the second half of 2008 and the path that he predicted recovery would take. I was struck by how scientific he was, spewing myriad statistics, employing technical terms by the boatload, and praising his econometric model. It was ‘very sophisticated’.  Given the nods and encouraged looks in the room, it seemed as though he had provided great comfort to the guests; they could go to bed confident that thanks to his science, they could trust that this man knew where we were headed.

I wasn’t quite so confident. Being the curious sort, before coming to dinner I had checked his forecast from a year earlier, mere months before the crash.  His spring 2008 forecast for the second half of 2008 was for modest positive economic growth for America.  This was not unusual; no credible economist predicted anything less rosy for the back half of 2008, although many now claim that they did.  I don’t blame or ridicule him for being cautiously optimistic mere months before the worst economic downturn in 80 years.  Economic forecasting is fraught with peril.

For me, the striking thing about the evening was that nothing changed about his models after they were shown to be hopelessly wide of the mark.  He just loaded up the equations, dumped in the latest numbers and started crunching away.  I asked him whether he had altered his models in the wake of his dreadful forecast of 2008; stunningly, he hadn’t thought of the question.

It struck me then and still does that this dinner is illustrative of a fundamental blind spot in modern science.  It has ventured far afield of its natural limits and is both creating problems and inhibiting progress.

The roots of the problem can be traced right back to Aristotle, the father of modern science, who around 400 B.C. laid down the first formal conception of cause and effect.  While there have been numerous advances to his thinking methodology, his fingerprints all still on everything a modern scientist does.

But as my Aussie philosopher friend, Tony Golsby-Smith has pointed out to me, as much as Aristotle was a proponent of his scientific logic, in the best scientific tradition, he established boundary conditions for his theory.  It was for the part of the world in which things could not be other than they are. An oak tree is an oak tree and cannot be something else.  A piece of granite is a piece of granite and can’t be something else.  For this world, Aristotle laid out the seminal scientific method and argued that it was the optimal way for understanding that part of the world.

As he laid out his scientific method for the part of the world that cannot be other than it is, he also cautioned that there is another part of the world that can be other than it is, and there was another method that needed to be used to understand it. The scientific method would be wholly inappropriate.

That part of the world consists of people – of relationships, of interactions, of exchanges.  In this part of the world, relationships can be good, bad or indifferent; close, distant or sporadic.  They change – they can be other than they currently are. For this part of the world, Aristotle said that the method used to develop our understanding and to shape this world is rhetoric; dialogue between parties that builds understanding that actually shapes and alters this part of the world.

I would argue that as the modern scientific world has embraced Aristotle’s science, it has utterly ignored his boundary condition and routinely pushes the scientific method far past the limits for which it was designed. Ironically, we have adopted Aristotle’s tool and then ignored Aristotle’s user manual.

In this respect, the economist’s behavior is completely unsurprising. In what he imagines to be proper Aristotelian fashion, he treats the economy as if it cannot be other than it is and predicts a future that is based on the past.  And when it is anything but, he returns to the same tools to do it again, believing that in doing so he is being meritoriously scientific. He is joined in this behavior in great numbers by people across the natural and social sciences, even some in the humanities.

Science advances our knowledge of the world in which things cannot be other than they are.  But the modern practice of applying science to the vast tract of the world where things can be other than they are is unhelpful, as demonstrated by the unreflective economist. Extrapolating the future to be a straight-line projection of the past is neither accurate, nor is it helpful in creating better understanding and newer ideas.

As much as it is helpful to the world to create, test and prove out novel new hypotheses about things that cannot be other than they are, I would argue it is more critical to the world to create novel new hypotheses for things that can be other than they are – like economic growth, environmental sustainability, and peace and security.

To do so, we have to break the iron grip of science on the part of our world that for which mere extrapolation of the past is ineffectual, for which the creation of a better future must be the goal. Part two of this column will introduce the American philosopher who provides a blueprint for a better way.

This is part one of this essay. Read part two here.

12 comments

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/

This article feels much like F.A. Hayek’s argument of whether economics is a legitimate science at all (The Pretense of Knowledge, Nobel laureate in economics).

While Hayek’s followers are not considered “credible economists” for refusing to accept econometric models, they were the ones accurately forecasting the back end of 2008.

Posted by Summerland81 | Report as abusive

Refer also to an excellent piece in Reuters recently, “The Physics of an Economic Crisis”, by Emanual Derman. He covers much of the same ground but from the fascinating perspective of a quantitative analyst.

I posted on that column that the physics of economics is much like meteorology, only far more complex. And it is not just the complexity which will bedevil any attempt at accurate forecasting, it is the total lack of scientific understanding of many of the fundamental causes and effects.

Until the most recent crisis, the most basic cause and effect went ignored–that of human motivation. It was basically assumed that all human actors participating in an economic situation would make logical decisions based on their economic well-being. This would be uproariously funny if only so many millions of people weren’t now suffering so badly from the miscalculation. Humans…acting rationally?!

So the underpinning of most models needs to be thrown away and replaced with, what exactly? It’s easy to state that humans are irrational. Yet it is very hard or perhaps impossible to chart our irrationality and feed it to a computer. Brain imaging is mapping our brains and showing impressive new evidence of how and why we think certain thoughts. But the progress so far is like trying to map the most intricate cave system in the universe with one person and a flashlight. It will take awhile to learn even a tiny fraction of what’s in there.

So we are left with a problem. We both need economic analysis and we need to understand the limits of economic forecasting. Even your weatherman can’t tell you with any degree of accuracy what the weather will be like in a week. We should expect even less of economists.

Posted by BajaArizona | Report as abusive

We’re too optimistic, that’s why casino business has been growing! We’ll grow/spend/tax/gamble our way out?

Posted by DrJJJJ | Report as abusive

Over all of history written, I agree that the “world of people” can change. But such change is relatively minor…being limited to individuals and their values and goals.

Those individuals at the helm of human affairs, however, rarely change. Those who become like Alexander the Great, Gengis Khan, Caesar, powerful Kings, the Kaiser, Hitler, Stalin, and our Presidents like Washington, Lincoln, Teddy Roosevelt, Franklin D. Roosevelt, Reagan are men whose values and goals are forged progressively over many years. Indeed it is the sheer mass of such accumulation as it accelerates the roll down the hill of destiny that draws more and more into it’s orbit and ultimate influence on humanity exploding.

We are battling within ourselves as we meet life’s challenges whether it will be our emotional self or our logical self that has the wheel as each curve must be negotiated. Similarly is the ever-present battle between our “selfless” self and our “selfish” self and it’s context. No wonder so much of life, in retrospect, looks more like a crap shoot with history written always by the winners.

I, too, read and posted here on “The Physics of an Economic Crisis”, by Emanual Derman. My “take” was quite the opposite of BajaArizona’s, as were my comments.

I concluded that “Mr. Derman shares with us some specifics as to what economists CAN’T do and admitting that this is pretty much everything they have attempted to do. They KNOW they don’t know what they’re doing. Duh? No one will pay for what another can’t do. Money is made by knowing with some certainty what another will or must do. The challenge is for economists to become capable of doing the function they have yet to fulfill.”

It is satisfying to read BajaArizona’s admission above that “We should expect even less of economists”.

Posted by OneOfTheSheep | Report as abusive

The fundamental problem with modern, i.e. Marshallian and after, economics is its assumption that subjective preferences may be maximized in the absence of objective benefits. The current theory is based on indifference analysis on the presumption that satisfying a subjective preference satisfies also objective needs. That is not the case.

Posted by larrymotuz | Report as abusive

I don’t have to wait for the second half of this article to say that the author has got his facts wrong. He writes “This was not unusual; no credible economist predicted anything less rosy for the back half of 2008, although many now claim that they did.” Among the economists who predicted exactly what he says they did not predict are : Joseph Stiglitz, Paul Krugman, Mohammad El-Erian, Nouriel Robini and several others. Yes, Economics is not an exact science; but it has analytical rigour that may sometimes outperform Physics. At the microeconomics level, Kenneth Arrow’s Impossibility Theorem is an example of the kind of rigorous analysis whose power surpasses that of Physics. Joseph Stglitz pioneered disequilibrium economics in a 1964 article that became the basis, from the early-1970’s on, for all post-graduate courses in “The Microfoundations of Economics”. As for the disparaging of Econometrics models, we have to admit the limits of econometric modeling: Economics Nobel prize winner, an excellent econometrician and mathematical economics expert, said so very explicitly in the mid-1970’s when he developed the “rational expectations” models. Already then, “systems analysis”, or what later became known as “complexity theory” had started finding its way in economic analysis and policy-formulation. But, economic policy-making is deeply influenced by politics. And it is not often that good economists who have the courage of their intellectual riopur get to enjoy the highest esteem of populist politicians.

Posted by MohamedMalleck | Report as abusive

THE SCIENTIFIC METHOD IS GROWING INTO AN ATHEISTIC RELIGIOUS CULT among modern armchair scientists, and the scientific propaganda needs to be trimmed down to its true size (science is truly wonderful, but it’s NOT a panacea)… It’s refreshing to read this now from a mainstream writer… Here’s my own philosophical article on a similar subject:
http://www.slyman.org/m_science.php

@Summerland81:
> “Hayek’s followers…were the ones accurately forecasting the back end of 2008.”
Are you suggesting that because Hayek’s followers were right this time around, we should listen to them next time (to the exclusion of other economists)?

@BajaArizona:
> “it is the total lack of scientific understanding of many of the fundamental causes and effects.”
There are plenty of theories about causal relationships in economics. It’s just that we don’t know which ones are right, or, to what degree they are right overall and what circumstances they tend to be most right in… (If we knew that, we’d have another [perhaps grander] economic theory, wouldn’t we?)

> “So the underpinning of most models needs to be thrown away and replaced with, what exactly? It’s easy to state that humans are irrational. Yet it is very hard or perhaps impossible to chart our irrationality and feed it to a computer.”
This is why economics is based on statistics. Like the physical science of gases, individual human beings can behave “randomly”, but they all share certain fundamental properties, and taken as a whole they are likely to follow certain laws.
Some of these laws are obvious, in the same way that we can predict that compressing a gas will increase its temperature, or, increase demand substantially within a market without substantially increasing supply => commodity becomes more expensive.
Other laws are less obvious and some; e.g. the science of economic downturns, which to extend my analogy is somewhat similar to the science of explosions in gas mixtures; are partly a random or statistical affair. It’s very hard to predict the precise moment at which an explosive mixture of gases will actually explode, and similarly hard to predict the precise moment at which a major economic downturn will commence.
If my analogy holds true however, we ought to be able to detect an explosive mixture of economic policies & circumstances – and we ought to have the courage to deal with the danger in a mature manner rather than pretending that the latest asset bubble is another manifestation of the inherent superiority of our own ideology (whether it be capitalism, communism or whatever else)…
http://blogs.reuters.com/david-rohde/201 1/11/10/chinas-cutting-edge-authoritaria nism/

Posted by matthewslyman | Report as abusive

Personally I’m looking forward to Part 2, and to reading some more of the classics e.g. Aristotle that I’ve barely touched directly so far… In the meantime:
http://xkcd.com/683/

Posted by matthewslyman | Report as abusive

In my professional opinion (and I work in science and engineering), it is not the method that is flawed, but mostly the assumptions made (oversimplified) and boundary/system conditions (too narrow) which leads to results that get then generalized beyond where they were intended too. I love the ones that assume “rational decision maker”. Newton and his reductionist approach has caused a lot of problems for fields that need to be encompassing rather than narrow.

Posted by Bert2 | Report as abusive

” the striking thing about the evening was that nothing changed about his models after they were shown to be hopelessly wide of the mark.”

That is the very definition of unscientific. When the observations don’t match the model, the model is wrong, not reality.

Posted by vinlander | Report as abusive

[...] Part One of this essay was published Thursday. This is Part Two. [...]

[...] read the first part here and look up two [...]

[...] and idea of the ‘logical leap of the mind’. He has published an article on the issue of the limits of the scientific method in economics and the world that is a must-read for anyone advocating for ‘evidence based policy’ or the adoption [...]

An odd familiar ring? “He just loaded up the funnel and dumped in the latest problem and started crunching away. I asked him whether the funnel actually improved the solving of problems; stunningly, the hadn’t thought of the question.”

Shouldn’t there be a consistency in what one professes and what one practices?

Posted by Rotkapchen | Report as abusive

[...] are saying the same thing ”The limits of the Scientific Method“ (Martin) and ‘Europe cold be in worst’ hour since WWII (Merkel). What they [...]

Yes, but…aren’t you just saying Science is a progression by which human knowledge is built up over time ?

Blaise Pascal in 1635 noted that Man is full of hubris : the world is not what it is. It’s what we know it to be. As our knowledge grows, it changes.

Thomas Kuhn said the logic of scientific revolutions is always a paradigm shift – rejecting the previous paradigm to move forward.

Nicholas Talleb said : until we know differently, the future will always be seen as an incrementally larger version of past trends. But this is wrong. Change happens through big non-incremental shocks to the system.

If Pascal, Kuhn and Talleb are correct, your economist is simply following in very good well-worn footsteps ?

On the other hand, when in 2004-05 we saw housing prices across the USA rising at 15-20% p.a., but disposable incomes only rising at 3% p.a., surely we should have suspected a bubble ?

Posted by PabloB | Report as abusive

[...] Martin is unhappy with the state of economics. One charge is that: [an economist] predicts a future that is based on the past.  And when it is anything but, [...]

[...] School of Management on “The limits of the scientific method in economics and the world” I  [...]