U.S. intelligence and the wisdom of crowds

By Bernd Debusmann
April 1, 2011

After a string of world-shaking events America’s spies failed to predict, most recently the turmoil sweeping the Arab world, a vast project is taking shape to improve forecasting. It involves thousands of volunteers and the wisdom of crowds.

It’s officially known as the Forecasting World Events Project and is sponsored by the Intelligence Advanced Research Activity (IARPA), a little-known agency run by a woman, Lisa Porter, who is occasionally described as America’s answer to the fictional Agent Q who designs cutting edge gadgets for James Bond. Much of IARPA’s work is classified, as is its budget. But the forecasting project is not classified. Invitations to participate are now on the Internet.

The idea is to raise five large competing teams of people of diverse backgrounds who will be asked to make predictions on fields that range from politics and global security to business and economics, public health, social and cultural change and science and technology. The project is expected to run for four years and stems from the recognition that expert forecasts are very often wrong.

One of the teams is being put together by University of Pennsylvania professor Philip Tetlock, whose ground-breaking 2005 book (Expert Political Judgment: How Good is It? How Can We Know?) analysed 27,450 predictions from a variety of experts and found they were no more accurate than random guesses or, as he put it, “a dart-throwing chimpanzee”.

“To test various hypotheses,” Tetlock said in an interview, “we want a large number on my team, 2,500 or so, which would make it almost ten times bigger than the number I analysed in my book.” There are no firm numbers yet on how big the other four teams will be. But Dan Gardner, the author of a just-published book that also highlights the shortcomings of expert predictions, believes the IARPA-sponsored project will be the biggest of its kind. It is expected to start in mid-2011.

The title of Gardner’s book, “Future Babble. Why expert predictions are next to worthless and you can do better,” leaves no doubts over his conclusion. The book is an entertaining, well researched guide to decades of totally wrong predictions from eminent figures. There was the British writer H.N. Norman, for example, who, in the peaceful early days of 1914, predicted there would be no more wars between the big powers of the time. World War I started a few months later.

There was the Stanford biologist Paul Ehrlich, whose best-selling 1968 book The Population Bomb predicted that hundreds of millions of people would starve to death in famines in the 1970s. There was an entire library of books in the 1980s that predicted Japan would overtake the United States as the world’s leading economic power.

Not to forget the U.S. Defense Intelligence Agency’s September 1978 prediction that the Shah of Iran “is expected to remain actively involved in power over the next ten years.” The Shah fled into exile three months later, forced out by increasingly violent demonstrations against his autocratic rule.

NO CLAIRVOYANTS

In a similar vein, U.S. Secretary of State Hillary Clinton said on January 25 that “our assessment is that the Egyptian government is stable and is looking for ways to respond to the legitimate needs and interests of the Egyptian people.”

Seventeeen days later, the leader of that stable government, Hosni Mubarak, stepped down in the face of mass protests.

“We are not clairvoyant,” America’s intelligence czar, James Clapper, told a hearing of the House Intelligence Committee where criticism of the sprawling U.S. intelligence community was aired. “Specific triggers for how and when instability would lead to the collapse of various regimes cannot always be known or predicted.”

True enough. Who could have predicted that the assassination of an archduke in Sarajevo in 1914 would lead to the deaths of 16 million people in World War I? Who could have predicted Japan’s recent earthquake, tsunami and nuclear reactor disaster? On the other hand, there were accurate predictions that U.S. troops invading Iraq in 2003 would not be showered with flowers, as Washington officials had so confidently predicted.

IARPA’s Forecasting Project is not the first American attempt at peering into the future with novel methods. The agency’s richer, bigger and older military sibling, the Defense Advanced Research Projects Agency (DARPA), caused outrage in 2003 with a plan to set up an online market where investors would have traded futures in Middle East developments including coups, assassinations and terrorist attacks.

The man who ran DARPA, the Pentagon’s research arm, at the time, John Poindexter, resigned and the project was killed so we’ll never know whether that market might have been a better indicator of the future than the usual, often over-confident analysts.

And the IARPA teams? The aim of the program, as explained in an online invitation to participate, is to “dramatically enhance the accuracy, precision and timeliness” of forecasts. Gardner, the forecast sceptic, thinks they will remind us that there are things that simply can’t be predicted.

(You can contact the author at Debusmann@Reuters.com)

27 comments

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/

Bernd,

Your column title “U.S. intelligence and the wisdom of crowds” Could just as easily be called:

British Intelligence and the wisdom behind being surrounded at Dunkirk.

Or how about, PM Chamberlain and horrible intelligence of Nazi Germany and its British appeasement?

Can keep going right up to your Royal Marines yomping in the Falklands, and forward…

I can not recall any works or historical documents, that indicate, Intelligence being an exact science. Yet, some how you equate lousy intell. to America. LOL. Take a look in your Britannic back yard brother, you will find no difference, except the shapes on the two flags.

Like it or not Bernd, America will continue to be who we are, and our great ally, England, will continue to be on our side,thankfully. Sorry dude, pretty much been that way since English explorers unloaded here in the 1600′s.

God save the Queen.

Posted by Av8ts | Report as abusive

What a bunch of balderdash. We live in a complex system, which is not predictable by linear methods, and since there are no ways to predict an unpredictable system, this is just the latest example of human ignorance and hubris.

Posted by dcrimso | Report as abusive

They don’t have crystal balls, and never will. Get over it.

Posted by Watcher23 | Report as abusive

Balderdash. hubris—> Face it,the problem is no one trusts.

Posted by Bejesus | Report as abusive

i appreciate him that at-least he try to conclude the scenario.

Posted by Xhan | Report as abusive

“Intelligence” agencies and other forecasting bureaucracies fail most of all because they hire and promote people whose primary opinion is “The Boss is always right!”

This is also the biggest single cause of mediocrity or worse in most American organizations. Flattery is seldom right. It amounts to self-promotion.

Posted by txgadfly | Report as abusive

The only way in my opinion to quantify these results or expect probabilities is to do just that…
“Go quantum” ;)

It ties it all together nicely and when needed, minces the dialogue from misunderstanding to “gates of truth”

Cheers.

Posted by avgprsn | Report as abusive

There was that king, the ancient legend goes, who gathered knowledge from around the world. After several years he managed to compile bullockloads of data about each and every aspect of human life. But he wanted the shorten the data, since he was too lazy to read so much. So the cycle of shortening began, with the king disapproving each new version and asking for an even shorter one. After several months, the king was finally satisfied

The final version was 3 sentences. “Men were born. They lived. Then they died”

There, did I just save US treasury trillions of dollars?

Posted by Stockport | Report as abusive

In my opinion two books get onto the playing field that explains why seemly knowledgeable predictions are wildly off. Blink by Malcolm Gladwell and The Black Swan by Nassim Taleb. Blink explains why I and several others knew there were no WMD in Iraq but could not really explain it. Black Swan is a more thorough discussion of why we miss the obvious. In the Middle East, the fact that the population is deeply unhappy about their life situation is obvious to all. Why the CIA missed it is not understandable since we all have been expecting it. Like the housing bubble burst, we all knew housing prices and cost escalation were unwarrented yet many were caught. My advice to those in power in the bureaucracy is open your eyes and look, quit funding programs.

Posted by wareagleone | Report as abusive

CIA grew from spy agency to beaurocracy. It might be best to wind down the old, and start anew.

Posted by SanPa | Report as abusive

I’m not buying that the CIA and Dept. of State didn’t know about the destabilizing of despot regimes in northern Africa and the Middle East.

I think that the CIA fomented the political unrest… a bit like tossing a handful of burning cigarettes out the window while driving through a savannah drought on a windy day.

The changes now occurring in the region are exactly the kinds of regional changes that the Neocons espoused during America’s military intervention in Iraq.

Coincidence? Perhaps… but I think not.

Posted by breezinthru | Report as abusive

Once you expatriate yourself from the good ole U S of A, it is easy to see the fallacy of the belief systems that sustain such an unsustainable society.Having a window back into America from outside is a real blessing. So few Americans realize what a propaganda state they live in. Even my own family doesn’t get it down there. Most of the rest of the world just sits in awe at the lack of understanding by Americans and the justifications that the USA lives with.

Posted by hapibeli | Report as abusive

Want to see the future? Read history. From Winston Churchill (former British Prime Minister).

Posted by patey | Report as abusive

Breezinthru,

I absolutely agree with you on that. A lot of times, the Head Lines on the News, is not all ways what they may seem!! Well said Breeze.

But please, lets not forget other NATO intell. services. CIA and NSA are not the only ones who play like that! America is not the only one playing in the sandbox. Our chosen friends(NATO) are there with us. IE MI5…etc..America is only pursuing its National Security Policy, to protect our (Corporate) interests!

Posted by Av8ts | Report as abusive

Not to forget the U.S. Defense Intelligence Agency’s September 1978 prediction that the Shah of Iran “is expected to remain actively involved in power over the next ten years.” The Shah fled into exile three months later, forced out by increasingly violent demonstrations against his autocratic rule.

— I believe that was called project AJAX. —

Posted by ckckckckck | Report as abusive

There a many average, ,,normal people ” out there that have followed world events in politics , cultures, economics etc. and i belive they have a good sense of what is going on and where we are heading..Maybe will be a good idea to engage this people in some kind of input apart from profesional inteligence…

Posted by constantin11 | Report as abusive

I don’t understand why people think the CIA, or any other government agency, should have been able to predict the recent events in North Africa. Even if someone had thought that something like this could happen, the odds against it would have been perceived as much greater than the odds in favor of it. As a result, there would have been no reason to have a policy that incorporated the prospect of a fall of North African governments in rapid succession.

Let’s just accept the fact that we can’t predict everything. (That probably includes 9/11, as well as the recent turbulence in North Africa.) The goal of our intelligence and diplomatic services should be to be able to predict what can be predicted and to be nimble enough to be in a position to respond quickly and appropriate when the unexpected happens.

It’s a major mistake (even a moral error) to assume that everything is subject to human control. Sometimes we have to deal with the unexpected, and it’s not anyone’s fault.

Posted by Bob9999 | Report as abusive

Even without specialized knowledge it was possible to state that the Arab uprisings would eventually happen. Chalmers Johnson, an insider if ever there was one, has given us an extensive discussion of blowback, to which I urge everyone’s attention.
I am sure independent specialists in Arab affairs had some idea of the instability in Egypt, especially, even if they could not offer predictions upon which day the revolution would occur.
The real problem is that official Washington, my own home town, is firmly married to group-think, the bosses here desire only to have their a priori positions supported and actually take severe and often career-ending actions against dissenters within their institutional setup.
Personally I have successfully predicted many of the things mentioned in this article; even making money as a result, but under no circumstances at at all would I collaborate with the killers, crooks and thieves in charge of my country.

Posted by ChrisHerz | Report as abusive

If there is one fact that had been made plain over time it is that professional prognosticators can’t see crap coming, even though it happens every day. This fact once again validates that establishment & intelligence when combined create a truly rotted cesspool of intellectual impaction.

Ignorance is curable, systemically enforced stupidity not so much.

Posted by zhmileskendig | Report as abusive

Oh my! Isn’t this the project the Bush Jr. regime proposed at some point only in a different manner?! He wanted public bets…

Posted by mdiavaro99ro | Report as abusive

We in the Middle East are amazed at the analyses and understanding of the “West”‘s experts and Think-Tanks when they attempt to analyze or comment on events in our region. That’s why the regular Arab accuses them of being either puppets of Israel or a propaganda instrument because they are always way off in their assessments, that seems obvious to the people on the street.
However, the reasons for these inaccuracies by those experts is simply that they want to understand and predict a situation on the streets while they sit in offices overlooking capital hill and they use Google maps to know Tahrir square.
If you had people on the ground, interacting with the locals, understanding their feelings and aspirations..you would have found out that they are humans that love democracy, what peace and prosperity for their country and they do not want make their kids suicide bombers.
Once you understand that then you have to work with them to promote democracy and peace in the region… you won’t need to give Billions to the governments of Israel or Egypt.

Posted by quateen | Report as abusive

With billions annual budget & dozens agencies, they either get it right or get out of this business.

The world will be better place spending that money on research new energy sources, economocal & social development of the under priviledged.

Posted by citoyendumonde | Report as abusive

I say shutdown the CIA and the NSA. Both groups are just a bunch of warmongers. We would be safer as Americans if that money was used to fund mental health clinics rather than militant lunatics.

Posted by M.C.McBride | Report as abusive

I predict it will fail …

Posted by idiotboy | Report as abusive

[...] Buddhist dhamma, with which he did have contacts Anyone can also check out this related post: http://blogs.reuters.com/bernddebusmann/ 2011/04/01/u-s-intelligence-and-the-wisd om-of-crowds/ Further you can see this related post: [...]

The prediction is only as good as the data and analysis points of the predicting person. To many people do not take into account enough factors or they do not have access to accurate data or they have cultural bias that limits their understanding of the data.

It is a science… but it is also an art. A bit like playing poker or handicapping race horses.

Posted by Suemanley1 | Report as abusive

[...] a large project is taking shape to improve forecasting. It involves thousands of volunteers and the wisdom of [...]

[...] In In US Intelligence and the Wisdom of Crowds (Reuters), we learn of the Forecasting World Events Project , in which crowdsourcing is used as an [...]

[...] 9. San Carlos (23rd May 1982) – British troops land at San Carlos Water. Although, British ships come under severe Argentine air attack in “bomb alley”, the Argentine’s In addition you can check out this related post: http://blogs.reuters.com/bernddebusmann/ 2011/04/01/u-s-intelligence-and-the-wisd om-of-crowds/ [...]

[...] of participants – to improve forecasting, Reuters´ World Affairs columnist Bernd Debusmann reports: The idea is to raise five large competing teams of people of diverse backgrounds who will be asked [...]

Is there customer service available if I need assistance setting this up?

Right now in the intelligence community, we are witnessing what I call the “collapse of paradigms”. Federal agencies like the CI, NASA, Homeland Security, and the terroriam task force and counterintelligence division of the FBI, have failed us. Their over-reliance on technology — some would say infatuation — is largely to blame.

To wit, things have evolved to the point where 70 per cent of our intelligenc operationsare currently outsourced. Read Tim Shorrock’s excellent book, “Spies for Hire”.

That said, what I put my faith in is Web-Bot Linguistic Analysis.

Posted by jsakowicz | Report as abusive

Related……

[...]just beneath, are numerous totally not related sites to ours, however, they are surely worth going over[...]……

Tumblr article…

I saw a writer writing about this on Tumblr and it linked to…

Great Blog…

Excellent weblog right here! Additionally your site loads up fast! What web host are you using? Can I am getting your affiliate link to your host? I desire my site loaded up as fast as yours lol…

Porno…

Porno izle…

Check this out…

[...] that is the end of this article. Here you’ll find some sites that we think you’ll appreciate, just click the links over[...]……

The Birch of the Shadow…

I think there may perhaps become a several duplicates, but an exceedingly useful listing! I’ve tweeted this. A lot of thanks for sharing!…

… [Trackback]…

[...] Read More: blogs.reuters.com/bernddebusmann/2011/04  /01/u-s-intelligence-and-the-wisdom-of- crowds/ [...]…