Using Twitter to predict stock moves

By Felix Salmon
December 23, 2010
This paper, which has now become a fully-fledged hedge fund, is certainly good at hitting buttons: not only does it include Twitter and stocks, but it even finds room to include an important role for Google n-grams!

" data-share-img="" data-share="twitter,facebook,linkedin,reddit,google" data-share-count="true">

This paper, which has now become a fully-fledged hedge fund, is certainly good at hitting buttons: not only does it include Twitter and stocks, but it even finds room to include an important role for Google n-grams!

The beating heart of the paper is this chart, which purports to show a connection between two data series. The blue line is the amount that the Dow rose or fell on any given day; the red line is the frequency with which people are saying “I feel calm,” or words to that effect, on Twitter.

calm.tiff

To my untrained eye, I have to admit that all I see here is two random lines layered on top of each other. But according to the paper, if you run this data through a Granger Causality Analysis and then a Self-Organizing Fuzzy Neural Network, you get all manner of enticing predictive power out the other end. In the chart, the shaded areas supposedly show the periods where Twitter successfully predicted where the stock market was going.

Conceptually, I suppose it makes a certain amount of sense that stocks would fall when people stop feeling calm (nervousness causes selling) and rise when they do feel calm, and therefore more comfortable betting on the future.

It also makes sense that if you’re going to try to use Twitter to predict moves in the stock market, you want to concentrate on what it’s good at, which is giving a real-time glimpse into the sentiment of millions of people. As Pascal-Emmanuel Gobry points out, the algorithm here deliberately strips out stock-related tweets.

I’m sure that the new hedge fund, called Derwent Capital Markets, will be shelling out the maximum $360,000 a year for access to 50% of Twitter’s live stream. So someone’s making money here. But I’m skeptical that Derwent is going to be very successful. After all, predictive power isn’t enough to make money in the stock market: it’s perfectly possible to make money 87.6% of the time but still lose money over the long run.

I also suspect that these kind of algorithms are going to have difficulty keeping up with Twitter as it evolves away from people broadcasting the minutiae of their lives, and towards more sophisticated conversations which are less susceptible to n-gram analysis. But the Derwent crew will certainly be doing some very sophisticated and interesting research on Twitter in the coming months and years. I hope that, eventually, they’ll make some of their results public.

More From Felix Salmon
Post Felix
The Piketty pessimist
The most expensive lottery ticket in the world
The problems of HFT, Joe Stiglitz edition
Private equity math, Nuveen edition
Five explanations for Greece’s bond yield
Comments
3 comments so far
Posted by LaurenDill | Report as abusive

Doing this with non-stock market specific tweets is meaningless. People may be “not calm” or calm for many reasons, especially teens and colleges kids with BF and GF, drugs and other growing issues, and most don’t even buy stock.

On the other hand, if you analyze stock-specific sentiment using natural language understanding and domain-specific knowledge, such as knowing the meaning of buying a call, selling a put, issuing dividends, targets and expectations, etc., the analysis and correlation will be more meaningful. That’s what mktsentiment.com does. We have found filtering rules that have proven to be very effective and can be a useful supplement to other technical indicators.

Spams are not a problem because each source is weighted by a vector and the spams are weighed zero or close to zero. Echoes are detected and treated differently too.

Posted by mktsentiment | Report as abusive

We are seeing much more mood reactions to stock, usually driven by media stories. Twitter seems to be the second wave of mood and investment reaction. If you analyze company-specific information, news articles and return the “media mood” you get a much clearer picture.

FlameIndex.com is a source that seems to be getting much closer to predicting what is about to happen on Twitter. It analyzes 10 K media sources per day and ranks companies based on how “on fire” they are in the media. It filters up the news articles as well. Now all they need is a stock overlay and we’ll be able to see the correlation instead of doing the calculations by hand.

Posted by jmedia | Report as abusive
Post Your Comment

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/