Comments on: Twitter datapoint of the day A slice of lime in the soda Sun, 26 Oct 2014 19:05:02 +0000 hourly 1 By: saumil07 Fri, 19 Nov 2010 01:14:40 +0000 You are in fact incorrect in claiming that 5% of the stream is roughly equivalent to 100% of the stream for capturing trends.

Secondly, trends are just one facet of all the interesting things that can be accomplished with the Twitter. For example, if you wanted to – given an arbitrary Twitter id – find out their topics of interest, good luck doing that with 5% of the stream.

Similarly, if you want to build a social media monitoring service (of the kind that Sysomos built and sold successfully last year) and then sell the service to large brands, once again, good luck doing that with 5% of the overall stream.

Lastly, the folks who license the firehose – and that list of companies is easily available via a google search – are inherently uninterested in being a reseller. They are not high-frequency trading shops but are mostly Silicon Valley companies trying to build innovative apps and services on top of this mass volume of data.

By: TFF Thu, 18 Nov 2010 18:32:12 +0000 @DanHess, I’ve heard several answers to that one.

“I work hard and am paid well. Ethics? What is unethical about working hard and getting paid?”

“HFT more than doubles the trading volume of the major stocks. This provides VALUABLE LIQUIDITY, allowing you to sell 100,000 shares in five seconds. Without HFT, you might have to spread out that order over five minutes. Of course this liquidity disappears the moment the market needs it the most, but heck, nobody’s perfect!”

“The middle-man has always taken a cut of the action. We don’t take any larger a cut per transaction than happened thirty years ago. In fact by doubling the number of transactions, we can claim that our cut per transaction is SMALLER. And the spreads are also tight, which is of critical importance if you want to sell something you just bought ten seconds ago.”

By: DanHess Thu, 18 Nov 2010 16:42:10 +0000 @DrWex —

Right, HFT does not rely on any real-world events, but mainly instantaneous pricing data that allows practitioners to identify block trades and front-run them.

This is completely unethical, whether or not unethical lawyers are able to justify it. If this is your profession, my question would not be, how does it work? My question would be, how do you sleep at night?

By: petewarden Thu, 18 Nov 2010 16:16:31 +0000 Hi Felix,
first off, a disclaimer. As a fellow data geek in Boulder I’m friends with the Gnip guys, but I have no financial or business relationship with them, and no access to non-public information on their business model. I also did one of the RWW articles about this announcement:  / p

So, with all that out of the way, I think you’re wrong, or at least ahead of your time, when you assume HFT’s are the customers who are paying for the half fire hose. Their customers are cagy about having their names released, but from knowing the market most of them are going to be firms re-selling social media monitoring services to large brands. They take in the stream, and pull out mentions of hundreds or thousands of different brands that they’re monitoring. Each end-user may only care about a dozen brands and be willing to pay 10 or 20k a year for it, but the monitoring firms can use the feed they get from Gnip to offer this to as many clients as they can sign up. I have zero idea if they’re actually customers, but think of companies like Radian6. It’s also the same problem Cisco’s new Social Miner is trying to solve: 349/index.html

It totally makes sense that HFT shops *should* be front-running Twitter data, what’s surprised me has been how little progress the few folks I’ve had contact with have made. After I attempted a release of public Facebook data to academics ( ser/2010/02/how-to-split-up-the-us.html ) I was approached by some of these companies. They were mostly data-driven hedge funds as far as I can tell, using traditional tools like polls, surveys and focus groups to predict consumer trends and then trade on that basis. They were intrigued by the idea of extending it to cover social media, but pretty hesitant. I think this Quora thread captures a lot of the reasons: itter-sentiment-analysis-guide-stock-mar ket-investment

There’s evidence from the paper mentioned at the beginning that Twitter is currently a good leading indicator, but it’s also trivial to imagine how to game it once people began to rely on it.

Anyway, this comment is far too long already, so I won’t belabor the point, but this is a fascinating area with some non-obvious problems. I’d be happy to geek out about this further off-line if you want to ping me.

By: DrWex Thu, 18 Nov 2010 15:21:57 +0000 You completely misunderstand how HFT works. News streams may form part of the input for algorithmic trading strategies, some of which operate at high frequencies, but most HFT is done without regard either to fundamentals or to news of any sort. Indeed, most traders tend to disparage news-sourced automated trading strategies because they are unproven at best.

If you’d like to talk about HFT some time, drop me an email.

By: q_is_too_short Thu, 18 Nov 2010 01:01:03 +0000 guess i won’t be entering any block equity orders into twitter before getting them filled!

By: Danny_Black Wed, 17 Nov 2010 23:56:48 +0000 Is the 5% 5% of ALL tweets? or the tweets of 5% of users at a given time. Does it include retweets, which i imagine is quite valuable to know?

I alluded, when you were casting around for post ideas, to role of such social media in news propagation. Certainly, a news story that certain people break is far more likely to make it into a echo chamber and be market moving. If I was an HFT shop, I’d want the feeds off them.