Felix Salmon

Data crashes: inevitable and mostly harmless

By Felix Salmon
August 22, 2013

When you look up the price of a stock on the Nasdaq stock exchange, you’re not really looking up the price at which it’s trading on that exchange. All of the Nasdaq stocks trade on dozens of exchanges, all of which have the right to trade in those stocks. That right is known in the market as unlisted trading privileges, or UTP. The job of the Nasdaq is to serve as the securities information processor, or SIP, for all those different exchanges: the exchanges report to the Nasdaq all of the information they have on bids and offers and trades, and then the Nasdaq aggregates all that information and presents it in one place. Most importantly, it shows the most recent price at which any given stock traded, on any exchange. That’s the price you’re looking at.

Now the Nasdaq is an exchange itself, but it doesn’t really have to be: it could halt all trading on its own exchange tomorrow, and no real harm would be done. So long as Nasdaq stocks could trade on all the other exchanges, and the Nasdaq could continue to keep tabs on all the trading going on across the different exchanges, no one would really notice. The real importance of the Nasdaq, then, is not its status as an exchange, but rather its status as a SIP.

The stock market is a bit like chess, or backgammon: it’s a game of (to coin a phrase) total information awareness. Everybody in the market receives the same information in real time. Of course it’s up to any individual trader how — and how quickly — they act on that information. When you enter the stock market, you basically sign a waiver giving up all privacy rights; the Nasdaq, doing its best NSA impression, watches your trades, collates the data on each one, and then republishes it. It’s a prime example of a situation where all the value lies in the metadata: if an exchange stops trading, the effect is much less drastic than if the SIP stops collating and disseminating its index of what’s going on.

Today, of course, that’s exactly what happened. In the parlance of the Nadsaq, there was “an issue at the UTP SIP”, and as a result, all trading, on all exchanges, came to a sudden stop at 12:14pm. The system stayed down for most of the afternoon, until 3:25pm.

The financial system can cope with a three-hour outage, of course. The entire stock market was closed for two days after Hurricane Sandy, and for four days after 9/11. Yes, unexpected intraday outages are bad things — they leave traders with open positions which can’t be closed. But in this case, the Nasdaq managed to reopen before the end of the day, and not much happened over the course of the three dark hours.

But the Nasdaq’s systems aren’t designed to go down and up like this. It’s really just good fortune that the Nasdaq managed to get things working again before the close. Our system of stock exchanges is so incredibly complex, with so much information flowing around at mind-boggling velocity, that it is certain to fail from time to time — and to fail in unexpected ways. We probably won’t know for days what caused today’s mysterious “issue” — and even if the Nasdaq manages to put patches in place to ensure that it doesn’t happen again, something else — something equally unexpected — will happen instead. As Alexis Madrigal says, the surprising thing isn’t that the Nasdaq broke, it’s that we don’t see this kind of thing far more often.

In fact, if I had the opportunity to interview Edward Snowden, that’s one of the questions I’d love to ask: How well do the NSA’s systems work? How often do they just crash, or otherwise stop working for an unexpected and unpredictable reason? The NSA is dealing with orders of magnitude more data than the Nasdaq, and has to do so in conditions of great secrecy. My guess is that things go wrong on a pretty regular basis. But the real-world consequences of today’s market outage, just like the real-world consequences of the flash crash, were pretty slim. And so too is it hard to determine what if any harm might be done by a temporary failure of America’s national security apparatus. When we look at an ultra-high-tech world of server farms and state-of-the-art code, we tend to assume that it’s all incredibly valuable and important. But it turns out, when we lose it, that most of us don’t even notice.

3 comments so far | RSS Comments RSS

Our stock exchanges have devolved. Instead of one big exchange which huge breath we have many smaller ones where there is no longer a market maker charged with maintaining an orderly market. Our order flow is sold so that the big hedge funds can jump in front of our trading. We need to go back to one big NYSE that is publicly owned with @2 brokers charged with providing liquidity in when no-body else will provide a bid or ask.

Posted by Sechel | Report as abusive

When a firm places a high speed order in the amount of $billions across many of the stocks listed on an exchange and it runs out of counter-party orders then the liquidity of the exchange is drained and the exchange freezes. or do I have that weong

When Shanghai had a similar problem the index spiked. The CEO of the offending firm was dismissed.

Posted by whoisit | Report as abusive

….”with so much information flowing around at mind-boggling velocity”….

Thank you for admitting that a lot of the high velocity “trading” is CIRCULAR.

Posted by kiers | Report as abusive

Post Your Comment

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/