Chart of the day, HFT edition

By Felix Salmon
August 6, 2012
Nanex, and shows the amount of high-frequency trading in the stock market from January 2007 to January 2012.

" data-share-img="" data-share="twitter,facebook,linkedin,reddit,google" data-share-count="true">

>

This astonishing GIF comes from Nanex, and shows the amount of high-frequency trading in the stock market from January 2007 to January 2012. (Which means that the Knightmare craziness of last week is not included.)

The various colors, as identified in the legend on the right, are all the different US stock exchanges. You might think there are only two stock exchanges in the US, but you’d be wrong: there are only two exchanges where stocks are listed. There are many, many more exchanges where stocks are traded.

What we see here is relatively low levels of high-frequency trading through all of 2007. Then, in 2008, a pattern starts to emerge: a big spike right at the close, at 4pm, which is soon mirrored by another spike at the open. This is the era of traders going off to play golf in the middle of the day, because nothing interesting happens except at the beginning and the end of the trading day. But it doesn’t last long.

By the end of 2008, odd spikes in trading activity show up in the middle of the day, and of course there’s a huge flurry of activity around the time of the financial crisis. And then, after that, things just become completely unpredictable. There’s still a morning spike for most of 2009, but even that goes away eventually, to be replaced with sheer noise. Sometimes, like at the end of 2010, high-frequency trading activity is very low. At other times, like at the end of 2011, it’s incredibly high. Intraday spikes can happen at any time of day, and volumes can surge and fall back in pretty much random fashion.

It’s certainly fair to say that if you take a long, five-year view, then you can see a clear rise in trading activity. But it’s also fair to say that there’s something quite literally out of control going on here. Just as the quants at Knight found themselves unable to turn off their machines for 30 long minutes last week, the HFT world in aggregate seemingly has a mind of its own when it comes to trading patterns. Or, to put it another way, if there’s a pattern here, it’s one incomprehensible to human minds.

Back in 2007, I wasn’t a fan of a financial-transactions tax; today, I am. And this chart shows better than anything why my opinion has changed. The stock market is clearly more dangerous than it was in 2007, with much greater tail risk; meanwhile, in return for facing that danger, society as a whole has received precious little utility. Are spreads a tiny bit tighter than they might be otherwise? Perhaps. But that has no effect on stock-market returns for long-term or even medium-term investors.

The stock market today is a war zone, where algobots fight each other over pennies, millions of times a second. Sometimes, the casualties are merely companies like Knight, and few people have much sympathy for them. But inevitably, at some point in the future, significant losses will end up being borne by investors with no direct connection to the HFT world, which is so complex that its potential systemic repercussions are literally unknowable. The potential cost is huge; the short-term benefits are minuscule. Let’s give HFT the funeral it deserves.

More From Felix Salmon
Post Felix
The Piketty pessimist
The most expensive lottery ticket in the world
The problems of HFT, Joe Stiglitz edition
Private equity math, Nuveen edition
Five explanations for Greece’s bond yield
Comments
38 comments so far

“Are spreads a tiny bit tighter than they might be otherwise?”

no – spreads are a METRIC CRAPLOAD tighter than they might be otherwise… than they were before… (i used the metric system for you, Felix, because I know you are from Across the Pond)

aside, I don’t know how, attempting to explain why tighter spreads don’t help, you invoke the returns of “long term or even medium term investors” at the same time you’re worrying about HFT – which has no effect at all on precisely those long and medium term investors.

Posted by KidDynamite | Report as abusive

Felix,

What data does the chart show, and how is it calculated? Is the Y axis shares traded? How do they identify “HFT” trades for the purpose of this chart? If the Y axis is shares traded, and given that we keep hearing that US stock trading volume is in decline, are we to believe that HFT accounted for a minuscule fraction of a large number in 2007/2008, and a much larger fraction of a smaller number now?

Posted by m_m | Report as abusive

“But inevitably, at some point in the future, significant losses will end up being borne by investors with no direct connection to the HFT world”

I’m a bit skeptical about this line. HFT trading is most deadly to HFT traders. But as long as they are well capitalized in the event of stupidity, I’m not concerned about their effect on the greater economy or my portfolio.

Posted by TFF | Report as abusive

Let me posit that HFT has driven a lot of people out of business: the specialists, execution brokers and day-traders who used to collect the large rents built into the system in the days of old. HFT and algo-trading practitioners are rent-seekers too, but their rents are far smaller. And the vast majority of the whining you hear is from exactly the people that HFT has displaced. I have read or heard nothing about how medium-to-long-term investors are being disadvantaged by HFT; to the contrary, their costs have gone down substantially.

As for destabilizing the system, sure, a lot can be done to improve the market structure and micro-structure. But we have just come through one of the most volatile and unstable periods ever; did you really expect this or any market structure to survive through this without showing the occasional crack? And what you got, even then, was the Flash Crash and some instances of erroneous behavior, the most egregious of which was Knight. In the days of specialist-and-broker intermediation, we got Black Monday without any macro stress of remotely comparable scale; we got lots of (human) fat finger errors all the time, too.

So, are you saying that it is morally and socially acceptable for one group of error-prone humans to extract large rents from the system, and it is not morally or socially acceptable for another group of (differently) error-prone humans to extract much smaller rents?

Posted by m_m | Report as abusive

@m_m: very well said, especially your first paragraph which is dead on accurate.

One thing I would add, which I have pounded the table on regarding this topic, is that HFT is “democratized” compared to the “Old Boys Club” that was the NYSE Specialist system. By which I mean, anyone can launch an HFT operation if they are capable. But not anyone could become an NYSE Specialist with a license to print money.

Posted by KidDynamite | Report as abusive

KD, tell me, what has happened to bid-offer spreads since 2007? Give me numbers, and then tell me there’s been a “metric crapton” of tightening.

Posted by FelixSalmon | Report as abusive

Yes a financial transaction tax is a fantastic idea. There is no social utility to ownership of these things changing as quickly as it does, and it would raise revenue from people who could afford it and make the financial secotr more stable and more appropriately sized compared to the rest fo the economy (i.e. smaller). We really really need to try as much as possible to maintain strong incentives for banks to practice traditional banking, and erect barriers to having a tenth or sixth of the economy just playing roulette with each other to no ones benefit.

Posted by QCIC | Report as abusive

TFF, that’s what I thought in January of 2008 about the impending disaster that would be caused by far too many unworthy mortgages being financed. I had no debt, and no equity in financial firms, yet still lost 40% of everything I owned. And while the stock markets have recovered from the crash, my total assets have not, and more important than my personal situation, the economy is weaker and far more fragile than it used to be.

If it were just the HF traders that will ultimately lose their shirts, I too, would not care. But I have to think that they will drag others down with them, who will eventually require a rescue or will further weaken the economy and reduce confidence in the nation’s ability to manage that economy, resulting in extreme reactions that will be as beneficial as cutting taxes during wartime.

I don’t think a transaction tax is necessary, although I wouldn’t care if one was levied, but inserting a delay into trades so they can’t be completed as fast as they are now would be just as effective, and would have a dampening effect on the positive feedback loop that HFT essentially adds to an already unstable system.

Posted by KenG_CA | Report as abusive

Felix –

electronic trading and HFT didn’t start in 2007.

You are a big hitter – I’m sure you can get the data from 1998 to now. teeny minimum before –> 1c or less now = 84% decline in spread… and many were larger than a teeny before.

Posted by KidDynamite | Report as abusive

also, Felix – in case you missed this post:

http://www.indexuniverse.com/hot-topics/ 12782-4-etf-lessons-from-knight.html

that’s what happens to spreads when you “give HFT the funeral it deserves”… VICTORY! everyone pays more! Great stuff! (/sarcasm)

Posted by KidDynamite | Report as abusive

KD, I think Felix was trying to make a secondary point that the main argument for ‘needing’ HFT’s, liquidity for cheap, was largely accomplished by 2007…and also, he’s working with the GIF above, which only covers those years.

So, is there a marked decrease in costs from 2007 to present, justifying this huge surge in activity, or was most of that taken care of from 1998 to 2007?

==RED

Posted by REDruin | Report as abusive

KD, I also read the link, and that seems to be about needing a market maker, not HFT. I do see that some HFT’s are slowly moving into the space Knight used to cover, but they still aren’t making up for a primary MM. Is Knight considered a HFT because they were a primary MM, too? Or how is that related for your argument on their neccessity?

==RED

Posted by REDruin | Report as abusive

@Red – yes, i think that the point that Felix was PERHAPS trying to make was that we may be nearing (or have reached) the limits of achievable improvements in markets (ie, limits of tighter spreads) from HFT, so perhaps we should stop catering to that segment now (by not building new data centers to house them, etc).

That would be a perfectly reasonable idea, but it’s very different from Felix’s conclusion: “Let’s give HFT the funeral it deserves.”

to your other question: these automated electronic market making platforms, like Knight’s, as a subset of HFT. My point was simply that when these HFT electronic market makers disappear, the effects are immediate and detrimental to bid/ask spreads.

Posted by KidDynamite | Report as abusive

The financial transaction tax would penalize too many people. In order to tame HFT, issue a mandatory minimum order length (1/10 of a second would do it), or a cancel fee (a penny per cancel would do that). One, the other, or both would cripple HFT and HFT only.

Posted by ebgold | Report as abusive

KidDynamite,

High-frequency trading harms longer term investors by distorting prices.

Fundamental to the orderly functioning of markets is the idea that asset prices reflect the underlying value of assets they represent. In the case of stock markets, this means that the price of a stock at any given time should represent the perceived value of the assets it represents. This implies that trades are being made which reflect informed opinions and judgments about whether the current price of an asset accurately represents the present and/or future value of its underlying assets.

However, algorithms do not have the capability to make these types of judgments. Nor do they care about the present or future relationship between price and actual underlying value. All they care about is pricing inefficiencies.

This is why high-frequency trading is so dangerous…i.e. because it does not care about the relationship between underlying asset value and current price which is the underpinning of orderly and well-functioning markets.

Why would any sane person invest in the stock market when they have no faith that asset prices are accurately reflecting the value of their underlying assets? High-frequency trading is now such a high percentage of overall trading volume that the long term value of an asset barely factors into its current price at all.

Ten years ago, a billion shares a day was considered to be huge volume….now its 4-5 billion shares a day, and let me tell you, all that extra volume isn’t coming from rational investors making judgments about the long terms values of stocks.

Posted by mfw13 | Report as abusive

Every time I trade a stock I get a mental image of my transaction getting beaten up by an HFT algo along the way who then steals my lunch money. . .

Posted by jpmist | Report as abusive

Yup. Forget that competing algos executing hundreds of thousands of high speed financial transactions every second are indistinguishable from the illegal practice of Quote Stuffing and Front Running. Forget that if the identical transaction patterns were being placed by individuals at speeds that humans could comprehend we’d be seeing perp walks, indictments and convictions. All of this has been known for years but no one is enforcing the law when algos are committing the crime. Is this unconscionable malfeasance on the part of private and public regulators and compliance officers? Yes. Nevertheless, the consequences of quasi-criminal HFT trading amounts to mouse nuts when compared to the potential financial damage that these algos could and probably will ultimately inflict on the market. The big problem lives at the intersection and interaction of competing HFT programs and trading systems.

There is a popular aphorism on Wall Street “Price is truth”. These algos are designed to secure a pricing advantage over other algos and human traders by setting, quoting, offering, cancelling and transacting price quotes. It does not matter how much testing on each individual algorithm or program is done within the vacuum of a market simulation. It is inherently impossible to predict how the algo will behave in every circumstance when competing with many other algos that are gaming the system for the same price advantage. And it all happens so fast and and at such high volume, that by the time humans can intervene and pull the plug, the damage is done.

http://www.dividist.com/2012/08/rise-of- algos-knight-capital-and.html

Posted by Dividist | Report as abusive

I used to trade stocks….
No more.
That’s the real danger here. Soon only algos will be fighting it out over fractions of mills.

Posted by mach1513 | Report as abusive

mfw13: HFT does not compete with the Grahams and Buffetts of this world. Think of it this way: does your broker, the exchange specialist, or your day-trading neighbor care about “value”? Now, replace all of those people by a machine, and there is your HFT/algo universe. Of course the machine does not care about long-term value: it was never meant to.

As for whether HFT cares about any notion of a valid price at all: of course it does. HFTs do not accumulate positions, they must sell what they have bought. The trend moves are still dictated by non-HFT, accumulative or distributive market participants. And stepping out of the bounds established by these participants can be very expensive for these HFTs, as Knight has amply demonstrated.

And there is one particular form of price validity that is driven entirely at this point by HFTs: if there is a big move in EURUSD, you would expect that move to be reflected in the interest rate markets, in the bond markets, in commodities, and in stocks, to varying degrees. At this point, it is the algo bots who mostly enforce this, keeping prices in line and markets “efficient”. In fact, once all is done and dusted, the world will likely consist of a large number of bots enforcing arbitrage relationships and deploying liquidity as and where needed (while extracting minuscule profits per trade for their builders and maintainers), and medium-to-long-term investors investing based on their notions of long-term value. The specialists, brokers, short-term traders: all gone. Surely, as people who want human ingenuity to be deployed in the service of long-term value, that is a future we can all get behind.

Posted by m_m | Report as abusive

KD, I’m not really seeing Point Two you made in response. What happened was a market maker in a niche was removed. In ANY event where you lose a market maker, the spread will balloon as you go back to pure human judgement. WOuld it not be more analogous to say that the inability of the exchange to assign another market maker with speed was more important then the fact that Knight was an enabler of HFT in a niche market?

I doubt you’ll find Felix arguing against the benefit of lower spreads that HFT helps bring on a competitive basis…the algos have forced the big names to lower their prices, sure enough. What I got out of his argument was that the ‘improvements’ over the last 5 years have been miniscule, and it’s time to slow down HFT to the rates of 5 years ago, live with the slim fraction of difference, and reduce the volatility and vulnerability of the market.

In short, turn HFT back to MFT by whatever means are simplest, and go on.

If he put in a little exaggeration, well, he’s a blogger!

Your thoughts?

==REDruin

Posted by REDruin | Report as abusive

“If it were just the HF traders that will ultimately lose their shirts, I too, would not care.”

@KenG, is the HFT leveraged? Or is it well collateralized? Big difference in the potential impact on economic stability.

“Why would any sane person invest in the stock market when they have no faith that asset prices are accurately reflecting the value of their underlying assets?”

Perhaps because any sane person can, from time to time, identify mispricing with conviction? And acting on this conviction can achieve exceptional returns? As long as *I* can accurately value the underlying assets, it is to my benefit if the market chooses to misprice them.

Posted by TFF | Report as abusive

It’s always interesting to me how HFT proponents take credit for tighter spreads in every circumstance.

As the following GAO study on stock price decimalization shows, spreads tightened significantly immediately after decimalization, so that by the end of 2002 we had almost all of the gains of the last 10 years already in place.

HFT activity in 2002 was about 10% of the market back then, as opposed to 60%-70% of it today. At least in terms of spreads, almost everything done since 2002 has been full of sound and fury, signifying nothing but tail risk.

http://www.gao.gov/new.items/d05535.pdf

Posted by MErskin | Report as abusive

I assume the Y axis is some measure of trading activity (trades quotes etc). While commentators above have dwelled on equities, the graph also shows data from options exchanges such as CBOE. The increasing activity from these exchanges around 2007 can be partly explained (besides of course increasing volatility) by a large number of options classes going penny pilot which reduced the spreads drastically…

Posted by traderjoe85 | Report as abusive

TFF, I don’t think it matters if the HF traders are leveraged or well collateralized, for they could cause price instability, that when coupled with other traders’ leverage or under-collateralization, could lead to bigger problems (and then those leveraged “victims” of HFT would cry “nobody could have foreseen it”). I’m not opposed ot HFT for the damage they can inflict on themselves, but for the distortion of the market they can cause.

Posted by KenG_CA | Report as abusive

@KenG, then insist on proper collateral. Split trading desks off from traditional banking. Limit the network of interdependencies that allow one major failure to bring down the entire system.

I’m happily in favor of shutting down HFT in favor of a more transparent system, but I don’t see it as a major risk to our economy.

Posted by TFF | Report as abusive

TFF, easier said than done.

And while it’s not a major risk for the economy, it’s all downside and no upside.

Posted by KenG_CA | Report as abusive

It is still beyond me which socially useful function of the stock markets cannot be covered with a single daily fix on limit orders.

Why would someone care about executing a trade any faster than in 24 hours ?

Stock holdings for legitimate and socially useful purposes – dividend distributions and voting rights – is measured in years, oftentimes decades. Anything else is gambling. So, stock markets should provide liquidity opportunities commensurate with this type of timescale, and nothing else.

Posted by Frwip | Report as abusive

I see a generally invalid and generally valid argument against HFT.

1. The HFT = tail risk argument. There’s this idea that the robots could all go on strike at once and the stock market would collapse. This could, and has, happened with human market makers as well. Such a collapse like the flash crash does not last long because it leaves mispricings available for profit. There’s this idea that the big mispricings wouldn’t be arbitraged because the robots have taken over…or something.

2. HFT exacts a tiny tax on regular investors. I find this argument more credible based on anecdotal evidence of how HFT works. My understanding is that the robots will place multiple orders to figure out where investors tend to have limits placed on limit orders. Then they will sell at the high limit and buy at the low limit. Some computers can also see order lists before other computers and adjust accordingly. I find a hard time justifying how HFT traders can make and cancel multiple orders just to see which ones are accepted, as well as borderline front-running flash trades and such.

But in general, the arguments about HFT causing impending doom are overblown. If their losses are limited to their own trading book, then I don’t see the issue. HFT, in and of itself, won’t change the long-term value of a stock.

As far as the more general argument that “no investor should care about executing orders in less than 24 hours,” there are a lot of economic fundamental reasons for intraday trading. In particular, to have stock prices adjust rapidly to new news. In theory at least, all the investors reacting to the latest quarterly reports and rumors makes sure that passive investors buying index funds do not overpay for the stock, or get too little when they sell.

Posted by mwwaters | Report as abusive

Surely the problem is that the liquidity argument for high-frequency trading is being undermined by the increasing frequency of these “flash crash” type events? Moreover, blaming the problem on human error seems to ignore the magnification effect of HFT, which means that a single error or change of market conditions can cause the liquidity provided by these algorithms to evaporate.

There are some thoughts on how this could impact value investors here:

http://www.mindfulmoney.co.uk/13459/inve sting-strategy-/knight-capital-the-rise- and-fall-of-the-machines.html

Perhaps Bill Joy had a point – “We are being propelled into this new century with no plan, no control, no brakes. Have we already gone too far down the path to alter course? I don’t believe so, but we aren’t trying yet, and the last chance to assert control – the fail-safe point – is rapidly approaching.”

Posted by MindfulMoney_T | Report as abusive

There is one large point that no one has mentioned…more and more, public pension plans are handing public money to hedge funds and HFT. This is dangerous because 1) returns to public institutions are rarely high on the priority list because public money tends to be dumb money and 2) if public money is being used to create exploitable dislocations, it is being used to cause harm to the public welfare.

Posted by ZEngineer | Report as abusive

Perhaps the film “Space Odyssey 2001″ is an apt cautionary tale for us in 2012.

What the Nanex GIF graphically demonstrates beyond question is that “HAL” lives and is visiting havoc upon the marketplace.

So, upon reflection, we can perhaps all agree that the acronym HFT really has a different meaning. Doesn’t it practically stand for “HAL F_CKS TRADERS”?

Posted by JdeCP | Report as abusive

Yep. Nothing to see here folks. Move along. The regulators will take care of this. LOL

Posted by robertsgt40 | Report as abusive

Ah, Kid Dynamite, heroic defender of market manipulators everywhere! Got a beef with the HFT thing? Here’s the Kid to show up on your site and explain why you’re just so wrong! Do you claim the gold market’s been rigged? Here comes the Kid, to inform you that you just don’t get the complicated swaps market, you silly unsophisticated goose!

Boiled down, HFT is just a fancy way to front run legit bids and offers, and enter spoof bids you can yank away in a blink (literally). It distorts market prices, which makes Kid’s claim that HFT “has no effect at all on… long and medium term investors” all the more hilarious. Maybe today was the day, I, the “long-term investor” decided finally to unload my shares I’ve been holding so long — well, thanks to some algo, I’ve sold back for less than I should’ve gotten. Or worse — maybe I’ve decided NOT to hold my long-term shares anymore because I’m alarmed at what a joke the whole flash-crash casino has become. So I sell just to get out — at a lower price maybe than I’d like — but hey, no harm done eh? Because the Kid says there’s none!

You know, seeing the way Kid stalks sites to jump in and pooh-pooh any possible “conspiracy” talk, I’m beginning to wonder if he himself is in reality a robo algo, too.

Posted by Orsius | Report as abusive

ZEngineer says
“…more and more, public pension plans are handing public money to hedge funds and HFT. This is dangerous because 1) returns to public institutions are rarely high on the priority list because public money tends to be dumb money and 2) if public money is being used to create exploitable dislocations, it is being used to cause harm to the public welfare.”

And would you say pension plans refer to “long-term investors”?

And are you listening, Kid Dynamite?

Again, HFT is nothing more than a way to enter spoof orders, and take as many free shots as you need to trip stops. Other than that though, it’s entirely harmless.

Posted by Orsius | Report as abusive

One of the most important things in that discussion is:
Goldman Sachs were, for years, the only one able to really “deploy” HFT because the had exclusive deals to put their servers into(!) the NYSE.
Back in that days, no other players were able to “deliver” the response times requried…
Then, over the last years, Goldman lost its exclusive deal, thus more and more “HFT-buddies” were able to put their servers directly into (or at least near to) the stock exchange.
Result of that is: The growth and volume of the HFT-traded assets received and exponential boom.
Regards
Andrea
http://www.lelala.de

Posted by AndreaLelala | Report as abusive

The Nanex graphics are predicted and explained by a recent stock price formula for high frequency trading available at the Social Science Research Network under http://papers.ssrn.com/abstract=1977561 The peaks and troughs are inevitable when a large number of trades are executed. Its called phase lock. It happens, and will happen, all the time.

Posted by gopetca | Report as abusive

Hi I would like to show this graphic and use some quotes from this blog in a session I’m doing on Social Media & Systems Thinking at the systems thinking summit in Cardiff in October. Is it OK to show the graphic or do I need permission.

Thanks

@sysparatem

Posted by sysparatem | Report as abusive

This and other topics that are relevant for speed traders and institutional investors will be discussed at High-Frequency Trading Leaders Forum 2013 London, next Thursday March 21.

Posted by EllieKim | Report as abusive
Post Your Comment

We welcome comments that advance the story through relevant opinion, anecdotes, links and data. If you see a comment that you believe is irrelevant or inappropriate, you can flag it to our editors by using the report abuse links. Views expressed in the comments do not represent those of Reuters. For more information on our comment policy, see http://blogs.reuters.com/fulldisclosure/2010/09/27/toward-a-more-thoughtful-conversation-on-stories/