Seven servers got the new code, and the eighth ran Power Peg
Knight Capital deployed code for the NYSE's new Retail Liquidity Program to seven of the eight servers in its order router, SMARS, and missed the eighth. The new code reused a flag that once switched on Power Peg, a feature retired in 2003, so on the eighth server the flag woke the old code, which no longer stopped when an order was filled. In about 45 minutes it sent millions of orders for 212 customer orders, and Knight lost more than $460 million.
Times are approximate, in ET (UTC−4), from the sources at the end. The diagram is simplified.
At 09:30 ET on Wednesday 1 August 2012 the New York Stock Exchange opened, and one of the eight servers in Knight Capital’s order router began sending orders that it would not stop sending. Over about 45 minutes, working from 212 small retail orders, it sent millions of orders into the market and received more than 4 million executions in 154 stocks, for more than 397 million shares. Knight, whose trading made up about a tenth of all trading in US-listed stocks, lost more than $460 million.
This account follows the SEC’s order against Knight, issued in October 2013.
How an order router works
A broker-dealer like Knight takes orders from customers, including other brokers passing on their own customers’ orders, and gets them executed. Execution happens on trading venues: exchanges and other places where buyers meet sellers. No single venue always has enough shares at the right price, so large firms use a router: software that decides where to send each order and how to split it up.
Knight’s router for equities was called SMARS, which the SEC describes as an automated, high-speed, algorithmic router. It received parent orders from the rest of Knight’s trading platform and, depending on the liquidity available, sent one or more child orders to venues to fill them.
Say a customer wants to buy 1,000 shares. The router might send child orders for a few hundred shares to several venues at once. Executions (fills) come back, and the router has to add them up and stop sending child orders once the total reaches 1,000. That bookkeeping is the whole difference between a router and a machine that buys shares forever.
Code like this is full of flags: fields on an order, or settings, that choose which path the software takes. A flag has no meaning by itself. It means whatever the code on the machine reading it says it means.
What happened
The Retail Liquidity Program, a new NYSE programme for retail orders, was due to start on 1 August 2012. To let its customers take part, Knight changed several systems, including new code in SMARS.
The new code was meant to replace old, unused code in the same part of the router. That code belonged to a feature called Power Peg, which Knight had stopped using in 2003 but never removed: it was still present and could still be called. The new code also took over the flag that used to switch Power Peg on. The plan was coherent: Power Peg would be deleted in the same change, so from then on, an order with that flag set would get the new behaviour.
One more thing had happened in between. In 2005 Knight moved Power Peg’s cumulative quantity function, the part that counted how many shares of a parent order had been filled and stopped routing at the total, to an earlier point in the SMARS code, for use elsewhere. Nobody retested Power Peg afterwards. There was no obvious reason to: it was not in use. But from then on, if anything ever called Power Peg, it would send child orders without ever stopping.
From 27 July, Knight deployed the new code in stages, a few servers on successive days. A technician did not copy it to one of the eight SMARS servers. No second technician reviewed the deployment, and Knight had no written procedure that required one. Nobody noticed that the eighth server still had Power Peg and did not have the new code.
The first sign came before the open. From about 08:01, orders for the new programme that were marked for pre-market trading went through SMARS, and an internal system started sending automated emails, called BNET rejects, that mentioned SMARS and an error described as Power Peg disabled. They were caused by the failed deployment, and the system sent 97 of them to a group of Knight staff before the 09:30 open. Those emails were not designed as alerts and were not normally read. Nobody acted on them.
At 09:30 the market opened. The seven updated servers handled their orders correctly. On the eighth, orders carrying the repurposed flag started Power Peg. With its fill count gone, it sent child orders in rapid succession for each parent order, regardless of how many shares had already been executed. Another part of Knight’s system did recognise that the parent orders were filled; that information never reached SMARS.
Senior staff saw a large volume of positions building up in PMON, Knight’s main risk monitor, almost as soon as the market opened. The positions were landing in what Knight called the 33 Account, which temporarily held executions its systems could not match to an open parent order. But the 33 Account held positions from several sources, so nobody could quickly tell what these were or where they came from. PMON only displayed data to people watching it: it raised no automatic alerts, did not show the limits, and ran behind during heavy trading like this.
Knight had no written procedure for an incident like this one. Its technology team tried to find the problem in a live market while SMARS kept sending orders. In one attempt they uninstalled the new code from the seven servers where it had been deployed correctly. That put those servers back on the old code, Power Peg included, and new parent orders with the flag set started activating Power Peg there too.
About 45 minutes after it started, Knight stopped sending the orders. By then it held a net long position of about $3.5 billion in 80 stocks and a net short position of about $3.15 billion in 74 more. Many of the orders had been short sales that were not marked as such. The loss on closing the positions came to more than $460 million.
In 75 of the stocks, Knight’s executions were more than 20% of the trading volume and contributed to price moves of more than 5%. In 37 of those, prices moved more than 10% and Knight was more than half of the volume. Other traders got better or worse prices than they otherwise would have.
Why 45 minutes cost $460 million
Each decision along the way made sense when it was taken. Leaving unused code in place looked harmless. Moving a shared function in 2005 didn’t seem to need a test of a feature nobody used. Reusing the flag was safe as long as Power Peg was deleted everywhere at once. Rolling back is what you do when trouble starts right after a release. What turned them into a $460 million loss was the missing checks around them:
The deploy was never verified. A technician copied code onto servers, with no second person checking and no automatic comparison of what each server was running. A single missed copy stayed invisible until the market opened.
The meaning of the flag depended on the server. A flag is only as safe as the oldest code that can read it. With Power Peg still present on one machine, the same order meant two different things depending on where it landed.
The only warnings were emails nobody read. For an hour and a half before the open, the system was saying, in plain words, that Power Peg was involved. The messages went to people, not to monitoring.
Nothing stood between the router and the market. SMARS had no control that compared the orders leaving it with those that entered it, and no procedure to halt it when it misbehaved. Knight did cap the limit price of each parent order and its children at 9.5% beyond the best market price when the order arrived, but that would not catch errors while prices moved less than that, and it did not apply to orders like these 212, which arrived before the open to take part in the opening auction.
Money limits did not touch order entry. Knight’s position limits did not count exposure from outstanding orders, and the firm had no firm-wide capital threshold that would automatically stop orders when crossed. The 33 Account had a $2 million gross position limit, but it was not connected to any automated control, so SMARS kept sending orders long after the account was past it.
The fix made it worse. Without a procedure that said when to disconnect a malfunctioning system from the market, the team debugged live, and the rollback spread the fault from one server to eight.
There was also an earlier warning at the level of the firm. In October 2011, after a weekend disaster-recovery test, one of Knight’s desks kept using the test data to generate automatic quotes when trading resumed on Monday, and lost nearly $7.5 million. Knight fixed that specific failure, but, as the SEC noted, did not ask more broadly whether its controls could stop erroneous orders from any system.
How it ended
The loss left Knight with significant net capital problems, and on 1 July 2013 Knight’s parent company combined with GETCO to form KCG Holdings.
In October 2013 the SEC found that Knight had violated the market access rule, Rule 15c3-5, which requires brokers with direct access to the markets to have controls that prevent erroneous orders and orders that exceed pre-set capital limits, and that it had broken short-sale rules on 1 August. Its 2012 certification of compliance, signed by the CEO, had stated only that Knight had processes to comply, a drafting error the CEO had not noticed. Knight settled without admitting or denying the findings. It was censured, ordered to cease and desist, fined $12 million, and required to hire an independent consultant to review its software development lifecycle, code deployment, order routers, capital thresholds and incident response.
The order is blunt about how small the fixes could have been. A written procedure requiring a simple double-check of the deployment could have found the missed server. Wiring the BNET reject emails into monitoring could have caught the problem before the open. A rule requiring Power Peg to be retested in 2005 could have shown that its fill count was gone. Any one of them might have made 1 August an uneventful launch day.
What I'd take from it
- Delete dead code; don't just stop calling it. Power Peg sat on production servers for nine years after it was retired, still callable. Code that isn't there can't wake up. Remove it, and anything that could still route to it.
- Never give an old flag a new meaning. Every server that still had the old code read the flag the old way. New behaviour gets a new flag, and an old one is retired only after nothing can read it.
- Ask the fleet whether the deploy worked. Nobody checked that all eight servers ran the new code. I'd make the deploy tool refuse to finish until every target reports the new build, and page someone when one doesn't.
- Put the last check next to the exit. SMARS had no control comparing orders going out with orders coming in, and no procedure to halt it. The check closest to the market is the one that still works when everything upstream is wrong.
- A rollback is a deploy too. Removing the new code from the seven good servers put them back on the old code with the same flag, and made things worse. Before rolling back, know what the old version does with today's inputs.