Snake-bit by Software–DOA (706)

The headline award goes to the blog post linked here: How to lose 172K a second for 45 minutes. That’s nearly a half billion dollars of venom. The company was DOA in under an hour.  Cause of death: long dormant programming code deep within the high-speed electronic trading system rose up and sunk its fangs into the firm’s jugular.  How? In just one of eight servers, a systems upgrade produced a coding glitch that inadvertently activated the long-forgotten trading-bot, whose corresponding cross-checks and controls no longer existed.  So, on the day of it’s debut, 12% of the system started executing trades without being restrained by how many securities the firm actually had to trade or could buy.  Then staff, trying to isolate the problem, turned off other servers, which routed more and more electronic trade orders to the renegade server operating without limits.  A “normal” accident, as unstoppable as it was statistically predictable, in the ever more complex and super-velocity world of electronic trading.