The widespread network failure, which knocked out trading at the CME Group for well over 10 hours, has been traced to human error at a data centre operated by CyrusOne, underscoring vulnerabilities in global financial infrastructure.
Reports indicate it was caused by a technician who made an error while performing maintenance work in Aurora, Illinois. The breakdown, on November 28, brought business to a standstill at some of the world’s most critical financial markets and underscored vulnerabilities in the infrastructure that supports derivative trading globally.
CyrusOne said the incident originated from on-site staff and contractors not following “standard procedures for draining cooling towers” that cool internal infrastructure before lower temperatures set in. As the temperature outside dropped, so did the temperature inside the facility when its cooling system stopped working. This failed several key cooling units, leading to higher room temperatures in the server rooms.
The servers that host CME’s Globex trading platform began to overheat. Robotic safety systems then shut down the equipment to avoid damage. With core servers offline, CME’s markets for futures, options, and products tied to commodities, currencies, and the Treasury spectrum abruptly ceased processing trades.
The pause reverberated around the global financial system. Traders in Asia, Europe, and the United States were cut off from live prices or unable to manage their positions. At a few of the largest markets, liquidity vanished altogether, as in gold and Treasury futures, leaving some traders unprotected as they waited for systems to come back online.
Markets shudder as the shutdown reverberates around the world
The outage occurred in the early morning hours in the United States, prompting traders to assess the situation. Markets had already opened for many traders in Europe and Asia by the time prices on CME-connected products suddenly stopped moving. Futures on prominent U.S. stock indexes ceased updating. On the commodity markets, including the crude oil and gold markets, there was only eerie silence. U.S. Treasury futures, a global fixed-income trading lynchpin, were offline for several hours.
Currency trading markets were also thrown into disarray as CME’s electronic brokering system for currency trading, Globex, went offline. Traders were unable to avoid exposure or take profits. Institutional desks faced similar woes, with some stating that their standard risk management tools have become ineffective.
The disruption showed how dependent modern markets have become on a small number of data centres. With CME’s primary engine offline, there was virtually no corner of global trading able to access one of its most critical pricing feeds. Some firms reported minor internal delays as activity surged on alternative platforms, while others warned that the inability to exit positions posed significant danger. Had equities sharply sold off during the halt, they said, the resulting losses could have been devastating.
CME’s long delay before returning to Black Friday trading was caused by the venue being disconnected so that its cooling systems could be stabilized and servers brought back online.
CME rethinks data center strategy after human error
After a human error was determined to be the primary cause of the circumstances, CyrusOne stated that it has already implemented more stringent cold-weather procedures. These include increasing the number of personnel on duty during periods of extreme weather, enhancing cooling infrastructure, hardening systems, and improving redundant systems to prevent the simultaneous failure of multiple units.
The incident has reignited debate over CME’s move to depend on a data centre it no longer owns. CME sold the Aurora site in 2016. Since then, it has relied on CyrusOne to run the infrastructure that supports much of its electronic trading. The outage has also prompted questions over whether such an arrangement affords CME sufficient control over its essential systems.
And questions arose about CME’s decision not to relocate to its backup facility. Emergency plans called for relocating operations at a second location, but CME declined to activate that site, as it received early indications that the outage would be short-lived. However, the few hours of cooling loss proved far more serious, and the delay ensured that the outage would last more than ten hours.
Industry analysts say the episode should serve as a wake-up call. Modern markets aren’t just software—they’re also hardware, with power supply and cooling systems. A breakdown in any of them can halt global markets.
Join Bybit now and claim a $50 bonus in minutes
Source: https://www.cryptopolitan.com/cme-market-outage-blamed-on-human-error/
