Last week, a major AWS outage took down gaming behemoths like Fortnite and Roblox and caused wide chaos by also impacting important services and programs outside of the gaming world. The cause of the outage has now been identified, revealing that the massive outage was the result of a single software bug.
As initially reported while Amazon was trying to figure out the cause of the 15-hour outage, the problem originated in DynamoDB, AWS’s DNS management system. As reported by PC Gamer, the issue originated with DNS Enactor, a component of DynamoDB that constantly updates domain lookup tables to stay on top of load balancing as conditions change.
The bug started when the DNS Enactor started to lag, and “experienced unusually high delays needing to retry its update on several of the DNS endpoints.” The DNS Planner continued to generate new plans, and while the first Enactor was delayed, a second DNS Enactor began to implement those plans.
Once the second Enactor had implemented the new plans, it began the clean-up process, which deletes all plans older than the one it had just implemented. At the same time, the first Enactor finally caught up and overwrote the newer plan, with a check that should have prevented that from happening also failing due to the high delays. The clean-up from the second Enactor then deleted the older plan, which had become the active plan. “As this plan was deleted, all IP addresses for the regional endpoint were immediately removed,” Amazon engineers explained, with the deletion of the active plan causing an “inconsistent state” that required manual intervention from an operator to eventually correct.
The cascade of bugs eventually led to a 15-hour outage, Amazon reported, and network intelligence company Ookla, which runs Downdetector, estimated that it was one of its largest outages on record. The outage took out gaming services including Roblox, Fortnite, Pokemon Go, PlayStation Network, Rocket League, and Wordle, along with a huge number of non-gaming services and platforms.
Comments are closed, but trackbacks and pingbacks are open.