Xbox Technical Director Scott Van Vliet followed up on the fix announced yesterday for Xbox network issues by explaining the cause and what the company plans to do to ensure this doesn’t happen again.
The root cause, according to Van Vliet, was a licensing service implemented outside of the Xbox network, affecting all sorts of license checks for logins, launching games, purchasing games, and even just viewing your full game library. Why this is an external issue when Microsoft is one of the largest server infrastructure companies in the world is a bit bizarre.
Van Vliet then explains how traffic was routed to healthier parts of the network infrastructure to restore service to users as quickly as possible, leading to mixed reports from Xbox gamers about who could and couldn’t play games.
Going forward, Van Vliet will look to eliminate single points of failure like this and, more importantly, be more transparent about what happens if something goes wrong.
One of the first factors was that the Xbox status page blithely claimed that everything was working fine, even though it retroactively identified 4:10am BST as the start of the problems, and even then the status page did not mark the game as affected, despite social accounts admitting that it was. But I would add an additional, deeper problem that should be addressed as we move closer to an all-digital future: Xbox DRM is still far too restrictive if it is also capable of removing games already downloaded for users on their “home console” of choice, and even physical games. A change in 2022 reduced the number of DRM controls for gamesbut it should go further.
Scott Van Vliet in full:
Earlier today, many gamers started experiencing issues logging in, listing their game libraries, and playing games. The team worked hard throughout the day to resolve the issue and we restored service at 2:30 p.m. PDT. However, this is an unacceptable situation and we must do better to support you.
Over the coming weeks, you’ll hear more about our services, our platform, and the work we’re doing to make XBOX better. And that starts with some transparency about this incident.
What happened. Late last night, a licensing service located outside of XBOX, but on which XBOX depends, began to fail. This caused some login scenarios to fail, as well as many scenarios requiring a rights check to fail, such as listing your entire library and launching games you own. This also affected several of our publisher and store partners who rely on these same systems, which is why some of you saw issues in specific games and not others.
Which we did. Our on-call teams picked up the first signals overnight using automated monitoring and declared a major incident. Once we isolated the failing infrastructure, we shifted traffic to the healthy parts of the system while the team continued to work on a root cause. Service returned unevenly across regions, which is why some of you recovered hours before others.
What’s next. We perform a complete post-incident review. I care less about the root cause of a single line than the real questions: why a failure in one service could destroy so much, why recovery took so long, and what we change so that a single point of failure can no longer ruin your night. This means strengthening the dependencies under logging in and launching the game, improving how we detect and contain this failure class, and being quicker and clearer with you when something breaks.
Thank you for your patience and for the reports that really helped us refine this point. We will do better.
Source: Scott VanVliet
