On October 22, 2020, the Commission’s Public Safety and Homeland Security Bureau (“Bureau”) released a Network Outage Report on the nationwide T-Mobile network outage on June 15, 2020 (PS Docket No. 20-183).
The Bureau estimates that at least 41% of all calls attempting to use T-Mobile’s network during the outage failed, including at least 23,621 failed calls to 911. Based on the results of the investigation, the Bureau recommends that network operators periodically audit the diversity of their networks and take appropriate measures to ensure network resilience.
The Bureau found that the root cause stemmed from T-Mobile installing routers in the southeast region of its network on June 15, 2020. During the installation and router configuration process, T-Mobile misconfigured the weights of the links of another router that was already active in the network but was not designed to process call signaling traffic. T-Mobile did not have a “fail-safe” process to prevent or provide notice of this misconfiguration.
Then, a fiber transport link in the southwest region of the network also failed, which was exacerbated by the routing misconfiguration. This caused the Atlanta market to become isolated and mobile device registration attempts in that market timed out. Mobile devices repeatedly attempted and failed to register using Wi-Fi and this network activity further exacerbated the congestion. The Voice over Wi-Fi and VoLTE registration attempts then spread across the country.
T-Mobile attempted to mitigate the effects of the outage through public communication, including direct communications with subscribers, and by notifying PSAPs. The Bureau did not receive complaints from PSAPs that T-Mobile did not notify them. However, public safety officials in Seminole County, Florida and Jefferson County, Colorado did send out emergency alerts related to the outage.
The Bureau concludes that T-Mobile did not follow network reliability best practices that could have prevented the outage or limited its effect. The Bureau will engage in stakeholder outreach and guidance regarding industry-accepted recommended network reliability best practices to protect against similar outages in the future.
Please Contact Us if you have any questions.