Cisco Webex Goes DownCisco Webex Goes Down

Yesterday, Sept. 25, Cisco Webex users began to report service issues, some of which – mainly for Webex Teams -- continue into today.

Cisco first identified a major outage on Monday at 9:22 P.M. ET on its status website, at which point the company stated: "Webex Teams services are currently impacted by an ongoing service outage. Engineering resources are online and working to restore services. We apologize for the impact and all hands are on deck to restore Teams, Meetings, Calling, Care and Context."

Roughly three minutes after Webex Teams reported the outage, the status page for Webex Messaging and Webex Meetings alerted users that availability of Webex Video Platform 2.0 was impacted by a known issue. As a result, users were unable to connect to Webex meetings from video endpoints or Webex Teams.

While Amy Chang, SVP of Cisco's Collaboration Technology Group, did not address the issue, at least publicly, CEO Chuck Robbins took to Twitter yesterday evening to address the outage and give assurances to users:

In addition, Cisco has been providing regular updates via its Webex Twitter page.

Fixes Ongoing
As of this writing, Cisco has reported that it has restored Webex Meetings to full operational status. It didn't disclose details, saying only that engineers rolled out a fix for a "specific behavior in the Webex infrastructure" that was affecting Video Platform 2.0 users.

Additionally, Cisco engineers have restored core messaging capabilities, and Webex Messenger is now operational again, with "full message and space history being progressively restored," Cisco said on its Webex Teams status page. In addition, "files are available, and avatar updates are functioning."

If users have previously stored messages on their device, they will be able to access message history. However, additional message history still aren't available from Cisco's cloud services, Cisco said. Other messaging issues that persist include the inability to flag messages or see user presence and continued intermittent issues with some 1:1 spaces. Further, while Cisco has restored whiteboarding and pairing capabilities for Webex devices, it is still working on restoring "one button to push" and the "ability to bind to a space," as well as device administration capabilities through Control Hub.

As of 10:17 A.M. ET, Webex Calling device activation is operational, and Cisco also reported that Webex Desktop Calling is unaffected by the incident. At 11:33 A.M. ET, Cisco noted the persistence of latency and errors with Webex Teams connectivity and room creation. At 12:50 P.M. ET, Cisco shared that "engineering is deploying additional services to address the ongoing Teams connectivity issues" that is causing some users to experience intermittent access or latency when they connect to Messaging using any of the Webex Teams apps. And the latest update as of this writing, at 2:26 P.M. ET, indicates that user latency with Webex Teams and Webex Devices is still expected as engineering continues to deploy additional fixes.

We will update this article as more information becomes available.

[Update]: At 3:43 P.M. ET, Cisco reported it has completed its deployments and Messaging services are now stabilizing, but that additional efforts around service restoration are still ongoing. "Some spaces or files posted during the period of latency may show decryption messages such as 'Decrypting title' or 'Decrypting content.' Users that see this message may restart their Teams client or leave and rejoin the space to allow the space or content to refresh," Cisco said on its Webex Teams status page. Additionally, Cisco advised Webex Teams users to restart any file downloads that stalled or failed during the latency period.

[Update]: At 5:45 P.M. ET, Cisco reported that Developer API services are now restoring. Additional service restoration efforts are ongoing.

Follow Michelle Burbick and No Jitter on Twitter!
@nojitter
@MBurbick

Related content: