Elevated latency and error rates on Skedda web and mobile apps
Incident Report for Skedda
Resolved
This incident has been resolved.
Posted Sep 07, 2022 - 20:32 UTC
Update
Although we are observing normal performance on our side, we are continuing to monitor Azure's status page for confirmation that residual impact has been resolved (https://status.azure.com/en-us/status). We will provide another update at that time.
Posted Sep 07, 2022 - 19:55 UTC
Update
We are observing normal performance levels again since we implemented the mitigation step. We are continuing to monitor and will provide another update within 60 minutes or as events warrant.
Posted Sep 07, 2022 - 18:15 UTC
Update
We have implemented a network mitigation step from our side in order to reduce the effects of the Azure Front Door incident. We are seeing signs of recovery for a significant portion of traffic. We are continuing to monitor and will provide an update within 30 minutes or as events warrant.
Posted Sep 07, 2022 - 17:36 UTC
Monitoring
Microsoft has confirmed the incident on its status page: https://status.azure.com/en-us/status

Considering the magnitude of this incident for Microsoft's services generally (Azure Front Door is used by many Microsoft services), we anticipate that Microsoft will resolve the issue promptly and we are continuing to monitor their efforts. We will provide an update within 30 minutes or as events warrant.
Posted Sep 07, 2022 - 17:06 UTC
Identified
We have isolated the cause of the issue to Skedda's Web-Application Firewall layer, which consists of a global network of edge nodes for performance and traffic management. The service that Skedda uses for this layer, Azure Front Door, is currently experiencing intermittent connectivity and performance issues when forwarding requests to our back-end servers.

Retrying requests may succeed, and some regions/geographies may be less effected than others.

We are continuing to work on a resolution for this issue and will provide an update within 30 minutes or as events warrant.
Posted Sep 07, 2022 - 16:58 UTC
Investigating
We are currently investigating intermittently elevated latency and errors on our web and mobile applications. We will provide an update within 20 minutes or as events warrant.
Posted Sep 07, 2022 - 16:41 UTC
This incident affected: Admin Dashboard and User Portal.