There have been no further issues and we are marking this incident as resolved. In summary, there were three or four short periods of time this morning (UTC time), each lasting no more than two or three minutes, where a portion of requests to the Skedda application (those handled by a web server in a particular availability zone) took longer to respond. The root cause was transient network errors on one of our database gateway clusters in that availability zone.
Posted Dec 02, 2021 - 08:58 UTC
Monitoring
We haven't seen any further errors or latency since the last update. The application is running normally for all users currently. We're in communication with our hosting provider about the infrequent transient latency spikes in the last few hours and will continue to monitor. We will provide an update here if the situation regresses or to mark this incident resolved.
Posted Dec 02, 2021 - 06:40 UTC
Identified
Over the last few hours, there have been short windows of time (usually a few minutes in duration each) during which requests to Skedda have had a higher latency and may "time out" and cause errors. At present, the Skedda application is functioning normally, however we are monitoring for further latency spikes. Skedda users affected by this issue going forward should have success when retrying after a short period of time.
The underlying cause is related to maintenance operations on the Skedda database layer, and we have contacted our hosting provider for further information on whether this will recur and to continue investigating this.
We will provide an update within 60 minutes or as events warrant.
Posted Dec 02, 2021 - 06:11 UTC
Investigating
We're currently investigating intermittent latency problems on Skedda. Users may currently experience longer loading times. We will provide an update within 30 minutes or as events warrant.
Posted Dec 02, 2021 - 05:47 UTC
This incident affected: Admin Dashboard and User Portal.