Degraded Google calendar sync

Incident Report for Cronofy

Resolved

Calendar sync for Google-backed calendars has remained healthy since the previous message, so we are considering this as resolved.

Google have updated their incident record of the underlying issue, where they likewise consider it resolved: https://www.google.com/appsstatus/dashboard/incidents/7uJZ5F1Uy4n1n74iMacQ

Posted Sep 21, 2023 - 17:31 UTC

Update

Error rates from Google's API have returned to normal levels, and calendar syncs for Google-backed calendars are healthy again.

Google Calendar sync performance dropped starting at 13:30 UTC, and was heavily degraded until improvements were seen starting at 16:00 UTC. A full recovery of service was reached around 16:10 UTC.

Posted Sep 21, 2023 - 16:45 UTC

Update

We're observing a much improved success rate from Google's Calendar API since 16:00 UTC.

Google have advised that a fix has been made on their side and is rolling out currently, and expect to be fully resolved within the hour: https://www.google.com/appsstatus/dashboard/incidents/7uJZ5F1Uy4n1n74iMacQ

We’ll continue to monitor the situation and update here.

Posted Sep 21, 2023 - 16:17 UTC

Update

Google’s Calendar API is still returning a high error rate, with no notable change to the situation since the previous update.

Google are tracking the incident at their incident dashboard
at https://www.google.com/appsstatus/dashboard/incidents/7uJZ5F1Uy4n1n74iMacQ

We’ll continue to monitor the situation and update here.

Posted Sep 21, 2023 - 15:55 UTC

Update

We continue to observe a high error rate on Google’s Calendar API. This appears to affect some user cohorts more than others, rather than being spread evenly across all of our syncronized Google calendars.

While this doesn’t affect the error rate of Cronofy’s API, it means that we’re failing to pick up the latest changes to affected users’ calendars, and may present stale availability information. Events written via the Cronofy API may be delayed before being pushed successfully to the external Google calendar. We queue and retry such operations automatically and do not anticipate any remedial action to be necessary as a result.

We'll continue to monitor things and update accordingly.

Posted Sep 21, 2023 - 14:51 UTC

Monitoring

Errors when communicating with Google calendar increased significantly across all data centers from 13:31 UTC.

We have taken action to reduce the impact on other calendar providers and continue to monitor.

Posted Sep 21, 2023 - 13:46 UTC

Investigating

We are seeing a significantly higher than normal amount of errors when trying to interact with Google calendars.

Posted Sep 21, 2023 - 13:40 UTC

This incident affected: Major Calendar Providers (Google) and Background Processing.