Google notification subscription errors
Incident Report for Cronofy
Resolved
This incident has been resolved.
Posted Jun 20, 2018 - 19:55 UTC
Update
Less than 0.1% of Google accounts do not have a full compliment of change notification subscriptions now.

We will continue to monitor but it now looks like service has been fully restored for virtually all accounts.
Posted Jun 20, 2018 - 16:36 UTC
Update
We've seen a significant drop in the number of errors when attempting to subscribe to change notifications as of around 14:40 UTC.

Google have reported that they are rolling out a fix but that it may take up to two days to propagate to all users.

We will continue to monitor and quantify the number of accounts still affected once the system has had some time to retry the failing subscription attempts.
Posted Jun 20, 2018 - 15:21 UTC
Update
From our investigations ~3% of Google accounts do not have a full compliment of change notification subscriptions, with ~0.5% having no subscription at present.

Google are aware of the issue and are working to resolve it, there is no ETA on a fix at present.

Our systems will continue to attempt to establish any missing subscriptions so once the underlying problem is resolved service will return to normal.
Posted Jun 20, 2018 - 10:42 UTC
Monitoring
We're seeing a high failure rate when subscribing to change notifications for Google calendars.

Not being able to subscribe to notifications will result in delayed notifications for changes made to affected calendars by the user.

This is not affecting all Google accounts, and in some cases we've been able to subscribe to a subset of calendars for an account. We are looking to more precisely quantify the impact.

There is an entry on Google's issue tracker (https://issuetracker.google.com/110373170) and we have reached out to our contacts at Google to try and make sure they aware of it.
Posted Jun 20, 2018 - 08:22 UTC
This incident affected: Major Calendar Providers (Google).