tag:status.cronofy.com,2005:/historyCronofy Status - Incident History2024-03-29T05:23:36ZCronofytag:status.cronofy.com,2005:Incident/203779202024-04-07T07:00:00Z2024-03-27T14:31:46ZDatabase instance upgrade in our UK data center<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>7</var>, <var data-var='time'>07:00</var> - <var data-var='time'>08:00</var> UTC</strong></p><p><small>Mar <var data-var='date'>27</var>, <var data-var='time'>14:31</var> UTC</small><br><strong>Scheduled</strong> - We will be upgrading the instance types used within our PostgreSQL cluster in our UK data center.<br /><br />As changing the instance types requires a failover when we switch to the new instances there will be up to two minutes of disruption during this time. This will happen shortly after 07:00 UTC on Sunday, 7th of April 2024, and we expect the instance type change to be completed by 08:00 UTC.<br /><br />We will provide notifications as this maintenance progresses via our status page.<br /><br />If you have any questions, please email support@cronofy.com.</p>tag:status.cronofy.com,2005:Incident/192512882023-12-10T08:30:45Z2023-12-10T08:30:45ZMinor database upgrade in all data centers<p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>08:30</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed successfully. We observed disruption at the following times:<br /><br />Singapore: 07:05:45 - 07:06:00<br />Australia: No disruption observed<br />UK: 07:34:15 - 07:34:45<br />Canada: 07:45:30 - 07:45:45<br />Germany: 07:55:15 - 07:56:00<br />US: 08:09:00 - 08:09:45<br /><br />If you have any questions, please email support@cronofy.com.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>08:19</var> UTC</small><br><strong>Verifying</strong> - The update is complete in our US data center. We observed disruption from 08:09:00 to 08:09:45 UTC.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>08:01</var> UTC</small><br><strong>Update</strong> - The update is complete in our Germany data center. We observed disruption from 07:55:15 to 07:56:00 UTC.<br /><br />We are beginning the update in our US data center.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>07:50</var> UTC</small><br><strong>Update</strong> - The update is complete in our Canada data center. We observed disruption from 07:45:30 to 07:45:45 UTC.<br /><br />We are beginning the update in our Germany data center.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>07:40</var> UTC</small><br><strong>Update</strong> - The update is complete in our UK data center. We observed disruption from 07:34:15 to 07:34:45 UTC.<br /><br />We are beginning the update in our Canada data center.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>07:28</var> UTC</small><br><strong>Update</strong> - The update is complete in our Australia data center. There was no observed disruption during this update.<br /><br />We are beginning the update in our UK data center.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>07:14</var> UTC</small><br><strong>Update</strong> - The update is complete in our Singapore data center. We observed disruption from 07:05:45 to 07:06:00 UTC.<br /><br />We are beginning the update in our Australia data center.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>In progress</strong> - The scheduled maintenance is in progress. We are beginning the update in our Singapore data center.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>16:30</var> UTC</small><br><strong>Scheduled</strong> - We will be performing minor PostgreSQL database engine upgrades in our Aurora clusters across all data centers.<br /><br />Minor database engine upgrades require a reboot of the database server, therefore there will be up to two minutes of disruption when this happens. We will upgrade each data center one at a time and will begin the first upgrade shortly after 07:00 UTC and expect the upgrades to be completed by 09:00 UTC on Sunday, 10th December 2023.<br /><br />We will provide notifications as this maintenance progresses via our status page.<br /><br />If you have any questions, please email support@cronofy.com.</p>tag:status.cronofy.com,2005:Incident/192483302023-11-30T10:05:47Z2023-11-30T10:05:47ZApple sync degraded<p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>10:05</var> UTC</small><br><strong>Resolved</strong> - At 07:47 UTC, we saw a sharp increase in the number of Service Unavailable errors being returned from Apple’s calendar servers across all of our data centers, causing sync operations to fail.<br /><br />This was escalated to our engineering team, who investigated and found that no other calendar providers were affected and so the issue was likely not within our infrastructure. However, very few operations between Cronofy and Apple were succeeding.<br /><br />At 08:20 UTC, we opened an incident, to mark Apple sync as degraded, as customers may have seen an increased delay in calendar sync.<br /><br />This coincided with a sharp drop in the level of failed network calls, which returned to normal levels at 08:18 UTC.<br /><br />The service stabilized and Cronofy automatically retried failed sync operations to reconcile calendars.<br /><br />Over the next hour, we saw communications with Apple return to a mostly healthy state, though there were still occasional spikes in the number of errors. Cronofy continued to automatically retry failed operations, so impact on users was minimal.<br /><br />At 09:15 UTC, these low numbers of errors decreased back to baseline levels and stayed there.<br />As we’ve now seen more than 30 minutes of completely healthy service, we are resolving the incident</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>09:25</var> UTC</small><br><strong>Update</strong> - We are still observing small numbers of errors. These are being automatically retried so the service is largely unaffected, and we are continuing to monitor.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>08:50</var> UTC</small><br><strong>Update</strong> - We are still seeing occasional, smaller number of errors. These are being retried automatically by Cronofy and the service is largely unaffected. We are continuing to monitor error levels.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>08:29</var> UTC</small><br><strong>Monitoring</strong> - We saw a rise in Service Unavailable errors from Apple's calendar servers between 07:46 UTC and 08:19 UTC.<br />Normal operation has resumed. Cronofy will automatically retry any failed communications with Apple, so no further intervention is required.<br />We are continuing to monitor the situation to be sure that the incident is over.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>08:20</var> UTC</small><br><strong>Investigating</strong> - We are investigating unusually high numbers of errors when syncing Apple calendars.</p>tag:status.cronofy.com,2005:Incident/189534432023-10-28T16:05:10Z2023-10-28T16:05:10ZDegraded Apple calendar sync<p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>16:05</var> UTC</small><br><strong>Resolved</strong> - Error rates from Apple's API have returned to normal levels, and calendar syncs for Apple-backed calendars are healthy again.<br /><br />Apple Calendar sync performance dropped starting at 13:36 UTC and finishing at 15:43 UTC. During this time no other calendar provider sync operations were affected.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>15:03</var> UTC</small><br><strong>Update</strong> - We continue to observe a high error rate on Apple's Calendar API. Around 22% of Apple Calendar operations are resulting in an error.<br /><br />We'll continue to monitor things and update accordingly.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>14:07</var> UTC</small><br><strong>Monitoring</strong> - Errors when communicating with Apple calendar increased significantly across all data centers from 13:36 UTC.<br /><br />This is not affecting communications with any other calendar providers. We continue to monitor the situation.</p>tag:status.cronofy.com,2005:Incident/188648042023-10-20T17:18:56Z2023-10-23T16:24:20Z8x8 conferencing requesting moderator login<p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>17:18</var> UTC</small><br><strong>Resolved</strong> - Since launching support for provisioning video conferencing when creating a calendar event in 2020 we have used 8x8.vc links as an explicit option and as an anonymous, browser-based conferencing fallback when calendar-native conferencing providers such as Google Meet and Microsoft Teams are not available.<br /><br />8x8 accounts for around 1% of the video conferencing we have provisioned in recent weeks, the other 99% being made up of calendar-integrated conferencing solutions like Google Meet and standalone providers like Zoom.<br /><br />8x8 removed support for 8x8.vc links being used anonymously earlier this week, without any notification and with no available workaround.<br /><br />Ideally, we would have received prior notification from 8x8 of this change so that we could have managed a graceful transition. As that did not happen, we will regrettably be dropping support for 8x8 as a conferencing option with immediate effect.<br /><br />When using our API, "8x8" can be selected explicitly, or chosen as a fallback when using the "default" conferencing option. We have released a change which stops the generation of the anonymous, browser-based 8x8.vc conferencing links and so calendar events will be created, but without any conferencing details.<br /><br />Put another way, "8x8" will have a similar effect to providing "none", and there will no longer be a catch-all conferencing option provisioned when using "default".<br /><br />https://docs.cronofy.com/developers/api/conferencing-services/create/#param-conferencing.profile_id<br /><br />We will continue to accept both values to as to not break any existing integrations. "8x8" has been deprecated and the documentation relating to "default" has been updated to reflect this change in behavior.<br /><br />If you subscribe to notifications based on conferencing being provisioned, you will be notified of the failure to provision any conferencing in cases where 8x8 would previously have been used.<br /><br />https://docs.cronofy.com/developers/api/conferencing-services/subscriptions/<br /><br />We truly regret this situation and can only apologize for the disruption this has caused. As there is no further action we are able to take on this, we are resolving this incident.<br /><br />If you have any further questions, please contact us at support@cronofy.com</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>14:59</var> UTC</small><br><strong>Update</strong> - [Message edited for easier reading]<br /><br />We announced the changes detailed in the resolution message but in the future tense.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>13:18</var> UTC</small><br><strong>Update</strong> - We have had a response from 8x8 confirming that this change in service is intentional, will not be reversed, and that there is no workaround for any existing 8x8.vc links.<br /><br />This is the worst outcome we could foresee and are sorry for any disruption this is causing.<br /><br />We will be finalizing and sharing our plan for handling this by 15:00 UTC.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>09:35</var> UTC</small><br><strong>Identified</strong> - Since launching support for provisioning video conferencing when creating a calendar event in 2020 we have used 8x8.vc links as an explicit option and as a fallback when calendar-native conferencing providers such as Google Meet and Microsoft Teams are not available.<br /><br />These links were previously anonymous, requiring no account to be set up for them to be used. These links no longer support anonymous access, instead users will now be presented with "Waiting for moderator" and an option to login.<br /><br />There does not appear to be a way for anyone to create an account meaning these 8x8 links are currently unusable with no known workaround.<br /><br />We did not receive any notice of this change, we first received reports relating to this on Tuesday October 17th, and have since confirmed the change in behavior ourselves.<br /><br />We have reached out to our contact at 8x8 about this change in their service and are evaluating our options</p>tag:status.cronofy.com,2005:Incident/185664212023-09-21T17:31:56Z2023-09-21T17:31:56ZDegraded Google calendar sync<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>17:31</var> UTC</small><br><strong>Resolved</strong> - Calendar sync for Google-backed calendars has remained healthy since the previous message, so we are considering this as resolved.<br /><br />Google have updated their incident record of the underlying issue, where they likewise consider it resolved: https://www.google.com/appsstatus/dashboard/incidents/7uJZ5F1Uy4n1n74iMacQ</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>16:45</var> UTC</small><br><strong>Update</strong> - Error rates from Google's API have returned to normal levels, and calendar syncs for Google-backed calendars are healthy again.<br /><br />Google Calendar sync performance dropped starting at 13:30 UTC, and was heavily degraded until improvements were seen starting at 16:00 UTC. A full recovery of service was reached around 16:10 UTC.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>16:17</var> UTC</small><br><strong>Update</strong> - We're observing a much improved success rate from Google's Calendar API since 16:00 UTC.<br /><br />Google have advised that a fix has been made on their side and is rolling out currently, and expect to be fully resolved within the hour: https://www.google.com/appsstatus/dashboard/incidents/7uJZ5F1Uy4n1n74iMacQ<br /><br />We’ll continue to monitor the situation and update here.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>15:55</var> UTC</small><br><strong>Update</strong> - Google’s Calendar API is still returning a high error rate, with no notable change to the situation since the previous update.<br /><br />Google are tracking the incident at their incident dashboard <br /> at https://www.google.com/appsstatus/dashboard/incidents/7uJZ5F1Uy4n1n74iMacQ<br /><br />We’ll continue to monitor the situation and update here.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>14:51</var> UTC</small><br><strong>Update</strong> - We continue to observe a high error rate on Google’s Calendar API. This appears to affect some user cohorts more than others, rather than being spread evenly across all of our syncronized Google calendars.<br /><br />While this doesn’t affect the error rate of Cronofy’s API, it means that we’re failing to pick up the latest changes to affected users’ calendars, and may present stale availability information. Events written via the Cronofy API may be delayed before being pushed successfully to the external Google calendar. We queue and retry such operations automatically and do not anticipate any remedial action to be necessary as a result.<br /><br />We'll continue to monitor things and update accordingly.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>13:46</var> UTC</small><br><strong>Monitoring</strong> - Errors when communicating with Google calendar increased significantly across all data centers from 13:31 UTC.<br /><br />We have taken action to reduce the impact on other calendar providers and continue to monitor.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>13:40</var> UTC</small><br><strong>Investigating</strong> - We are seeing a significantly higher than normal amount of errors when trying to interact with Google calendars.</p>tag:status.cronofy.com,2005:Incident/182084222023-08-21T10:13:10Z2023-08-29T13:57:14ZIncreased error rates<p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>10:13</var> UTC</small><br><strong>Resolved</strong> - A change deployed at 09:08UTC introduced a bug that lead to an internal component being overwhelmed and dropping API requests. Our alerting spotted this and we rolled back, completing at 09:19UTC.<br /><br />We will be conducting a post-mortem to understand why this wasn’t caught before release and to improve our QA processes.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>09:31</var> UTC</small><br><strong>Investigating</strong> - We’re investigating an increased error rate in all data centers; the error rate has returned to normal but we’re investigating any side affects.</p>tag:status.cronofy.com,2005:Incident/178361022023-07-12T15:00:00Z2023-07-13T16:05:36ZOutlook and Zendesk Scheduler Extension Loading issue<p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>15:00</var> UTC</small><br><strong>Resolved</strong> - From 14:07 UTC - 16:08 UTC, Scheduler Extensions (such as Outlook and Zendesk) were not able to load the Scheduler, instead showing only the loading spinner.<br />Scheduler integrations (such as Greenhouse and Workday) were unaffected.<br /><br />This was due to an update being deployed which expected some data only available on the Scheduler website but not in the extensions, which was not caught by our tests or QA before release. We apologise for any inconvenience and will be improving our processes to be more rigorous.</p>tag:status.cronofy.com,2005:Incident/175636402023-06-13T21:33:54Z2023-06-13T21:33:54ZUS data center infrastructure issue<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>21:33</var> UTC</small><br><strong>Resolved</strong> - AWS's us-east-1 region, where our US data center is hosted, experienced an issue affecting some services Cronofy's platform relies upon.<br /><br />The impact to service was low, at its peak resulting in a small degradation in performance within our US data center and a handful of server errors being returned by the service.<br /><br />This incident has been resolved.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>21:09</var> UTC</small><br><strong>Update</strong> - AWS are reporting many services as fully recovered in us-east-1 and we have observed stable service in our US data center since around 21:45 UTC.<br /><br />We will continue to monitor but expect to resolve this incident soon.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>20:41</var> UTC</small><br><strong>Update</strong> - The issue with AWS Security Token Service in us-east-1 has become worse for roughly the past 15 minutes.<br /><br />This is mainly impacting scheduling tasks from triggering as they are not able to authenticate properly within the US data center to perform their tasks. This will lead to degraded service in tasks such as the polling of Apple calendars.<br /><br />The impact of this has been minimized for core services as we prevented them from scaling in earlier in the incident.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>20:05</var> UTC</small><br><strong>Update</strong> - The impact on service in our US data center still appears to be minimal.<br /><br />AWS are resolving the underlying issue in us-east-1 and we will continue to monitor until their incident is closed.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>19:44</var> UTC</small><br><strong>Monitoring</strong> - We have seen elevated errors in our US data center, these appear to be related to an open issue in AWS us-east-1: https://health.aws.amazon.com/health/home#/account/dashboard/open-issues?eventID=arn:aws:health:us-east-1::event/MULTIPLE_SERVICES/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE_798F8_DDDD18AFADD&eventTab=details<br /><br />The impact appears low at present, but given it is at the infrastructure level it could quickly become significant and so we have opened this incident.</p>tag:status.cronofy.com,2005:Incident/173313292023-06-11T10:00:55Z2023-06-11T10:00:55ZMajor database upgrade in our DE, SG and US data centers<p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>10:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been successfully completed. Our application was unavailable in each data center during the following times (UTC):<br /><br />DE: 06:16 - 07:33<br />US: 07:41 - 08:54<br />SG: 09:05 - 09:55<br /><br />The maintenance in our DE and US data centers took longer than anticipated. When the upgraded database cluster came back online and we initialized background tasks, the performance of the data center was significantly worse than expected. Our investigation found that this was down to a poor query plan being used on a hotpath through the system. We failed over the database cluster to force a reset of the database's query planning and this resolved the issue and we proceeded to bring the data center fully online.<br /><br />If you have any questions, please email support@cronofy.com.</p><p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>09:58</var> UTC</small><br><strong>Update</strong> - The maintenance in our Singaporean data center is complete. Our application was unavailable from 09:05 to 09:55 UTC.</p><p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>08:58</var> UTC</small><br><strong>Update</strong> - We are beginning the scheduled maintenance in our Singaporean data center.</p><p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>08:55</var> UTC</small><br><strong>Update</strong> - The maintenance in our US data center is complete. Our application was unavailable from 07:41 to 08:54 UTC.<br /><br />The maintenance in our DE and US data centers took longer than anticipated. When the upgraded database cluster came back online and we initialized background tasks, the performance of the data center was significantly worse than expected. Our investigation found that this was down to a poor query plan being used on a hotpath through the system. We failed over the database cluster to force a reset of the database's query planning and this resolved the issue and we proceeded to bring the data center fully online.</p><p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>07:37</var> UTC</small><br><strong>Update</strong> - We are beginning the scheduled maintenance in our US data center.</p><p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>07:34</var> UTC</small><br><strong>Update</strong> - The maintenance in our German data center is complete. Our application was unavailable from 06:16 to 07:33 UTC.<br /><br />This took longer than expected, we will provide further details in our closing maintenance message.</p><p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>06:01</var> UTC</small><br><strong>In progress</strong> - We are beginning the scheduled maintenance in our German data center.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>16:47</var> UTC</small><br><strong>Scheduled</strong> - This maintenance window is the final one of three.<br /><br />We will be performing major PostgreSQL database engine upgrades in our Aurora clusters across the named data centers.<br /><br />Major database engine upgrades require the Cronofy system to be taken offline, therefore there will be an outage of 45 - 60 minutes in each of the named data centers. During the outage all traffic to our API, Developer Dashboard and Scheduler will be presented with a maintenance response and background processing will be paused.<br /><br />We will upgrade each data center one at a time and will begin the first upgrade shortly after 06:00 UTC on Sunday, 11th June 2023.<br /><br />If you have any questions, please email support@cronofy.com.</p>tag:status.cronofy.com,2005:Incident/174873902023-06-06T11:19:08Z2023-06-06T11:19:08ZDegraded Outlook.com sync performance<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>11:19</var> UTC</small><br><strong>Resolved</strong> - The performance of Outlook.com requests has stayed stable at levels in line with its usual performance for over an hour.<br /><br />At 07:54 UTC we saw a sharp increase in the number of connections failing to Outlook.com, which lasted until 09:30 UTC. This may have caused some changes to take longer to sync. Due to recent issues with Microsoft platforms, we have monitored for an extended period before resolving this issue.<br /><br />We will continue monitoring but we believe the issue has been resolved.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>10:32</var> UTC</small><br><strong>Update</strong> - We're seeing some improvement in Outlook.com requests, but still with enough failures that we consider it to be in a degraded state. We're continuing to monitor the situation.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>08:47</var> UTC</small><br><strong>Monitoring</strong> - We are currently experiencing degraded sync with Outlook.com. This is not having an impact on our overall system performance. We are closely monitoring the situation.</p>tag:status.cronofy.com,2005:Incident/174822242023-06-05T23:02:27Z2023-06-05T23:02:27ZDegraded Microsoft 365 sync performance<p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>23:02</var> UTC</small><br><strong>Resolved</strong> - From 19:22 to 22:27 UTC calendar sync operations to Microsoft platforms were degraded. During this time around 60% of operations between Cronofy and Microsoft resulted in failure.<br /><br />We took steps to mitigate the effects of this on other platforms and began monitoring the issue which has now been resolved.<br /><br />This was a recurrence of an earlier issue, https://status.cronofy.com/incidents/5ftvqz5tp40d.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>21:29</var> UTC</small><br><strong>Update</strong> - We continue to experience degradation with Microsoft 365 sync operations. We're monitoring our systems closely.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>20:29</var> UTC</small><br><strong>Monitoring</strong> - We continue to experience degradation with Microsoft 365 sync operations. We're monitoring our systems closely.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>19:42</var> UTC</small><br><strong>Identified</strong> - We are seeing a recurrence of the earlier issue with Microsoft 365, https://status.cronofy.com/incidents/5ftvqz5tp40d.<br /><br />We are taking the same steps to mitigate the effects of this on other background processing operations.</p>tag:status.cronofy.com,2005:Incident/174793992023-06-05T16:56:22Z2023-06-05T16:56:22ZDegraded Microsoft 365 sync performance<p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>16:56</var> UTC</small><br><strong>Resolved</strong> - From 14:13 to 15:47 UTC calendar sync operations to Microsoft platforms were degraded. During this time around 69% of operations between Cronofy and Microsoft resulted in failure and led to a backlog when processing requests.<br /><br />We took steps to mitigate the effects of this on other platforms and began monitoring the issue which has now been resolved.<br /><br />We believe this is related to Microsoft alert: MO571683 (https://portal.office.com/adminportal/home?#/servicehealth/:/alerts/MO571683)</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>16:13</var> UTC</small><br><strong>Monitoring</strong> - Calendar sync operations to Microsoft services appear to have returned to normal around 16:00 UTC. We are continuing to monitor.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>15:48</var> UTC</small><br><strong>Update</strong> - Our telemetry is showing that calender sync operations to Outlook.com and Microsoft 365 are also affected.<br /><br />We have mitigated the performance impact this issue was having on calendar sync operations with other providers.<br /><br />We are tracking open Microsoft incidents, with IDs MO571683 and EX571516, and continuing to monitor our platform closely.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>14:53</var> UTC</small><br><strong>Identified</strong> - We have identified an issue with calendar sync operations when connecting to the Microsoft Graph API.<br /><br />We are working to mitigate this impact on the rest of our background processing jobs.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>14:31</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating degraded performance with background processing in both our DE and US data centers.</p>tag:status.cronofy.com,2005:Incident/173313272023-06-04T09:24:07Z2023-06-04T09:24:07ZMajor database upgrade in our AU, CA and UK data centers<p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>09:24</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been successfully completed. Our application was unavailable in each data center during the following times (UTC):<br />AU: 06:03 - 06:58<br />UK: 07:17 - 08:01<br />CA: 08:19 - 09:03<br /><br />If you have any questions, please email support@cronofy.com.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>09:11</var> UTC</small><br><strong>Verifying</strong> - The maintenance in our Canada data center is complete. Our application was unavailable from 08:19 UTC to 09:03 UTC.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>08:12</var> UTC</small><br><strong>Update</strong> - The maintenance in our UK data center is complete. Our application was unavailable from 07:17 UTC to 08:01 UTC.<br /><br />We will now start the maintenance in our Canada data center.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>07:05</var> UTC</small><br><strong>Update</strong> - The maintenance in our Australia data center is complete. We will be starting the maintenance in our UK data center momentarily.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>06:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will be upgrading our data center in Australia first.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>16:47</var> UTC</small><br><strong>Scheduled</strong> - This maintenance window is the second of three.<br /><br />We will be performing major PostgreSQL database engine upgrades in our Aurora clusters across the named data centers.<br /><br />Major database engine upgrades require the Cronofy system to be taken offline, therefore there will be an outage of 45 - 60 minutes in each of the named data centers. During the outage all traffic to our API, Developer Dashboard and Scheduler will be presented with a maintenance response and background processing will be paused.<br /><br />We will upgrade each data center one at a time and will begin the first upgrade shortly after 06:00 UTC on Sunday, 4th June 2023.<br /><br />If you have any questions, please email support@cronofy.com.</p>tag:status.cronofy.com,2005:Incident/173313252023-05-28T07:39:50Z2023-05-28T07:39:50ZMinor database upgrade in all data centers<p><small>May <var data-var='date'>28</var>, <var data-var='time'>07:39</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been successfully completed.<br /><br />We observed disruption in each data center at the following UTC times:<br />SG 06:41:34 - 06:44:49<br />AU 06:51:30 - 06:52:15<br />UK 06:59:06 - 07:00:11<br />CA 07:09:22 - 07:09:23<br />DE 07:16:19 - 07:19:21<br />US 07:20:48 - 07:21:39<br /><br />Any background processing jobs will have been automatically retried.</p><p><small>May <var data-var='date'>28</var>, <var data-var='time'>06:30</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>16:47</var> UTC</small><br><strong>Scheduled</strong> - This maintenance window is the first of three that will happen over the coming weeks. This first step is required in preparation for upcoming major database upgrades.<br /><br />We will be performing minor PostgreSQL database engine upgrades in our Aurora clusters across all data centers.<br /><br />Minor database engine upgrades require a reboot of the database server, therefore there will be up to two minutes of disruption when this happens. We will upgrade each data center one at a time and will begin the first upgrade shortly after 06:30 UTC on Sunday, 28th May 2023.<br /><br />If you have any questions, please email support@cronofy.com.</p>tag:status.cronofy.com,2005:Incident/172158092023-05-11T08:37:01Z2023-05-11T08:37:01ZDegraded Apple calendar sync<p><small>May <var data-var='date'>11</var>, <var data-var='time'>08:37</var> UTC</small><br><strong>Resolved</strong> - Apple calendar synchronization has returned to normal operation and credentials have been reinstated where the invalidation was related to the incident.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>08:24</var> UTC</small><br><strong>Update</strong> - Apple's calendar servers appear to have started responding normally since 08:15 UTC.<br /><br />We will reinstate credentials where possible in the next 20 minutes so long as things continue to appear normal.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>08:12</var> UTC</small><br><strong>Update</strong> - Some Apple credentials have been invalidated during this incident before we stepped in to prevent that process from happening.<br /><br />Once Apple's calendar servers are responding normally we will reinstate the credentials invalidated during the window of this incident where possible.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>07:46</var> UTC</small><br><strong>Monitoring</strong> - We have seen an elevated number of errors when attempting to synchronize Apple calendars across all data centers since around 07:25 UTC.</p>tag:status.cronofy.com,2005:Incident/171176822023-05-07T08:17:54Z2023-05-07T08:17:54ZElasticache service updates in our CA and UK data centers<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>08:17</var> UTC</small><br><strong>Completed</strong> - The maintenance has been successfully completed.<br /><br />In our UK data center we observed errors between 07:25:45 and 07:25:53 UTC. In our Canada data center we observed errors between 07:59:05 and 07:59:40 UTC.<br /><br />Any background processing jobs will have been automatically retried.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>In progress</strong> - This scheduled maintenance is in progress</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>Scheduled</strong> - We will be performing ElastiCache service updates in our Canada and UK data centers.<br /><br />ElastiCache service updates require a failover, therefore there will be up to a minute of disruption when this happens. We will perform the upgrades one at a time, starting with the UK data center before moving on to the Canada data center. The first upgrade will begin shortly after 07:00 UTC on Sunday, 7th May 2023.<br /><br />The disruption will result in errors being returned by our application when using the scheduler or our API. Any background processing jobs will automatically be retried.<br /><br />If you have any questions, please email support@cronofy.com.</p>tag:status.cronofy.com,2005:Incident/169557512023-04-30T20:20:15Z2023-04-30T20:20:15ZElasticache service updates in our AU, CA, SG and UK data centers<p><small>Apr <var data-var='date'>30</var>, <var data-var='time'>20:20</var> UTC</small><br><strong>Completed</strong> - Firstly, we need to apologise for the lack of communication around this scheduled maintenance. This is not the standard we set ourselves, we should have done better, and for that we are very sorry.<br /><br />The maintenance this morning was started outside of the advertised window. Updates in our Australia and Singapore data centers were triggered shortly after 10:00 UTC. There were two brief moments of elevated error rates in those environments. In Australia they took place between 10:20:39 - 10:22:28 and 10:24:02 - 10:24:37. In Singapore the they took place at 10:14:35, and between 10:17:39 - 10:18:11.<br /><br />Because the elevated errors rates happened outside of the maintenance window, we decided to not go ahead with the upgrades in our Canada and UK data centers. We intend to perform this maintenance in those environments on Sunday, 7th of May.<br /><br />Once again, we sincerely apologise for not communicating what happened with this maintenance sooner.<br /><br />If you have any questions, please email support@cronofy.com.</p><p><small>Apr <var data-var='date'>24</var>, <var data-var='time'>20:12</var> UTC</small><br><strong>Scheduled</strong> - We will be performing ElastiCache service updates across the named data centers. The DE and US data centers are excluded as this service update has already been automatically applied in those environments. We normally do not let these sorts of updates automatically apply, so for the remaining data centers we want to have more control over when the update is applied.<br /><br />ElastiCache service updates require a failover, therefore there will be up to a minute of disruption when this happens. We will upgrade the data centers two at a time, starting with AU and SG, and finishing with CA and UK. The first upgrades will begin shortly after 06:30 UTC on Sunday, 30th April 2023.<br /><br />The disruption will result in errors being returned by our application when using the scheduler or our API. Any background processing jobs will automatically be retried.<br /><br />If you have any questions, please email support@cronofy.com.</p>tag:status.cronofy.com,2005:Incident/168319722023-04-12T03:00:00Z2023-04-13T15:33:51ZUS API degradation<p><small>Apr <var data-var='date'>12</var>, <var data-var='time'>03:00</var> UTC</small><br><strong>Resolved</strong> - Our US data center experienced a failover of the primary database at around 03:15 UTC. From this point around 20% of API requests encountered an error resulting in a 500 response, for specific operations such as creating events the failure rate was 80%.<br /><br />This lasted until around 05:30 UTC (around 2h15m later) when the elevate errors were recognized and all processes restarted. Errors then returned to normal levels a few minutes later.<br /><br />From an initial investigation we believe "write" connections were attempting to write to a node which had become a replica as part of the failover, which then failed. Some processes restarted automatically, but API processes appear to have not done so, nor did in the face of intermittent errors. That issue was cleared when we made all the processes restart and service then returned to normal.<br /><br />We will be conducting a full postmortem of this event and will post that against this incident by the end of the week.</p>tag:status.cronofy.com,2005:Incident/167112302023-03-31T16:30:00Z2023-03-31T16:54:20ZElevated 500 errors on Read Event<p><small>Mar <var data-var='date'>31</var>, <var data-var='time'>16:30</var> UTC</small><br><strong>Resolved</strong> - From 16:30 UTC to 16:36 UTC, calls to the Read Events API encountered a higher level of 500 Server Error responses than usual. This was due to a bug released at this time which allowed a small number of events to cause an error while being rendered. This was immediately spotted and rolled back, which was completed at 16:36 UTC.</p>tag:status.cronofy.com,2005:Incident/165428922023-03-26T09:14:49Z2023-03-26T09:14:49ZDatabase engine upgrades in our AU, CA, DE, SG and UK data centers<p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>09:14</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been successfully completed.<br /><br />There was a maximum of 1 minute and 45 seconds of elevated errors as the upgrade was applied in each data center. Our telemetry shows that elevated errors were observed during the following time periods, all times are in UTC:<br />AU: 07:36:00 - 07:36:10<br />CA: 07:58:05 - 07:58:15<br />DE: 08:15:35 - 08:16:40<br />SG: No errors were observed<br />UK: 08:54:20 - 08:56:05<br /><br />Background processing jobs will have been queued and automatically retried.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>07:30</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>10:51</var> UTC</small><br><strong>Scheduled</strong> - We will be performing PostgreSQL database engine upgrades in our Aurora clusters across the named data centers. The US is excluded as we performed the upgrade in that environment as part of the last scheduled maintenance.<br /><br />Database engine upgrades require a reboot of the database server, therefore there will be up to a minute of disruption when this happens. We will upgrade each data center one at a time and will begin the first upgrade shortly after 07:30 UTC on Sunday, 26th March 2023.<br /><br />If you have any questions, please email support@cronofy.com.</p>tag:status.cronofy.com,2005:Incident/162487062023-02-26T09:37:58Z2023-02-26T09:37:58ZDatabase Instance Change in our US Data Center<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>09:37</var> UTC</small><br><strong>Completed</strong> - When provisioning the new nodes shortly after 08:00 UTC we found that AWS refused to provision the new instances as they were incompatible with the minor version of PostgreSQL our US database cluster was running. This was not flagged when running the Terraform plan prepared for this change in instance type.<br /><br />After discussion, we decided to apply the necessary minor version upgrade to the database cluster first, before proceeding with the planned maintenance.<br /><br />The minor version upgrade resulted in two periods of elevated errors, ~30 seconds from 08:46:50 UTC as a failover happened, and ~10 seconds as one of the nodes restarted from 08:48:30 UTC.<br /><br />This then allowed us to proceed with adding the new nodes which completed at around 09:17 UTC.<br /><br />Removing the old nodes included the planned failover and resulted in a third period of elevated errors lasting ~4 minutes from 09:24 UTC.<br /><br />We then monitored for a while before marking this maintenance work as complete.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>08:42</var> UTC</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>08:37</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is in progress in our US data center.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>12:30</var> UTC</small><br><strong>Scheduled</strong> - We will be changing the instance types used in our PostgreSQL cluster in our US data center. This will require a failover when we switch to the new instances, therefore there will be up to two minutes of disruption when the failover takes place. This will happen shortly after 09:00 UTC on Sunday, 26th February 2023.<br /><br />As this is happening at short notice, this work will count against our SLA. We believe there is enough value in doing this proactively to help reduce risks to system stability.<br /><br />If you have any questions, please email support@cronofy.com.</p>tag:status.cronofy.com,2005:Incident/161377242023-02-14T17:40:23Z2023-02-14T17:40:23ZApple sync degraded<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>17:40</var> UTC</small><br><strong>Resolved</strong> - At 15:58 UTC, communication between Cronofy and Apple began to timeout for a proportion of our requests. This quickly worsened, causing Apple calendar syncs to take several attempts.<br /> <br />A change was deployed to slow down the rate of communication with Apple to reduce the pressure on their API and to avoid service being impacted for other calendar providers.<br /><br />At 16:50 UTC, we started to see most communication succeed again and monitored the situation until it had remained stable for 30 minutes. <br />At 17:20 UTC, we removed the rate-limiting and continued to monitor the situation.<br /><br />Normal service has continued, so we are marking this incident as resolved.<br />If you have any queries, please contact us at support@cronofy.com</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>16:52</var> UTC</small><br><strong>Monitoring</strong> - Apple API traffic has recovered, and normal operation has resumed.<br />We are continuing to monitor the service.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>16:37</var> UTC</small><br><strong>Identified</strong> - We are seeing an increase in connection errors to Apple's API globally. Apple sync is still operating in a degraded state.<br />We will update at 17:00 UTC.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>16:14</var> UTC</small><br><strong>Investigating</strong> - We are seeing increased errors from Apple's calendar API since 16:00 UTC; we are investigating and taking mitigating action.</p>tag:status.cronofy.com,2005:Incident/161199512023-02-13T10:00:00Z2023-02-13T10:59:27ZElevated API 500 errors<p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>10:00</var> UTC</small><br><strong>Resolved</strong> - From 09:59 UTC to 10:07 UTC, calls to our API encountered a higher level of 500 Server Error responses than usual. This was due to a bug released at this time which was not caught by our automated tests or during code review.<br />Once we were notified of the issue with the release, we initiated a rollback, which took about 7 minutes to roll out.<br />We will be improving our test suite and adding better error handling in this area.</p>tag:status.cronofy.com,2005:Incident/159284822023-01-25T10:18:37Z2023-01-25T10:18:37Z365 synchronization errors<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>10:18</var> UTC</small><br><strong>Resolved</strong> - The performance of syncing Microsoft 365-backed calendars was degraded for about 90 minutes.<br /><br />Microsoft 365 began returning an elevated number of errors and timing out just after 07:05 UTC until 08:30 UTC. This was part of a wider issue within Azure, where Microsoft 365 is hosted.<br /><br />Steps were taken to increase capacity and be more aggressive in timing out connections to Microsoft 365 to mitigate the impact on background processing. The sudden change to the volume of background jobs did lead to a slight delay in processing being present for much of the issue.<br /><br />Normal service resumed a few minutes after the Microsoft 365 APIs began to operate normally, once a surge of calendar update notifications was processed.<br /><br />We will be reviewing our mechanisms for reducing the side effects of such incidents in the future.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>08:44</var> UTC</small><br><strong>Update</strong> - Error rates for Microsoft 365-backed calendars have returned to normal levels, and we have processed a surge in calendar updates.<br /><br />We will continue to monitor the service but believe this issue to now be resolved.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>Update</strong> - Microsoft have sent a 365 health notification (MO502273) relating to some users being unable to access multiple Microsoft 365 services. This is likely related to the issue we are seeing.<br /><br />Errors for 365-backed calendar continue to be elevated.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>07:35</var> UTC</small><br><strong>Monitoring</strong> - We are seeing a higher than usual number of errors affecting calendars hosted on Microsoft 365 across all our data centers for connections using Exchange Web Services (EWS) and Microsoft's Graph API.<br /><br />Background processing has scaled up to compensate but may be degraded as a result.</p>