All Systems Operational
Uptime over the past 90 days. View historical uptime.
Scheduler Operational
90 days ago
100.0 % uptime
Today
API Operational
90 days ago
100.0 % uptime
Today
Background Processing Operational
90 days ago
99.97 % uptime
Today
Developer Dashboard Operational
90 days ago
100.0 % uptime
Today
Major Calendar Providers Operational
Apple Operational
Google Operational
Microsoft 365 Operational
Outlook.com Operational
Conferencing Services Operational
GoTo Operational
Zoom Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Past Incidents
Apr 26, 2024

No incidents reported today.

Apr 25, 2024

No incidents reported.

Apr 24, 2024

No incidents reported.

Apr 23, 2024

No incidents reported.

Apr 22, 2024
Background processing degraded
Postmortem - Read details
Apr 23, 17:32 UTC
Resolved - Background task processing has remained normal and the incident is resolved.

Around 11:00 UTC, a change was deployed to production which inadvertently broke the background processing.

We received our first alert at 11:12 UTC. We reverted the offending change at 11:15 UTC, but this did not restore background processing completely.

Multiple changes were made to minimize the impact of the issue, which restored most functionality for newly created jobs by 11:54 UTC. This left a backlog of work queued for processing between 11:00-11:54 UTC stuck in an unprocessable state.

The stuck backlog work was due to requeue at around 13:00 UTC due to reaching a configured timeout of 2 hours. We attempted to find a way to process this work sooner but were unsuccessful. As anticipated, the work became available from around 13:00 UTC and was processed for about half an hour, completing by 13:30 UTC.

This fully restored normal operations.

A postmortem of the incident will take place and be attached to this incident in the next 48 hours. If you have any queries in the interim, please contact us at support@cronofy.com.
Apr 22, 13:54 UTC
Monitoring - The backlog of jobs has been processed and we are monitoring to verify that everything remains normal.
Apr 22, 13:29 UTC
Update - New work continues to process as normal, and the stuck jobs are expected to be processed in the next hour.
Apr 22, 12:57 UTC
Identified - Capacity has been increased but the stuck jobs are still not processing due to an issue with the queue. We are working to find a way to process these jobs. New jobs are avoiding the stuck queue and are processing as expected.
Apr 22, 12:32 UTC
Update - We have mitigated most of the impact for background processing but have jobs stuck in US & DE. We are working to further increase capacity and execute these jobs.
Apr 22, 12:04 UTC
Update - Processing has recovered in all data centers apart from US & DE, which have one partly recovered. We are working to mitigate the issue in these two data centers.
Apr 22, 11:41 UTC
Investigating - We have seen background processing degrade in all data centers after a recent deployment. We have reverted the change and are investigating the cause.
Apr 22, 11:23 UTC
Apr 21, 2024

No incidents reported.

Apr 20, 2024

No incidents reported.

Apr 19, 2024

No incidents reported.

Apr 18, 2024

No incidents reported.

Apr 17, 2024

No incidents reported.

Apr 16, 2024

No incidents reported.

Apr 15, 2024

No incidents reported.

Apr 14, 2024

No incidents reported.

Apr 13, 2024

No incidents reported.

Apr 12, 2024

No incidents reported.

← Incident History Powered by Atlassian Statuspage