Between approximately 02:25 UTC on July 1st and 10:11 UTC on July 7th 2026, scheduled notifications in our US data center failed to send. The affected notifications were queued but never picked up, and were flushed once the underlying cause was cleared.
The impact was limited to customers in the US data center using notification-driven API features. Approximately 9,000 Event Trigger notifications for calendar events starting and ending were delivered late. A small number of Scheduler Workflows lifecycle notifications were also delivered late, and a small number of scheduled Meeting Agents did not arrive to their meetings.
The root cause was a regression introduced on April 22nd, in an internal service responsible for sending scheduled notifications. The service uses leader election so that a single process handles distributing the imminent notifications at any one time, coordinated through a lock. The regression made that configuration fragile. If a leader did not release the lock cleanly, no other process would override it. When the leader stopped in the early hours of July 1st, it left the lock held, and with no process able to take over, notifications quietly backed up from that point.
Nothing alerted us to this. The failure mode was an absence of activity rather than an error, and we had no alerting on notification throughput that would have caught the drop-off.
During the incident, we manually cleared the stuck lock, which allowed the service to resume and flush the backlog within four minutes.
All times are in UTC.
We always ask the same three questions.
Once we were looking at the right thing, resolution was fast. The probable cause was raised within eight minutes of opening the channel, and the backlog was flushed four minutes after that. We are satisfied with how the resolution itself went.
This is where we fell short, and badly. The failure was live for almost six days. Two things let it run that long. First, we had no alerting on notification throughput, so a complete stop in sending produced no signal. It is harder to alert on things not happening, but that is exactly the case this warranted. Second, the first support ticket arrived on July 2nd and did not surface the pattern. It took a second ticket, four days later, for the connection to be made. Six days is too long for a customer-facing feature to be silently failing.
The regression was introduced in April and sat latent until a leader failed to shut down cleanly for the first time. The fragility was in the lock's release path, the case where a leader stops without releasing cleanly, which is precisely the kind of edge that is easy to miss in review and under-covered by tests. We did not catch it at either stage.
We have added alerting on notification throughput, so that a drop-off in sends is positively and quickly identifiable rather than depending on customer reports.
We have fixed the code so that a lock held by a stopped leader is reliably released and picked up by another process, rather than leaving the work stuck.
We have added tests around the configuration to ensure we cannot fall foul of the misconfiguration by accident, and integration tests to show that an orphaned lock can be obtained by another process.
We are reviewing how we handle support tickets that describe an absence of expected behavior, so that a single ticket of this kind prompts a check of the relevant telemetry sooner.