Between 20:45:15 and 20:47:30, and 21:03:15 and 21:03:30 UTC, we observed increased API error rates in our US data center. The first window of these was caused by a database failover, the second of these was caused by a reboot of the database cluster reader node.
We took this action in an attempt to alleviate a small but persistent number of database errors that we had seen since the earlier maintenance. During the maintenance, we believed these errors were transient and that they would dissipate throughout the day. As this did not happen, we felt it was necessary to take action now, and that action was to failover, reboot, and then ultimately replace the affected node. Since the replacement, we have not observed the aforementioned database errors.
We're continuing to monitor the situation. An additional spike in server errors was observed between 21:03:15 and 21:03:30
Posted Nov 30, 2025 - 21:11 UTC
Monitoring
We have chosen to initiate a database failover in our US data center.
Because of this, customers may have received a 500 Internal Server error between 20:45:15 and 20:47:30. The vast majority of these errors were observed between 20:45:15 and 20:46:00.