Firstly, we need to apologise for the lack of communication around this scheduled maintenance. This is not the standard we set ourselves, we should have done better, and for that we are very sorry.
The maintenance this morning was started outside of the advertised window. Updates in our Australia and Singapore data centers were triggered shortly after 10:00 UTC. There were two brief moments of elevated error rates in those environments. In Australia they took place between 10:20:39 - 10:22:28 and 10:24:02 - 10:24:37. In Singapore the they took place at 10:14:35, and between 10:17:39 - 10:18:11.
Because the elevated errors rates happened outside of the maintenance window, we decided to not go ahead with the upgrades in our Canada and UK data centers. We intend to perform this maintenance in those environments on Sunday, 7th of May.
Once again, we sincerely apologise for not communicating what happened with this maintenance sooner.
We will be performing ElastiCache service updates across the named data centers. The DE and US data centers are excluded as this service update has already been automatically applied in those environments. We normally do not let these sorts of updates automatically apply, so for the remaining data centers we want to have more control over when the update is applied.
ElastiCache service updates require a failover, therefore there will be up to a minute of disruption when this happens. We will upgrade the data centers two at a time, starting with AU and SG, and finishing with CA and UK. The first upgrades will begin shortly after 06:30 UTC on Sunday, 30th April 2023.
The disruption will result in errors being returned by our application when using the scheduler or our API. Any background processing jobs will automatically be retried.