When provisioning the new nodes shortly after 08:00 UTC we found that AWS refused to provision the new instances as they were incompatible with the minor version of PostgreSQL our US database cluster was running. This was not flagged when running the Terraform plan prepared for this change in instance type.
After discussion, we decided to apply the necessary minor version upgrade to the database cluster first, before proceeding with the planned maintenance.
The minor version upgrade resulted in two periods of elevated errors, ~30 seconds from 08:46:50 UTC as a failover happened, and ~10 seconds as one of the nodes restarted from 08:48:30 UTC.
This then allowed us to proceed with adding the new nodes which completed at around 09:17 UTC.
Removing the old nodes included the planned failover and resulted in a third period of elevated errors lasting ~4 minutes from 09:24 UTC.
We then monitored for a while before marking this maintenance work as complete.
Posted Feb 26, 2023 - 09:37 UTC
Update
Scheduled maintenance is still in progress. We will provide updates as necessary.
Posted Feb 26, 2023 - 08:42 UTC
In progress
Scheduled maintenance is in progress in our US data center.
Posted Feb 26, 2023 - 08:37 UTC
Scheduled
We will be changing the instance types used in our PostgreSQL cluster in our US data center. This will require a failover when we switch to the new instances, therefore there will be up to two minutes of disruption when the failover takes place. This will happen shortly after 09:00 UTC on Sunday, 26th February 2023.
As this is happening at short notice, this work will count against our SLA. We believe there is enough value in doing this proactively to help reduce risks to system stability.