CA data center - Interruption of service

Summary

On October 23rd, between 10:11 and 10:18 UTC, we experienced a partial outage in our Canadian data center.

During this time customers may have received server errors or experienced increased response latency when using the scheduler or our API. Background processing was not impacted during this incident.

At 10:09 UTC, we released a change that increased how much work each individual web server process can handle at a time. This inadvertently caused those processes to hit resource limits after a short period of time, which then resulted in those processes being restarted. This then kept repeating until we reverted the change.

The first errors happened at 10:11 UTC, and our engineers were alerted to this at 10:14. At 10:17 we reverted the change, and by 10:19 UTC the server errors had stopped and response latency had returned to normal.

Postmortem

We always ask the questions:

• Could it have been resolved sooner

  • No, reverting the change happened very quickly after we were alerted to the errors. The errors stopped as soon as our deployment processed had rolled back

• Could it have been identified sooner

  • No, alerts came through within minutes of the initial change being deployed. Our external monitoring tools also check the availability of our API and scheduler every minute.

• Could it have been prevented

  • Yes

    • This was not tested in our test environment before deploying. Replicating production traffic loads in difficult, so had we deployed this in our test environment we may not have encountered the exact same issue. However, there may have been some signs of increase resource utilization.

    • Rather than increasing how much work each individual web server process can handle, we could have increased the number of processes running. This is known as scaling horizontally, rather than scaling vertically.

      • There was also an oversight on how resource utilization would be affected by this change. From this we have now learnt more about the nuance involved with making these specific changes.

Actions

• Make it harder to make this same mistake again by adding comments to the codebase to ensure if a similar change is made in the future that we also appropriately increase resource limits

• We felt that we could have been quicker to communicate about the issue on our status page. To help with this in the future we will:

  • Update our Incident Playbook to ensure guidelines on communication timelines are clear to all
  • Give more engineers access to post and update incident on our status page