Seeing few 500 error from the sharpened variant in https://api.sherbert.cimpress.io

Postmortem Nov 6, 2023 1:59 AM EST

The technical teams have completed their root cause and corrective action analysis. The post-mortem can be viewed [here https://cimpress-support.atlassian.net/wiki/spaces/DCR/pages/15709241345/Sherbert+outage+due+to+swamping+AKeyless#2023-10-04].

Resolved Oct 4, 2023 6:06 PM EDT

It has been an hour since this incident was contained, and we have not seen a recurrence. We are marking this incident as resolved. Within five business days, we will post a link the the post mortem of this incident here.

Monitoring Oct 4, 2023 6:02 PM EDT

AKeyless errors have subsided and the team is bringing the infrastructure up to normal operation levels.

Problem Identified Oct 4, 2023 4:13 PM EDT

The Sherbert team proceeded with further scaling down to ease the load on Akeyless.

Problem Identified Oct 4, 2023 3:39 PM EDT

The Akeyless and Sherbert teams are currently addressing this issue, and we are continuously monitoring both traffic and error rates.

Problem Identified Oct 4, 2023 2:35 PM EDT

It seems that error rates for uploads have returned to their usual or standard levels. The technical team is still working on fixing the issue.

Problem Identified Oct 4, 2023 1:43 PM EDT

Akeyless that traffic levels have decreased. The Gateway is up and working through its backlog.

Problem Identified Oct 4, 2023 1:25 PM EDT

The sherbert’s team is actively working with akeyless support team to resolve this issue.

Investigating Oct 4, 2023 1:09 PM EDT

The team is experiencing 429s and long response times even from our gateway instance and reached out to akeyless support team.

Investigating Oct 4, 2023 12:46 PM EDT

Sherbert is experiencing issues. As per the initial investigation, it seems sharpened variant generation failed. The technical team is investigating. We will update this status page again as soon as we have more information or in 30 minutes, whichever is sooner.