Artwork Processing

Seeing few 500 error from the sharpened variant in https://api.sherbert.cimpress.io

Post-mortem
The technical teams have completed their root cause and corrective action analysis. The post-mortem can be viewed [here https://cimpress-support.atlassian.net/wiki/spaces/DCR/pages/15709241345/Sherbert+outage+due+to+swamping+AKeyless#2023-10-04].
Løst

It has been an hour since this incident was contained, and we have not seen a recurrence. We are marking this incident as resolved. Within five business days, we will post a link the the post mortem of this incident here.

Overvågning

AKeyless errors have subsided and the team is bringing the infrastructure up to normal operation levels.

Problemer identificeret

The Sherbert team proceeded with further scaling down to ease the load on Akeyless.

Problemer identificeret

The Akeyless and Sherbert teams are currently addressing this issue, and we are continuously monitoring both traffic and error rates.

Problemer identificeret

It seems that error rates for uploads have returned to their usual or standard levels. The technical team is still working on fixing the issue.

Problemer identificeret

Akeyless that traffic levels have decreased. The Gateway is up and working through its backlog.

Problemer identificeret

The sherbert’s team is actively working with akeyless support team to resolve this issue.

Undersøgelse

The team is experiencing 429s and long response times even from our gateway instance and reached out to akeyless support team.

Undersøgelse

Sherbert is experiencing issues. As per the initial investigation, it seems sharpened variant generation failed. The technical team is investigating. We will update this status page again as soon as we have more information or in 30 minutes, whichever is sooner.