Artwork Processing

Seeing few 500 error from the sharpened variant in https://api.sherbert.cimpress.io

Gewonnene Erkenntnisse
The technical teams have completed their root cause and corrective action analysis. The post-mortem can be viewed [here https://cimpress-support.atlassian.net/wiki/spaces/DCR/pages/15709241345/Sherbert+outage+due+to+swamping+AKeyless#2023-10-04].
Gelöst

It has been an hour since this incident was contained, and we have not seen a recurrence. We are marking this incident as resolved. Within five business days, we will post a link the the post mortem of this incident here.

Beobachtung

AKeyless errors have subsided and the team is bringing the infrastructure up to normal operation levels.

Problem identifiziert

The Sherbert team proceeded with further scaling down to ease the load on Akeyless.

Problem identifiziert

The Akeyless and Sherbert teams are currently addressing this issue, and we are continuously monitoring both traffic and error rates.

Problem identifiziert

It seems that error rates for uploads have returned to their usual or standard levels. The technical team is still working on fixing the issue.

Problem identifiziert

Akeyless that traffic levels have decreased. The Gateway is up and working through its backlog.

Problem identifiziert

The sherbert’s team is actively working with akeyless support team to resolve this issue.

Untersuchung

The team is experiencing 429s and long response times even from our gateway instance and reached out to akeyless support team.

Untersuchung

Sherbert is experiencing issues. As per the initial investigation, it seems sharpened variant generation failed. The technical team is investigating. We will update this status page again as soon as we have more information or in 30 minutes, whichever is sooner.