Artwork Processing

Seeing few 500 error from the sharpened variant in https://api.sherbert.cimpress.io

Retour sur l'événement
The technical teams have completed their root cause and corrective action analysis. The post-mortem can be viewed [here https://cimpress-support.atlassian.net/wiki/spaces/DCR/pages/15709241345/Sherbert+outage+due+to+swamping+AKeyless#2023-10-04].
Résolu

It has been an hour since this incident was contained, and we have not seen a recurrence. We are marking this incident as resolved. Within five business days, we will post a link the the post mortem of this incident here.

Sous surveillance

AKeyless errors have subsided and the team is bringing the infrastructure up to normal operation levels.

Problème identifié

The Sherbert team proceeded with further scaling down to ease the load on Akeyless.

Problème identifié

The Akeyless and Sherbert teams are currently addressing this issue, and we are continuously monitoring both traffic and error rates.

Problème identifié

It seems that error rates for uploads have returned to their usual or standard levels. The technical team is still working on fixing the issue.

Problème identifié

Akeyless that traffic levels have decreased. The Gateway is up and working through its backlog.

Problème identifié

The sherbert’s team is actively working with akeyless support team to resolve this issue.

En voie de résolution

The team is experiencing 429s and long response times even from our gateway instance and reached out to akeyless support team.

En voie de résolution

Sherbert is experiencing issues. As per the initial investigation, it seems sharpened variant generation failed. The technical team is investigating. We will update this status page again as soon as we have more information or in 30 minutes, whichever is sooner.