Artwork Processing
Seeing increased responses times in sherbert
It has been an hour since this incident was contained, and we have not seen a recurrence. We are marking this incident as resolved. Within five business days, we will post a link the the post mortem of this incident here.
The root cause is currently under in-depth investigation. It seems to be associated with scaling DNS pods within EKS from a cold start. In this regard, EU-West-1 exhibited a smooth scaling process, whereas US-East-1 encountered difficulties with a steeper ascent. This discrepancy explains why everything functioned correctly in EU-West-1, while US-East-1 experienced over an hour of extended response times. The team is actively working to ascertain why the scaling process wasn’t smooth, which appears to be linked to a bug in the latest version of Kubernetes.
The issue seems to have subdued and we are seeing normal response rate in Sherbert.
Team is actively working on it to find the root cause.
Seeing increased responses times in sherbert v2 us-east-1 region. Investigation in progress.
“HTTP request timed out to an endpoint on host api.sherbert.cimpress.io” in NA