Cimpress Technology

Asset Management

  1. Przegląd
  2. Artwork Technology
  3. Asset Management
  4. Problem

Spike in studio errors when processing uploads

Rozwiązany

Incident Resolved: CTIM-992
Scaling the AI service fixed the immediate problem, and the service will remain scaled throughout Cyber Monday. The increase in load was caused by the expanded rollout of smart recommendations to additional locales and teams are co-ordinating on long-term changes to make the overall system less sensitive to spikes in the future.

The ‘Unknown Errors’ are normal, and we’re looking at options to reduce these, but given that they’ve always been around, the only correlation found here was increased volume leading to higher error counts.

this issue is being flagged as contained, but teams continue to monitor.
Please contact CTIM or see ticket for more details.

Zaktualizowana wersja

UPDATE:

Teams continue to investigate the spikes - The primary issue is resolved, with a likely root cause identified (a slowdown in Classifications and sharpened jobs which caused a backlog which started to affect studio - Systems have been scaled to help proactively deal with this load). Teams continue to investigate to determine if there are any other related issues that may have spun off because of this.

Rozpoznanie

New Incident: CTIM-992
Priority: High
Escalation sent to: Artwork Tech: Support for review.
we are seeing a spike in errors in Studio when processing uploads. here is an example query https://api.sherbert.cimpress.io/v3/assets/pyL0-RjE2xS60ObA–Ai8~200?signature=b32cf54065559d736e210556c7a26031419951f4&showDeleted=true. Sherbert team is already working on it.