Artwork Processing
Users seeing Alert Box while designing in Studio
Incident Resolved: CTIM-1135
Sherbert team recently deployed changes to production to resolve the ‘sharpened/metadata’ issue reported yesterday. It is now clear that a recent change disabling auto-save process for Marvel products brought up such a noticeable spike in these issues. The team is now in monitoring mode as this deployment is expected to resolve these incidents, but it’s success will be seen in metrics over time. CTIM will tentatively Contain this incident with the possibility of re-opening if that becomes necessary.
Please contact CTIM or see ticket for more details.
UPDATE:
At this time, Sherbert team has identified one incident going on since January 29 affecting sharpened images in Studio. The team has determined a root cause and action for this and will roll this out by the start of US business hours tomorrow. Next update when the team is back online tomorrow.
UPDATE:
Teams continue to investigate the issues reported in this ticket.
- At present the error seems to manifest: ‘precisely at the moment when the image freezes in the sharpening state, and at this time the automatic saving of the document is triggered, and it is at this moment that the user sees this error.’ - The Studio team is attempting to add interrupts for requests that take a long time to process.
Sherbert Squad has:
1) Identified the cause on the sharpened/metadata call and have a fix they plan to deploy before the start of US hours tomorrow.
2) Spoke with the Artwork automation squad about the PDF errors. This appears to be an issue regarding the APDFL endpoint. Calling patterns have recently increased overwhelming their infrastructure eventually resulting in the timeout experienced. The infrastructure team has been added to the investigation to dig deeper into this aspect of the problem.
The next update will be in ~1 hour or as soon as we have more updates to share.
UPDATE:
Current response from Vista is that the ‘Sharpen image’ issue appears to be the highest priority with the highest error rate. Sherbert and other teams will prioritize this while Vista team is in channel and can always respond with more impact information if found. Any new updates will be shared right away with a check in cadence currently at 1 hour mark until teams say otherwise and sign off for the night (this is business hours, highest priority).
UPDATE:
Sherbert team, along with other Artwork teams are actively investigating, and currently understand there to be two separate issues. One affecting PDF Files and one affecting image sharpening. Current request is to Vista to help understand which of these two issues is highest priority to set priorities more clearly for investigating teams. Next update with Vista response or there’s more information on root cause.
New Incident: CTIM-1135
Priority: High
Escalation sent to: Artwork Tech: Support for review.
It was reported in Vista IRT channel at 2am ET today that users are seeing an Alert Box Error while designing in Studio. This started on January 29 and has until now impacted 4k users (199% more than historical range). Vista’s DJ Squad has narrowed this down to potentially an issue with Sherbert Service. The CT Artwork and Infrastructure team is already aware of this issue and investigating actively. This is currently prioritized as ‘High’ priority meaning it is the top priority for teams during their business hours. Next Update in 1 hour max.