Prepress
Prepress PDF Processing Service Degradation Due to Excessive Duplicate Requests
UPDATE:
While Platform Support was able to reinject four Vista orders, there are still some orders that are facing a Cockpit error and will need remediation by Commerce teams tomorrow. Next update during IST business hours tomorrow.
UPDATE:
Platform Support online and already looking to reinject the affected orders. Next update when reinjection is complete.
UPDATE:
Remediation Underway. Teams have identified the orders that need reinjecting. Request to Platform Support to take this step during business hours. Next update during IST business hours when Platform Support is actively taking on this step.
Incident Resolved: CTIM-2421
Incident Contained. Teams are still aligning on any further remediation actions. Next update in max 2 hours on progress of any remediation.
Please contact CTIM or see ticket for more details.
UPDATE:
While this incident is Contained, with the potential of having affected Prepress PDF processing in any business using the EU-West-1 region CTIM is opening the channel to any business so impact can be reported and Postmortem information shared.
UPDATE:
The request has been stopped on the customer side. The support team has started running a script to clear these jobs from the system. The next update will be provided within one hour or sooner.
New Incident: CTIM-2421
Priority: High
Escalation sent to: Artwork Tech: Artwork Processing Support for review.
The Prepress PDF processing service experienced degradation in the eu-west-1 region after receiving more than 10,000 duplicate print-prep requests from a client identified as Druck.Amazon.OrderSync. The excessive request volume filled available storage and caused the PDF processing service to run out of space, leading to failures in image information extraction and overall PDF processing instability.
The issue impacted the Artwork Technology prepress pipeline and risked affecting PDF processing for multiple businesses relying on the service. As an immediate mitigation, cluster capacity was increased to maintain partial service availability while the team worked to identify and stop the source of the duplicate requests. Relevant team has been contacted to reduce the requests from the particular region. Next update in max 1 hour.
The Druck team was engaged to halt the incoming requests and prevent further load on the system. Additional mitigation steps included identifying and removing affected job IDs from the print-prep queue and exploring options to block the problematic client if necessary. Monitoring and investigation remain ongoing to ensure system stability.