Prepress

UPDATE:

Final Update: All impacted orders, totaling 34K+, have been successfully loaded and are now in the Prepress process. The order flow was disrupted for approximately 5.5 hours (2:40 PM ET – 8:25 PM ET), but everything is now operating as expected. The incident is considered resolved. A summary will be prepared and shared during business hours.

UPDATE:

The order flow is working as expected, and we have successfully loaded 12K+ orders to various Vista plants. Prepress queues have increased, and the Prepress team is actively monitoring them. We will consider this issue resolved once all orders have been successfully loaded.

UPDATE:

The change has been deployed, and we can see that the order flow has resumed. Currently, there are approximately 22K requests waiting to be downloaded. We will provide an update once all the orders have been downloaded.

UPDATE:

The Optic squad has come online and confirmed the URL. The MIS Production Models team is now attempting to update the endpoint in production to resolve the issue.

UPDATE:

We are still waiting for the on-call person from Optic squad to come online and assist with the order manager API endpoint.

UPDATE:

Production Models has prepped the change to the endpoint URL that should resolve the problem. They’re waiting to deploy until we can confirm this is a name change only and no changes were made to the service itself.

UPDATE:

Order flow has been near zero since ~2:30pm ET. We’re currently waiting for optic squad to come online to help diagnose the timeouts to the order manager API endpoint.

UPDATE:
Escalation sent to: eCommerce: Optic

Request assistance from Optic squad.

UPDATE:
Escalation sent to: eCommerce Support

Calls to cw-order-manager.commerce.cimpress.io are timing out. Engaging eCommerce Support.

UPDATE:

MIS UFI has been engaged for assistance. Production Models suspects some problem with the underlying VPCs for their services. Currently investigating that theory and suggest raising a critical issue with AWS if we continue to think that’s a potential cause.

UPDATE:

Upgrading this issue to MPI for visibility. The viper order loading continues to be problematic since first observed at 10am this morning. There was some recovery from 12-1pm but the problem reoccurred and orders continue to not load into viper.

The plants who have weekend shifts will likely start to feel the impact soon. This issue was raised as an MCP problem initially but appears to be hung on the MIS side, which is still being diagnosed.

UPDATE:

Production Models is currently reviewing some errors they’ve identified in the logging after the latest drop in orders.