Inventory and Delivery Planning
Order Flow Issue
Issue was resolved on Mar 18, 2024 7:00 PM EDT but inadvertently reopened due to an additional update being sent after containment.
Incident Updated: As of 19:30 ET on March 18, Routing Service had caught up on processing backlog following the incident wherein Capacity workflow was failing and items stuck in Routing. Incident is now Resolved. No further updates, CTIM Summary to follow within 24 hours.
Incident Resolved: PS-103867
No further blockages in workflows for ordered items have been found. Current backlog of affected orders are processing as expected. Incident is now contained.
Please contact Platform Support or see ticket for more details.
Incident Updated: Capacity service remains stable and orders are continuing to process to all fulfillers.
Incident Updated: Issue with failed tasks within the ECS and related EC2 instances for capacity workflow has been identified and remediated. Items now appear to be running through the workflow as expected but will take some time to progress through all the steps and routed to fulfillers. We will wait to contain the incident until we have a better idea of how long it will take to process through the backlog.
Incident Updated: Teams have confirmed the errors to be originating from the prod capacity service workflow. Attempting to restart those ECS instances to resolve the status ‘503’ errors.
Incident Updated: Support teams are currently working the issue. Currently waiting for diagnosis from Feds squad regarding errors originating from capacity services. Next update in 30min or sooner if additional information becomes available.
Incident Updated: Redirecting escalation to PCD: Dispatch (already online)
New Incident: CTIM-572
Priority: High
Escalation sent to: eCommerce Support for review.
Order flow stopped from MCP to Vista at ~15:30 ET. Items appear to be held in document review but sending to eCommerce for review before engaging Artwork support team directly.