Product Configuration
Items suspended in Shipment Planner workflow. Manual retry does not seem to be working
Incident Resolved: CTIM-1374
Dispatch squad has generated new plans for the remaining 128 orders. The team will coordinate with the Optics squad to verify if there are any additional suspended items to address. At this point the incident is contained.
Please contact CTIM or see ticket for more details.
UPDATE:
Dispatch and Optics squads continue to work with the highest priority to clear the remaining 128 suspended orders. The next update will be shared in 30 minutes or sooner.
UPDATE:
Dispatch squad has successfully generated new shipment plans for 384 out of the 512 orders that were stuck in the Shipment Planner. The Optics and Dispatch squads are now jointly working with the highest priority to resolve the remaining 128 orders. Next update in 30 minutes or sooner.
UPDATE:
Dispatch squad was able to manually push a test order through successfully and is now actively working on scripting a fix to retry the rest of the affected orders (~500 items) with high priority. Next update in 30 minutes or sooner.
UPDATE:
Dispatch squad identified that the Shipment Planner is failing to retrieve Fulfillment Delivery Options (FDOs) during the ‘patching to item’ step. The issue is being treated as critical and the team is actively investigating to determine the root cause and restore functionality.
UPDATE:
The Dispatch squad is online and actively investigating the issue with the highest priority. Next update within 30 minutes or sooner.
New Incident: CTIM-1374
Priority: Critical
Escalation sent to: PCD: Dispatch Squad for review.
Orders are currently suspended in the Shipment Planner workflow and manual retries appear to be ineffective. The issue began with failures on April 21 at 1:00 PM UTC which led to suspensions starting around 8:30 PM UTC the same day. As of now, 507 orders across various merchants are affected.