Europe
Suppliers are unable to ship items via either the POM or API
Incident Resolved: CTIM-909
No additional problems have been reported. We’re containing this incident again.
Please contact CTIM or see ticket for more details.
UPDATE:
Additional items were found where items were not marked as shipped in MCP. The batch of ~21k shipments was remediated by 6:30am ET today.
Incident Resolved: CTIM-909
Backlog has been cleared. Please let us know of any outliers so they can be checked. At this point, the incident is contained.
Please contact CTIM or see ticket for more details.
UPDATE:
Both large batches of backlogged orders have been cleared and Snorlax has been working on the remaining shipments (~500) that have been retried and were duplicated. Some additional reports from fulfillers have come in and are currently being reviewed but are likely latent effects from the original issue.
UPDATE:
Snorlax deployed a fix which should prevent the issue from recurring. However the buggy code existed for 23 hours, meaning that orders shipped during that time may be affected. The team is currently working on a solution to address these past orders with an initial focus on those that were not marked as shipped at all. The incident will be considered contained once all past orders have been resolved.
UPDATE:
The Snorlax team has identified the issue as a buggy code pushed yesterday. They are deploying a fix immediately.
UPDATE:
The fulfillment teams including Tradeprint and other fulfillers are facing issues where order statuses are not updating correctly, specifically failing to show as ‘shipped’ despite production and shipping completion. Pixartprinting and Easyflyer are also affected even though production and shipping are proceeding normally. More recent examples are being gathered to support further investigation.
UPDATE:
Snorlax team is currently online and looking into the issue. Next update will be shared in 30 minutes.
New Incident: CTIM-909
Priority: Critical
Escalation sent to: Logistics: Snorlax for review.
Seven suppliers are currently impacted and are unable to ship items through either POM or API. No errors are appearing but shipment events aren’t processing as expected. When items are marked as shipped they revert back to ‘In Production’ status.