Supplier Integrations
Order flow down due to routing for over 30 mins
Incident Resolved: CTIM-1037
The Dispatch squad informed that FNES was impacted because it depended on the Swagger definition of another service, which was updated yesterday afternoon. As the service scaled (it doesn’t need to scale much), new instances of the service broke when they tried to access the other service’s Swagger definition. This caused orders to slowly get stuck. The Dispatch squad has deployed to instances and pushed to production. The backlog is currently at 4.5K orders. eCommerce confirmed that the issue is now resolved, as Viper is now receiving orders at all its plants.
Please contact CTIM or see ticket for more details.
UPDATE:
The dispatch squad has joined Slack and found the issue with FNES. They are currently working on the fix.
New Incident: CTIM-1037
Priority: Critical
Escalation sent to: PCD: Product Operations for review.
eCommerce have receive a pager for “Signal lost for 3 minutes on ‘Reduced Fulfillment Workflow Throughput’ “seems like a bunch of orders stuck in routing. This IMPACTS order flow, no order is being processed.