Supplier Integrations

Incident Updated: Rate of suspended orders has decreased at this point. There were some infrastructure modifications to Order Manager service, although the actual root cause has yet to be determined and teams are still reviewing the 4XX errors seen previously in FES calls to other services.

Incident Updated: Support teams are working multiple options to identify the root cause, including checking Item Service and COAM errors. To mitigate the current symptoms, a workaround is being put in-place to translate the 4XX errors to 5XX in FES, which will allow those items to be retried. Additional logging on the retries that continue to fail should provide more insight into the original cause.

Current backlog is being pushed manually in small batches to minimize the risk of flooding downstream services. Current stuck items are below ~1000.

Incident Updated: Stuck orders still increasing and outpacing attempted retries, currently ~3700 total stuck in routing. Optic squad online to assist FEDs with the diagnosis of intermittent 400 errors (‘Unable to get fulfillment request’). Will update again in 30min or sooner if the problem is identified.

Incident Updated: Support teams are still investigating and trying to identify the cause of the downstream 404s being returned to FES. Requesting Optic squad (via eCommerce support) to join the discussion to find out if any of their services might have had a recent change that could be returning those error responses.

Incident Updated: FEDs Squad is still investigating the orders stuck in routing issue at the highest priority. Currently the team is working together in Zoom to resolve this, but have not yet had luck with any of their containment attempts. In the meantime, they are able to mitigate the impact but retrying orders and keeping the backlog of orders stuck in Routing low. Next update in 30 minutes.

Incident Updated: FEDs squad is continuing to investigate. Currently, orders are still stuck in Routing with investigation continuing at the highest priority. Next update in 30 minutes.

Incident Updated: Product Catalog’s FED’s Squad is actively investigating high number of orders that are currently stuck in Routing. Next update in 30 minutes.