Inventory and Delivery Planning

UPDATE:

Support teams are still monitoring the EDO metrics and fallback rates. The services seem stable at this point. We will continue to monitor the channel for activity and provide another update in 1 hour.

UPDATE:

Fallbacks remain elevated but still improved from the initial onset of the incident earlier today. Support teams are narrowing the possible causes. Next update in 1 hour or sooner.

UPDATE:

Spike in fallback rates continues since ~3:20pm ET. Database has stabilized though no recovery at this point. Incident does not appear to be related to ongoing Item Service issues. Next update in 30 minutes.

UPDATE:

Expected Delivery timeouts have increased again, so we are re-opening this issue. PCD teams are actively working the problem. Next update in 30 minutes.

UPDATE:

Services have been stable for ~30 minutes now, confirmed by PCD and Vista teams. We will monitor for another 30 minutes before closing the issue.

UPDATE:

Support teams have identified some services that are hitting their current scaling limit, causing periodic degradation. They are updating the config to allow more instances which should help. Next update in 30 minutes.

UPDATE:

Vista teams indicate that the fallback rate improved at ~12:03pm ET, but spiked again in the past 10min. This is confirmed by CT PCD, who acknowledged some lingering issues that they are working to correct. Next update in 30 minutes or sooner.

UPDATE:

Team is still investigating at the highest priority. Next update in max 30 minutes.

UPDATE:

The appropriate teams are investigating the EDO timeouts issue at critical priority. Next update in max 30 minutes.

UPDATE:
Escalation sent to: PCD: Dispatch Squad

Paging additional teams to investigate.