Vista

UPDATE:

Incident is contained.

UPDATE:

Production has continued to work through the occasional slowness as dev teams identify the edge cases where the system hits a bottleneck. The load on the affected services has also decreased, which should reduce the occurrences of slow scans in shipping.

We will monitor the channel for any changes and close the incident by the next on-call shift for CTIM (~6 hours) if no further action is needed.

UPDATE:

Fulfillment dev squads have confirmed they’re aware of the slowness in the system related to a specific shipping group and are monitoring the issue. We will update again by 5pm ET with any additional findings.

UPDATE:
Escalation sent to: Fulfillment: Planning/Data Investigation

The example provided belongs to a large order that could require longer than usual wait times when scanning those items, especially during periods where there’s additional planning ongoing in the background from the related services. Production has been able to ship the packs during retries.

The services themselves are stable, but show some increases in transaction times. We’ve asked production to continue to report anomalous behavior, especially if an item will not ship after retry. Next update in one hour as we continue to monitor the situation.

Adding Fulfillment dev squad for visibility, but not paging at this point.