North America

UPDATE:

Patch is ready to be deployed. Specifically, the changes are targeted at BHI, DPK and VEN which will help the situation we’re seeing in WND. We will follow up with the plant after they’ve each had a chance to deploy in the next 12-24 hours.

UPDATE:

Prod patch is still being finalized. We will update again once it’s ready for deployment to production.

UPDATE:

The fulfillment are planning to release a patch by tomorrow morning in addition to the ongoing work in the database cleanup that should help mitigate the impact of the ‘task canceled’ errors. Next update after the patch is deployed (tomorrow morning ET).

UPDATE:

Support teams have identified a database issue as the likely cause. The cleanup process for the database will require some additional time to implement and process. We will confirm a timeframe by tomorrow and follow up with the next steps.

UPDATE:

Fulfillment support/dev teams have narrowed the possible causes and will resume the investigation during business hours.

UPDATE:
Escalation sent to: Fulfillment: Logistics: Gengar

Teams continue to investigate and attempt various mitigations for the ‘task canceled’ errors in Windsor.

UPDATE:

Still monitoring after the patch was deployed. The improvements have helped resolve the observed issues with the query that was originally identified. Some ‘task canceled’ errors are still occurring due to a different query which will be evaluated. We will keep the incident open and check in next week.

UPDATE:

Patch released that should help with the slowness. Plants can deploy as needed. We will follow up tomorrow with an update after the teams have had some time to evaluate the fix.

UPDATE:

The support team is working on a change to the Orders GraphQL service to address the slowness. The next update will be provided after the change is implemented.

UPDATE:

The team is still investigating the high-priority issue affecting the Windsor plant, where ‘task was canceled’ errors are being seen from the Shipping Builders Service and Orders Service. Next update in 1 hour or less.