UPDATE:
Tradeprint will not be able to transition to the new app until tomorrow which will help with the transfer of new orders. They have increased the timeout period to 10 hours to allow more time for the data of backlogged orders to download. Teams will reconvene in the morning to see if the increased timeout has resolved the issue. Next update to be tomorrow during CET business hours.
UPDATE:
Upon moving the Switch app, Tradeprint still observed errors in transmitting the needed data. CT and Tradeprint are discussing next steps towards remediation. Next update once more information is available or EoD ET business hours.
UPDATE:
Tradeprint experienced another blip in incoming data and are attempting to move the Switch app to test. Next update when results from this testing are shared with CTIM.
UPDATE:
Multiple teams are actively working on order remediation. Next update when we have more information or max in 2 hours.
UPDATE:
Multiple teams are engaged to discuss the remediation. Next update in max 2 hours.
UPDATE:
Escalation sent to: PCD: Product Operations
Requesting the support team to assist further with remediation. The next update will be provided within a maximum of 2 hours.
UPDATE:
Tradeprint has shared the list with the CT teams to verify whether all jobs received were transmitted and to identify any missing jobs. Tradeprint also informed that there will be no one from online until 9 AM GMT (9 AM UTC). CTIM will check with Product Ops once they log in within 3 hours.
UPDATE:
Tradeprint teams will continue to work through remediation during business hours. Next update to be sent tomorrow during CET business hours.
UPDATE:
The results of the potential fix appear to be successful. Tradeprint will be making an adjustment to force download the data. Once completed, teams will run another cycle and confirm if all orders were successfully sent. Next update by EoD today PST business hours.
UPDATE:
Teams are still running the receive cycle and awaiting its results. Next update to be provided by EoD PST business hours.
UPDATE:
Teams are awaiting results of rejecting and resubmitting orders before following the process with all the affected orders. Next update in 1 hour.
UPDATE:
CT and Tradeprint teams continue to troubleshoot additional options. Teams may be able to reject the orders and manually resubmit. Next update in 30 minutes.
UPDATE:
CT and Tradeprint teams are still actively working the issue. They are trying to prevent additional failed items in the backlog while adjusting the timeouts to ensure files are transferred successfully. Next update in 30 minutes.
UPDATE:
Support teams are still determining the best options for remediation considering the current integrations/versions of switch that are being used. Next update in 30 minutes.
UPDATE:
Product Catalog dev teams are currently working to review the orders and prepare to retransmit the affected data. Next update in 30 minutes or sooner.
UPDATE:
Escalation sent to: PCD: Trdelnik Squad
Adding Product Catalog dev team for assistance.
New Incident: CTIM-2465
Priority: Critical
Escalation sent to: PCD: Product Operations for review.
Tradeprint has reported an outage that resulted in disrupted downloads for ~200 jobs from MCP Switch adapter. The orders include many Vista rush items that are unable to be retrieved. We’re engaging Product Catalog teams for assistance.