Inventory and Delivery Planning
Elevated response times from Link and Product services
Incident Resolved: CTIM-1193
At this point, the incident is contained. Please let us know if there’s any additional reports of problems with order routing or delivery options.
Please contact CTIM or see ticket for more details.
UPDATE:
Fallback rates for EDO are down to ~1%. The impact to customers should now be minimal. We will continue to monitor the stability of the affected services until we can be sure the issue is fully contained.
UPDATE:
The majority of orders are now successfully being routed. IDP response rate remains increased. Printdeal was confirmed to be impacted with issues to product APIs and routing. The issue is still under investigation.
UPDATE:
For the past hour, Feasibility and Fulfillment Options remain stable, though the IDP response times continue to be higher than usual. EDO fallback rate is currently at 7%. Rate limiting has been implemented temporarily while PCD teams work to identify the cause.
UPDATE:
EDO currently showing 88% success rate. Delivery Feasibility and Fulfillment Option Data service appear to have been affected during the same timeframe. IDP response times remain elevated. Teams are monitoring the issue.
New Incident: CTIM-1193
Priority: High
Escalation sent to: PCD: Product Services for review.
Teams are proactively investigating monitoring alerts for increased response times from Linker and Product Services. This is being investigated at the highest priority by the appropriate teams due to the high risk on IDP/EDO and other downstream services.