Product Configuration
Increased 404 errors on merchant orchestration API service
Incident Resolved: CTIM-1524
Cimpress team has deployed an infrastructure change at 2am EST today with positive results thus far. Incident Contained. Teams to monitor for 24 hours before fully Resolving the incident.
Please contact CTIM or see ticket for more details.
UPDATE:
Team has identified a few infrastructure optimizations that may help resolve the issue and is currently working on implementing them. Next update when we hear back from them or max in 2 hours.
Incident Resolved: CTIM-1524
Teams actively investigating this contained incident. Next update in max 2 hours.
Please contact CTIM or see ticket for more details.
New Incident: CTIM-1524
Priority: High
Escalation sent to: PCD: Product Operations for review.
On June 11, 2025, between 00:00 and 00:05 UTC, an abnormal spike in 404 errors was observed when calling the Merchant Orchestration API for thousands of different product keys. This caused active products to be incorrectly treated as retired, impacting product recommendations. Despite early engagement from Platform Support, CT Product Operations, and the DEV team, the issue remained unresolved for over a week, prompting concerns from stakeholders. Requesting the team to look into this issue during the business hours.