Product Configuration
Unexpected Surge in Order Item Failures (~500) with "500 Internal Server Error"
Incident Resolved: CTIM-1400
Contain
Please contact CTIM or see ticket for more details.
UPDATE:
PCD is looking into possible short-term fixes and reviewing changes that might have caused the issue. Current impact has been mitigated, though we recommend Vista monitor for additional failed items should another spike occur. We’ll keep the incident open for now in case there’s any further reports from the fulfiller.
UPDATE:
Vista Orders team has resubmitted the failed orders. CT Product Catalog teams are still investigating the errors from traffic spikes in the Product Config Service.
UPDATE:
Product Operations is aware of intermittent errors from the Product Config service caused by extreme load spikes. An immediate remediation is to retry the call to the Config service, as the issue is intermittent. The Order team is currently working on retrying the failed orders. Next update will be provided in 30 minutes or less.
UPDATE:
The team is investigating a critical priority incident. where Vista has observed a significant increase in order failures over the past few days across multiple products and product categories. Next update will be provided in 30 minutes or less.
New Incident: CTIM-1400
Priority: Critical
Escalation sent to: PCD: Product Operations for review.
CTIM is working on a critical priority incident. Since May 5, Vista has observed an unexpected increase of approximately 500 order item failures, all returning a “500 Internal Server Error.” The issue appears to be related to a failure during order validation, specifically when attempting to retrieve a product configuration from the CDN. Next update will be provided after the Product Operations team joins the Slack channel.