CimCommerce v2
Users Blocked at Checkout
Incident Resolved: CTIM-2483
Vista has confirmed that the checkout and shipping issues have been resolved. Incident is now contained.
Please contact CTIM or see ticket for more details.
UPDATE:
All the affected services have been stable for over an hour. We will wait for confirmation from Vista before containing the incident.
UPDATE:
Support teams were able to reduce processing for the RANCH service and response times have stabilized again after a spike from 11:30-11:45am ET. Product Catalog teams are reviewing additional options for mitigation to minimize the risk of reoccurrence.
Vista reports correlations in blocked checkouts for each of the RANCH spikes but have improved overall since the initial wave. The occurrences of ‘no shipping options’ have remained low since the recovery of SPEEDO/EDO services.
We will continue to monitor the situation and provide another update in 30 minutes.
UPDATE:
The RANCH service saw another spike from ~10:55am - 11:20am ET but has stabilized again. SPEEDO and EDO have been stable since 11:05am ET. Support teams continue to monitor the situation. Next update in 30 minutes.
UPDATE:
Recent changes to EDO that affected the scaling capability of the service are being reverted which should help recovery. We will continue to monitor the service health and provide another update in 30 minutes or sooner.
UPDATE:
RANCH service remains stable since ~10:10am ET. CT Commerce and Product Catalog teams are reviewing the SPEEDO and EDO services which are still showing increased response times and error rates. Next update in 30 minutes.
UPDATE:
Escalation sent to: PCD: Dispatch Squad
Engaging Product Catalog devs to review the ongoing degradation of the SPEEDO service.
UPDATE:
RANCH service response times have improved after teams recycled the underlying pods. Vista reports that shipping options are also affected. This could be a related symptom but that’s still under investigation. Next update in 30 minutes or sooner.
New Incident: CTIM-2483
Priority: High
Escalation sent to: Commerce Support for review.
Vista has reported an issue blocking users from checkout due to product validation errors. The problem has already affected 365 customers on the Vista website. CT Commerce teams have already joined and found increased response times from the RANCH and SPEEDO services.