CimCommerce v2

UPDATE:

All the affected services have been stable for over an hour. We will wait for confirmation from Vista before containing the incident.

UPDATE:

Support teams were able to reduce processing for the RANCH service and response times have stabilized again after a spike from 11:30-11:45am ET. Product Catalog teams are reviewing additional options for mitigation to minimize the risk of reoccurrence.

Vista reports correlations in blocked checkouts for each of the RANCH spikes but have improved overall since the initial wave. The occurrences of ‘no shipping options’ have remained low since the recovery of SPEEDO/EDO services.
We will continue to monitor the situation and provide another update in 30 minutes.

UPDATE:

The RANCH service saw another spike from ~10:55am - 11:20am ET but has stabilized again. SPEEDO and EDO have been stable since 11:05am ET. Support teams continue to monitor the situation. Next update in 30 minutes.

UPDATE:

Recent changes to EDO that affected the scaling capability of the service are being reverted which should help recovery. We will continue to monitor the service health and provide another update in 30 minutes or sooner.

UPDATE:

RANCH service remains stable since ~10:10am ET. CT Commerce and Product Catalog teams are reviewing the SPEEDO and EDO services which are still showing increased response times and error rates. Next update in 30 minutes.

UPDATE:
Escalation sent to: PCD: Dispatch Squad

Engaging Product Catalog devs to review the ongoing degradation of the SPEEDO service.

UPDATE:

RANCH service response times have improved after teams recycled the underlying pods. Vista reports that shipping options are also affected. This could be a related symptom but that’s still under investigation. Next update in 30 minutes or sooner.