Vista
Increases in Payment Core errors
Incident Resolved: CTIM-1610
Error rates have dropped and the situation has remained stable for the past 30–40 minutes. We are now proceeding to contain the incident.
Please contact CTIM or see ticket for more details.
UPDATE:
The team has restarted the application and everything looks good on the database side. The customer has observed a drop in the error rate and we are seeing signs of improvement. The issue is expected to be fully resolved within a few minutes. We will monitor this for another 30 minutes to make sure its completely resolved.
UPDATE:
The team is now able to place new orders from their account. However, a few SQL errors are still being observed, and the team is working on resolving them with high priority. The next update will be provided in 15 minutes.
UPDATE:
The query execution was interrupted as the team had to scale up the RDS instance due to a CPU spike. A reboot was also performed, during which the database instances were swapped— the reader became the writer and the writer is now acting as the reader. The team is currently on a call with AWS Support and actively seeking their guidance on the next steps. Next update in another 15 minutes
UPDATE:
The team is currently running the query, which is expected to take around 10 minutes to complete. So far, there have been no new cart requests and the impact remains minimal. The next update will be shared once the query completes successfully or max in 15 minutes.
UPDATE:
The team is able to connect to the database but high CPU usage is causing delays in the indexing query. The issue has been escalated to AWS Support. We will actively follow up with the next update in 15 minutes.
UPDATE:
The team is able to connect to the database but high CPU usage is causing delays in the indexing query. The issue has been escalated to AWS Support. We will follow up with the next update in 15 minutes.
UPDATE:
Team is trying to connect to our Database after the reboot. They have involved Devops in this and they are fixing the connection issue with priority. Next update in 15 minutes.
UPDATE:
The team has restarted the instance and they are working on fixing by adding some necessary indices on their prod database with high priority. Next update in max 15 minutes.
UPDATE:
No updates so far and team continues to investigate the issue. Next update in max 15 minutes.
New Incident: CTIM-1610
Priority: Critical
Escalation sent to: CTIM for review.
Vista users seeing a significant increase in Payment Core errors causing VIsta’s successful checkout to drop to zero. The relevant team is actively working on fixing the issue of Payment Core errors. Next update in max 30 minutes