Vista
Viper Access Failure Due to Load Balancer Issue Impacting Plant Users
Incident Resolved: CTIM-2482
Support Teams have contained or resolved the issue. Please contact MIS PBM or see ticket for more details.
UPDATE:
Escalation sent to: Vista: Networking
Adding additional team for further investigation. The issue has been resolved and we are requesting the team to prioritize the investigation during business hours.
UPDATE:
The team has rebooted all impacted VPR servers in a controlled manner to stabilize the environment and all servers are now back online. We will continue monitoring system behavior and user access to ensure there is no recurrence.
Next update will be shared once RCA is confirmed or if any further impact is observed. We have requested the Dev team to assist with this investigation. The next update will be shared when the team is online during US business hours or within a maximum of 4 hours.
UPDATE:
Escalation sent to: Fulfillment: Infrastructure Systems Engineering
Adding additional team for assistance.
UPDATE:
The team has fixed the issue and customer also confirmed that they are able to access viper again at the first try. Devops team will continue investigating the issue to find the root cause. We will monitor this for some time to ensure everything is working fine. Next update when we get more information on the root cause or max in 1 hour.
New Incident: CTIM-2482
Plant: VEN
Priority: Critical
Escalation sent to: Fulfillment: Infrastructure Dev Ops (UFI) for review.
Users in the VEN plant are experiencing connectivity issues while attempting to access Viper via Remote Desktop, receiving connection errors . The issue was initially reported on March 30 and persists as of March 31, impacting a large number of users and preventing them from performing day-to-day operational tasks such as scheduling work and accessing required tools.
The problem appears to be related to the Viper load balancer, which is currently unavailable or not routing traffic correctly. The team is investigating this with high priority. Next update in max 30 minutes.