Vista

UPDATE:

BHI IT has confirmed that plant Draw Gangs are working as expected. Incident Contained. No further updates.

UPDATE:

Team has identified problematic system and are working on mitigation with highest priority with some steps completed. Next update in max 30 minutes.

UPDATE:

Teams have identified a potential affected system and are taking action at highest priority to resolve the items stuck in draw gangs. Next update in max 30 minutes.

UPDATE:

It is unclear whether the Draw Gangs issue is related to the initial database issue, as it is impacting all plants. As a result, a new incident channel (CTIM-2238) has been created to share updates for all plants currently affected by gangs failing to draw. CTIM will continue to provide updates on the Draw Gangs issue in this channel as well.

UPDATE:

BHI IT is back online and confirmed that plant is producing again without issue. However, they have found one lingering issue with Drawgangs Service. Team is taking action to contain this lingering issue. Next update in max 30 minutes.

UPDATE:

The Database team reports that all production and reporting databases are back in a healthy state. The BHI IT team is currently offline, so end-user confirmation is still pending; however, system monitoring indicates that Scheduling and VIPER are functioning as expected.

Based on this, we will tentatively contain the incident at this time. If any issues reoccur, please re-engage VIPER Support. The Database team is not planning any further changes, and the environment is expected to remain stable.

No further updates are planned unless there is a change in status.

UPDATE:

Database teams are reporting the restore of the Database has completed and production should now be able to continue. The team is now focusing full effort on restoring the Reporting databases. Next update when we can confirm production is back up and running or in max 30 minutes.

UPDATE:

Database restore is currently in progress and being worked at highest priority by the DB team.

Early indications suggest approximately ~1 hour to bring the database back to a usable state; however, this is not yet guaranteed and may change as the restore progresses.

CTIM will share another update once there is clearer confirmation on recovery progress or timing or in max 1 hour.

UPDATE:

Team is working on restoring databases with highest priority. There is no current ETA. Next update in ~15 minutes.

UPDATE:

Fulfillment Database team is aware of incident and are working on this at highest priority. Next update in 15 minutes. Incident is a MPI for wider Bhiwandi plant communications.