Order Management

UPDATE:

After previously mentioned steps taken by Optic team, VIPER is seeing order flow coming back for the last ~15 minutes. There are roughly 12k orders at MCP level that need to be processed and make their way into production. CTIM is awaiting confirmation these items have made it through the process and into production before considering this incident contained. A Summary, including what we know about Root Cause at this time, will go out within 24 hours. Next update when backlog is processed.

UPDATE:

While there is yet no confirmation on Root Cause, Optic Squad has found a long running process that was hanging up their services as well as the previously mentioned steps of redeploying app and running an update. We are awaiting results of these steps from the team. For now, order flow is still blocked and this issue continues to be looked at with highest priority.

UPDATE:

After further investigation, Optic Squad has found a potential lead. The team is currently re-deploying one of their applications as well as running an update. Other teams continue to check on service health as we wait to see if actions by Optic squad show positive results. Next update in 30 minutes.

UPDATE:
Escalation sent to: PCD: FuN (Trdelnik)

Multiple teams from multiple domains continue to investigate, but so far no root cause has been found affecting order flow. CTIM requesting Trdelnik squad to do a health check on their services.

UPDATE:
Escalation sent to: PCD: Product Operations

Vista PBM has confirmed that Vista-cart is healthy and customer check outs are not affected. However, we still have no clear understanding of what is blocking order flow between customer check-outs to the MCP to VIPER. E-Commerce Domain has exhausted their leads and CTIM is casting a wider net. CTIM is now adding Product Operations to escalation with request they help us to see if there are any leads on their side. Next update in 30 minutes.

UPDATE:

Teams continue to investigate extreme low order volume. It is still unclear where the orders are not coming from. CTIM has contacted Vista PBM to check again that customer check-outs are successful. Other teams continue to investigate, but have since found that while their services may be affected, they are not at root cause.

UPDATE:

E-Com Support as well as other MCP teams continue joining channel. Investigation continues. Next update in 30 minutes or less.