CimCommerce v1
Orders Stuck in RFF Due to Queue Processing Issue in Cockpit
Incident Resolved: CTIM-2410
After monitoring, we are containing this issue with no further reports observed. Incident is contained.
Please contact CTIM or see ticket for more details.
UPDATE:
The queue is currently clearing. There are no messages stuck in the queue and several orders have already been processed successfully. The number of impacted WFDR items has reduced from ~680 to ~250 and the ‘Ready for Fulfillment’ items are also gradually being cleared. We will monitor this for sometime before containing this incident.
UPDATE:
Team has redeployed the instances and do see some improvement now. They can see messages are getting cleared from queue at a steady pace. Next update in max 30 minutes.
UPDATE:
After a drop in messages, we are again seeing an increase in orders getting stuck. Team is currently increasing the instance count and restarting the server. Next update in max 30 minutes
UPDATE:
Team has restarted the relevant instances for all impacted Merchants and currently monitoring them now. They see drop in stuck messages from the SQS. Next update when we have more details or max in 30 minutes
New Incident: CTIM-2410
Priority: Critical
Escalation sent to: Commerce Support for review.
An issue was reported where orders were not progressing in Cockpit due to messages stuck in SQS queues, resulting in orders being held in the RFF and WFDR stages and causing event status mismatches. The problem is impacting multiple business units.
Initial reports indicate that over 310 items are already impacted, and new orders are continuing to get stuck as messages remain unprocessed in the affected queues. The Commerce domain support team has begun investigating the issue with high priority and CTIM will provide next update in max 30 minutes