CimCommerce v1
Backbone Customer Service Outage Impacting Discount Functionality
Incident Resolved: CTIM-2476
No additional problems have been reported. Incident is contained.
Please contact CTIM or see ticket for more details.
UPDATE:
The customer has confirmed that the issue appears to be resolved. At this point, we consider the incident contained but we will continue to monitor for some time before formally closing it.
UPDATE:
We are seeing continued issues with promocode updates not persisting (UI shows success but changes revert) which is impacting time-sensitive promotions. Team continues to look into this with high priority. Next update in max 30 minutes.
UPDATE:
However, a few residual issues are still being reported. Pixart has flagged a problem where promocode updates appear successful but are not being applied. The Commerce team is actively investigating this to determine whether it is related to the incident or a separate issue.
In parallel, teams are validating that the rollback has not introduced any unintended side effects. Follow-up work is planned, including coordination with AWS support, improving monitoring and preparing a safer rollout plan for next week.
Next update will be provided once all business units confirm stability and the remaining reported issues are validated/resolved.
UPDATE:
Rollback has been completed and signs of recovery are observed. Some services are starting to come back online, with impacted flows showing improvement. We will continue to monitor this for some time before proceeding to contain the incident.
UPDATE:
Engineering and DevOps teams are actively investigating and a rollback is being considered as a potential mitigation. Current estimates suggest a possible rollback within the next 10–15 minutes, though this is still tentative.
Next update will be shared when we have more details or max in 30 minutes.
UPDATE:
Adding affected Business Units
New Incident: CTIM-2476
Priority: Critical
Escalation sent to: Commerce Support for review.
A critical outage was reported in the Backbone customer service, resulting in a complete disruption of discount functionality across the platform. Users are unable to add discounts on the website, impacting multiple business units including Pixartprinting and Easyflyer, along with other dependent services.
The issue began around 11:45 AM IST (06:45 CET) and is affecting all users attempting to apply discounts. Initial investigation suggests a possible database failover event that led to dropped connections, causing the service outage.
Teams are actively investigating the root cause, including potential database and network-related issues. Coordination is ongoing to restore service functionality and validate system stability. Next update in max 30 minutes.