The technical teams have completed their root cause and corrective action analysis. The post mortem can be viewed here - https://cimpress-support.atlassian.net/wiki/spaces/DCR/pages/15872196734/Jan+31+2024+-+Order+backlogs+in+prepress+steps+are+increasing.].
It has been an hour since this incident was contained, and we have not seen a recurrence. We are marking this incident as resolved. Within five business days, we will post a link the the post mortem of this incident here.
We believe we have contained this incident and will continue to monitor it. Unless there is a recurrence, we will mark this incident resolved in one hour.
The team is working to clear the job backlog and will update you soon.
Several hundred callbacks are currently stuck in retries, and it seems unlikely that they will succeed. This is possibly due to the callback URL being configured for the development or testing environment. We keep monitoring it.
The job count appears to be gradually decreasing. Our team is actively addressing this issue and will provide you with an update soon.
It seems the job count is decreasing. We believe we have contained this incident and will continue to monitor it. Unless there is a recurrence, we will mark this incident resolved in one hour.
print-prep side looks more or less normal; there are no backlogs or a constant stream of errors, various services do have small matches of retries that are normal, but could also be responsible for delays.
the order backlog in prepress steps is increasing. The artwork processing service is getting timeout The technical team is investigating. We will update this status page again as soon as we have more information or in 30 minutes, whichever is sooner.