All workflows queued for document transformation are becoming stalled

分析 2024年04月08日 08:34 EDT

The technical teams have completed their root cause and corrective action analysis. The post mortem can be view [https://cimpress-support.atlassian.net/wiki/spaces/DCR/pages/15988360858/Apr+1+2024+All+workflows+queued+for+document+transformation+are+becoming+stalled].

解決済み 2024年04月01日 04:11 EDT

We are marking this incident as resolved. Within five business days, we will post a link the the post mortem of this incident here.

監視中 2024年04月01日 03:49 EDT

We are noticing some enhancements, with a few orders now beginning to progress smoothly.

原因確定 2024年04月01日 03:46 EDT

The issue stemmed from a caching problem in the rule service client. Despite today’s secret rotation, the rule service continued to use the outdated secret due to caching. The team has now updated the secrets, and this action is expected to resolve the issue.

検証中 2024年04月01日 03:20 EDT

We found that all workflows entering doc transformation are getting stuck. As per the initial investigation, it seems the rules service started getting time to return the rule configuration. The technical team is investigating. We will update this status page again as soon as we have more information or in 30 minutes, whichever is sooner.