All workflows queued for document transformation are becoming stalled

Post-mortem Abr 8, 2024 8:34 EDT

The technical teams have completed their root cause and corrective action analysis. The post mortem can be view [https://cimpress-support.atlassian.net/wiki/spaces/DCR/pages/15988360858/Apr+1+2024+All+workflows+queued+for+document+transformation+are+becoming+stalled].

Resolvido Abr 1, 2024 4:11 EDT

We are marking this incident as resolved. Within five business days, we will post a link the the post mortem of this incident here.

Monitorando Abr 1, 2024 3:49 EDT

We are noticing some enhancements, with a few orders now beginning to progress smoothly.

Problema Identificado Abr 1, 2024 3:46 EDT

The issue stemmed from a caching problem in the rule service client. Despite today’s secret rotation, the rule service continued to use the outdated secret due to caching. The team has now updated the secrets, and this action is expected to resolve the issue.

Investigando Abr 1, 2024 3:20 EDT

We found that all workflows entering doc transformation are getting stuck. As per the initial investigation, it seems the rules service started getting time to return the rule configuration. The technical team is investigating. We will update this status page again as soon as we have more information or in 30 minutes, whichever is sooner.