Supplier Integrations
Increased Suspended Workflows in FRWC
Incident Resolved: CTIM-729
Team suspected an issue with the network configuration and has refreshed the instances. This refresh has addressed most of the issues they were noticing, but they have a plan to implement additional guardrails monitoring. Currently, the service seems to be in a stable state, so we are moving this incident to contained status. We will keep it under monitoring for the next few hours before resolving it.
Please contact CTIM or see ticket for more details.
Incident Updated: The team is currently using retries to prevent orders from being stuck for too long while continuing their investigation. Although they haven’t found any suspended errors, they have implemented additional monitoring and are closely observing the situation while reviewing past logs.
Incident Updated: Adding Garuda to check fulfillment config service. Example error being seen during retry of suspended items:
“Unable to get data from: https://api.fulfillmentconfigurations.cimpress.io/v0/versions/bfd80b40-c947-11eb-8e66-b1f7494dc205”
Incident Updated: PCD, FEDs and Routing teams are still investigating to determine the cause. Suspended items are being managed by retries, though the queue rises again afterward. Incident is ongoing.
Incident Updated: Support teams in PCD are reaching out to routing for additional assistance in identifying the problem. Will update again in 30min or sooner if new details become available.
Incident Updated: Suspended items queue is being actioned to keep orders processing, but it rises again after each cycle. FEDs are continuing to work on identifying the cause.
Incident Updated: FEDs online and reviewing the issue. Suspended items have been retried manually to mitigate the current impact.
Incident Updated: Teams are currently reviewing the suspended items. Updated category/service to ‘Supplier Integrations’.
New Incident: CTIM-729
Priority: High
Escalation sent to: PCD: Product Operations,PCD: FEDs for review.
Alert was triggered for an increase in suspended workflows for FRWC. Adding Product Ops and FEDs to help review the ongoing issue. Initial backlog was retried but has been increasing again (>20).