Inventory and Delivery Planning
Increased failures from "Shipment Planner" investigation
Incident Resolved: CTIM-671
Since token renewal was undertaken by Firecamp all affected services owned by Dispatch have remained stable and Pixart has not reported any new occurrences. CTIM to Contain the incident at this point and finish up with a Summary within 24 hours.
Please contact Platform Support or see ticket for more details.
Incident Updated: Firecamp has finished token renewal process and Dispatch is reporting all workflows previously suffering from failures are now in a healthy state. CTIM to confirm with Pixart that there is no further impact.
Incident Updated: Firecamp did not have success with the first round of new tokens. The team is currently investigating why these were rejected. Pixart still only BU who has shared impact and they are reporting they’ve seen some improvement. Next update at resolution or in 30 minutes.
Incident Updated: Firecamp is working on fixing an issue wherein Gitlab had set expiration dates on tokens where there previously were none. These now expired tokens have caused some instances for Dispatch service to become unhealthy resulting in some items stuck in Planned Shipment process. Next update when incident resolved or at 30 minutes.
Incident Updated: Firecamp and Dispatch investigating lead that issue could be related to Gitlab System Status. Next update in 30 minutes.
Incident Updated: Firecamp (Dev Ops Infrastructure for Dispatch services) is currently investigating potential relationship between storage issues (ECR images) and its potential affect on autoscaling of Shipment Planner.
New Incident: CTIM-671
Priority: High
Escalation sent to: CT Incident Management for review.
Dispatch Squad (Shipment Planner owners) along with AT Squad (Workflow Pricer owners) are currently investigating an increase in 503 errors causing items to become stuck in routing at the shipment planner step.