Order Management

We are restarting Item Service. To handle the requests, a small number of healthy instances will be scaled up. This will cause an approximately 40 minute window of increased down time.

Order injection from the websites to the MCP Order Workflow will remain unaffected. There will be some delays with orders reaching manufacturing.

Order processing errors experienced by fulfillment systems will be retried. All customers of MCP Shipping are experience errors shipping orders.

We are currently working with AWS to get the instances healthy (by terminating dead instances manually), we believe that once we restore its health things should improve.
But still there is no update in ETA yet.
Impacted services:- No such major impact on new orders, services dependent on Item-service will function with degraded service.
Printdeal has few orders for which they are getting 500 error, while Venlo is facing issue with shipment.

Update : Team is attempting to downgrade the degraded instances and add new one to cluster.
There is not update on ETA yet.
Will share the update in next 30 mins or there is some update whichever is sooner.

Update : There is no new update on the incident. Pipeline deployment is still failing, working with AWS support team to fix this.

Update : There is no new update on the incident. Pipeline deployment is still failing, working with AWS support team to fix this.

Update : The current manual deployment failed due to degraded instance, hence we have escalated to AWS to assist in recovering the degraded instances which shall help in re-deploying the pipeline.

Update : Pipeline running on production went into error, hence team will be attempting to manually push the fix to the pipeline