Incident Resolved: CTIM-1732
Venlo team confirmed that disabling Verification service resolved the issue. Incident is contained.
Please contact CTIM or see ticket for more details.
UPDATE:
Venlo team has disabled the Verification Service as a test and are monitoring to see if this resolves the issue. Team continues to monitor to see the results. Next update when we get more information from the team or max in 2 hours.
UPDATE:
Teams continue to investigate at highest priority. Next update in max 1 hour.
UPDATE:
Support teams are going to continue monitoring the issue overnight to gather additional data for review. Next update will be in ~12 hours (in the AM for EU).
UPDATE:
Both CT Access and Logistics teams have reviewed and tested their services but found no issues that would present the reported 403 errors. Support teams are still investigating to find another possible cause. Next update in 1 hour or sooner if additional details become available.
UPDATE:
Logistics teams are currently diagnosing the issue. No additional details at this point. Next update in 30 minutes.
UPDATE:
Escalation sent to: Fulfillment: Logistics: Snorlax
Adding Logistics dev team for assistance.
UPDATE:
Access teams found no permissions changes or problems for the clients being used in the requests. Support teams are currently discussing other possibilities for the errors. Next update in 30 minutes.
UPDATE:
Escalation sent to: Access Domain
Adding Access teams for assistance.
New Incident: CTIM-1732
Priority: Critical
Escalation sent to: CTIM for review.
VEN production has reported issues when shipping items. They receive a 403 ‘Forbidden’ error at first but a successful response on retry. Logistics Dev teams have already checked the Shipping API but were unable to find any problems. We’re currently reviewing the logs before we page additional teams.