Incident Resolved: CTIM-1608
The error rate has started increasing again but investigation has confirmed that the issue lies with the Logo Maker. For now, the incident is contained. A separate incident will be created if a new issue arises that requires involvement from the CT team.
Please contact CTIM or see ticket for more details.
UPDATE:
The error rates have been dropping steadily since the rollback. We’re checking now with Vista teams to verify if they’ve reached an acceptable level to consider the issue contained.
No further reports of problems in the past ~8 hours. Next update once we’ve received confirmation and can close this issue.
UPDATE:
The team rolled back the changes an hour ago and v2 URLs should now appear in Logomaker. We are currently awaiting confirmation from the customer.
UPDATE:
Team has identified a possible correlation between the issue and the recent Logomaker deployment involving the update from Sherbert endpoints. The team has agreed to proceed with a rollback which is expected to take less than an hour. This will help us confirm if the deployment is linked to the issue.
In parallel team is working on reproducing the issue in a non-prod environment to support a longer term fix. Next update in max 1 hour.
UPDATE:
The team continues to actively investigate the spike in API timeouts for the endpoint issue. They are currently reaching out to the Logomaker team to gather additional details that may help identify the root cause. The next update will be provided in max 1 hour.
UPDATE:
The team is actively investigating the spike in API timeouts for the endpoint. The next update will be provided within 1 hour.
UPDATE:
An escalation has been sent to the relevant team to investigate this with priority during their business hours. The current impact remains unchanged with no further increase. We will provide the next update once we hear back from them, unless there is an urgent need to escalate further.
New Incident: CTIM-1608
Priority: High
Escalation sent to: Artwork Tech: Artwork Processing Support for review.
Since 5AM, 21 July (UTC), there has been an increasing trend of API Timeout which is causing missing previews for graphics in Logomaker Studio.