Seeing a spike in DSS errors coming into Studio6

Postmortem feb 26, 2024 6:49 AM EST

The technical teams have completed their root cause and corrective action analysis. The post mortem can be view [here https://cimpress-support.atlassian.net/wiki/spaces/DCR/pages/15905787127/Feb+19+2024+Seeing+a+spike+in+DSS+errors+coming+into+Studio6

Resuelto feb 19, 2024 4:20 PM EST

It has been an hour since this incident was contained, and we have not seen a recurrence. We are marking this incident as resolved. Within five business days, we will post a link the the post mortem of this incident here.

Supervisando solución feb 19, 2024 4:00 PM EST

We have redirected all current traffic to a backup server, so everything should be functioning properly for the most part.

Problema identificado feb 19, 2024 3:47 PM EST

We have currently enabled a failover to a different server as deployment may take some time, there might be a little bit of latency though, but we think the error rate should go down in the coming 2-3 minutes.

Problema identificado feb 19, 2024 3:46 PM EST

We found that there are around 30% of requests are failing currently, We have looked through logs and found that there is a client secret rotator used to rotate secrets which was setup recently and it seems it was rotated yesterday but the token was cached till today, We have made some changes now and it should be deployed in about sometime, will update here once it is done.

Investigando feb 19, 2024 3:29 PM EST

The technical team is still investigating this issue. We will update this status page again as soon as we have more information or in 30 minutes, whichever is sooner.

Investigando feb 19, 2024 2:54 PM EST

we are seeing a spike in DSS errors coming into Studio6. We’re seeing an error while getting design document 401s. The technical team is investigating. We will update this status page again as soon as we have more information or in 30 minutes, whichever is sooner.