Incident Resolved: CTIM-972
Contain
Please contact CTIM or see ticket for more details.
UPDATE:
Incident Remediation complete. CTIM to prepare a Summary to be shared Monday.
Incident Resolved: CTIM-972
The Prepress team has been collaborating with GSO agents who were confused about the Designer being disabled in their tool. This highlights that agents were not adhering to the SOP (as evidenced by the impact), given that Designer should not have been used for these items even before it was disabled. Team is addressing this gap internally but nothing is currently blocked. For now the incident is contained.
Please contact CTIM or see ticket for more details.
UPDATE:
CARE communication has been sent with details of the issue and the remediation process for shipped items across all locales, except the IN locale. Most of the items have been addressed, and the remaining items are expected to be completed by tomorrow morning. For the IN locale, a list has been forwarded to the local care team for further action.
UPDATE:
Remediation efforts continue, teams have met and aligned on further actions. Next update tomorrow morning IST when CTIM team is back online for non-urgent incidents. Please page if there is anything urgent for us to contribute to the remediation.
UPDATE:
Remediation efforts continue. The final list of shipped orders has been compiled. A version of that list including any related actions by CARE is in progress. Non-shipped items are all fixed. Existing customer projects are also being updated to use the correct version of the document should the customer reopen that project again.
UPDATE:
The Prepress Review Apps team has deployed the change for the Premium Finish App as well. Now they are working on the scaling property for all the apps and embroidery review application. We will update here once its done.
UPDATE:
The team has deployed the change (disabling the loading of the Designer when text paths or resizing options are present in text fields of incoming Cimdocs) in the following apps:
Design Review App
Substrate Review App
Aux Code App
QC App
Arrow Design App
Calendar Review
VP Digital App
We are now awaiting the remediation of the remaining items which is expected to be completed within the next 4 to 5 hours to close this incident.
UPDATE:
The Vista Order Management team has remediated another batch of orders, which included approximately 711 unshipped items. In total, across both batches, there are still about 499 remaining cases where the document URL could not be updated. To resolve these, the team needs to implement a codebase change. Since the changes require testing and approval, we will need to wait until next business hours. The incident will remain open until all the items have been fully remediated.
UPDATE:
The Vista Order Management team has remediated 2,374 items out of 3,070. Of the remaining items, 322 have been shipped and require support from the Care team, while 374 are pending due to some validations in place. The team will address the remaining items during business hours. This incident will remain open until all items have been remediated.
UPDATE:
The team is continuing to remediate the affected items that have not yet been shipped.
UPDATE:
Teams continue to investigate remediation options. Testing of bulk-resubmission of orders has proven effective and teams are now working on bulk-resubmission of all affected items that have not yet been shipped. Affected items that have been shipped CAN also be remediated, however this is a business question (due to the CX and Cost associated) for the CARE and/or the consumer team.
UPDATE:
We’re reopening this incident due to the scope of the remediation.
UPDATE:
Artwork teams continue to work on defining which orders need remediating with priority. Next update when they are finished or in 2 hours.
UPDATE:
The incident is still contained for any new orders, but teams are still working on putting together a complete list of items that need remediated. Current theory is this will be a large list and CiM will be needed to help. Next update when final list of orders for remediation is complete or in 1 hour.
UPDATE:
Changes to Design Review Application are completed and there should no longer be effects on new orders. The Artwork Integration team is still working on a remediation process for previously affected items. Next update when that work is complete or in 2 hours.
UPDATE:
Artwork teamsis still actively working on two actions outlined to resolve the issue with Prepress review application affecting rich text. Next update when they have completed testing and deployed changes or in 2hours.
UPDATE:
Artwork teams have a plan to implement a fix to prevent this issue from occurring that will be tested and deployed by end of day tomorrow (IST). They also plan to gather and compare the data of edited docs in designer (from the past week) to find affected orders and proactively resubmit those items for the customer. We will keep this incident open until the fix is deployed and send another update at that time.
UPDATE:
The impact for this issue is determined to be a subset of orders where GSO agents have edited designs on orders that originally contained the rich text. In these cases, since the agent tools do not yet support rich text, the content is altered. Artwork dev teams have been working on a permanent fix while also identifying possible methods of immediate mitigation. Since the impact is limited, the actual fix will be during IST business hours tomorrow.
New Incident: CTIM-972
Priority: Critical
Escalation sent to: Artwork Tech: Support for review.
Vista customers have reported receiving products using curved rich text that were printed incorrectly. Artwork teams are already engaged to discuss the problem. The issue had been identified previously and a fix in progress but hasn’t been deployed since there were problems during testing.