Incident Resolved: CTIM-1853
Support Teams have contained or resolved the issue. Please contact MIS PBM or see ticket for more details.
UPDATE:
Currently waiting on confirmation of the data needed for remediation. We will update again once we can verify the customer care teams are able to move forward with the process.
UPDATE:
Support team has shared the list with Windsor team. Next update after we gear back from support team.
UPDATE:
Due to the significant amount of manual effort required to generate an accurate impact list, it’s not expected to be completed until Monday. We will follow up with the dev team during their next business hour to confirm the ETA.
UPDATE:
Latest update from the support team is that progress is being made on the new list but will require another couple hours due to the manual work involved. They expect to be near completion by 4pm ET and we will provide another update at that time.
UPDATE:
New version of the impact list is still in-progress (for items to be proactively reprinted). Support teams are currently evaluating the output of items that have been caught in production and placed on-hold (then re-rendered) as the problem seems to persist for some items.
Next update in 2 hours as the various support teams work through the ongoing remediation process.
UPDATE:
Re-rendering has been confirmed to fix the impacted items. Next steps are to reprint items currently on hold and proactively re-render any affected items once identified. Team is working on preparing a list of impacted items (VP Item IDs) within the next hour so plants can take action. Next update in 1 hour.
UPDATE:
The team continues to work on the script to get the list of impacted items. Team has asked for an extended time of 2 more hours to complete it. Next update in 2 hours from now on.
UPDATE:
The team started working on the script during the early EST morning hours and provided an ETA of 2 hours. CTIM will follow up in 2 hours and provide an update here.
UPDATE:
As per the latest update, both parties have agreed to reconnect today around 5:30 PM IST to review the list again. CTIM will follow accordingly.
UPDATE:
The list is in the final review stage and will be shared once completed. This will then require remediation efforts with help from Vista CARE teams which we’ll begin tomorrow AM EST. Next update will come once we have the list and are ready to engage additional teams for reorders.
UPDATE:
Artwork teams continue to actively work the issue. The impacted items list is being narrowed down through a custom script which is still processing. We will send another update by 6pm ET or sooner once the script has completed.
UPDATE:
Impacted order list generation is still in-progress by Artwork teams. They have indicated it will take a few more hours to complete. We will check in and update again at 4pm ET (1.5 hours).
UPDATE:
Some of the impacted orders may still be in the CDN and will require another 8 hours to refresh. The next update will be shared at 5 PM EST.
UPDATE:
Remediation of the shared impacted orders has been completed. The team is now checking if there are any remaining affected orders that still require remediation. Next update in max 1 hour
UPDATE:
Artwork team is still working on remediating the affected items at the highest priority, which is expected to take a few more hours. CTIM will follow up in 2 hours and share the findings.
UPDATE:
Artwork teams are currently working on the remediation effort of the affected items at highest priority. Next update in max 3 hours.
UPDATE:
We’re currently awaiting completion of the regenerated assets from Artwork teams. There will be some additional time needed to filter the impact lists by business unit and send those items to the appropriate teams for processing. CTIM will follow up with the dev squads tomorrow AM (IST, ~1-2 hours) to begin remediation of the affected orders.
UPDATE:
A list of affected items has been created and Artwork teams are working to regenerate the assets for those orders. Once that process is complete, we will need to filter the list for Vista-specific items to be remediated. Next update in 1 hour.
UPDATE:
Since the issue had a wide impact, additional time will be needed to generate the list of affected orders. We will continue to provide updates and share the list asap.
UPDATE:
This issue is related to a previous incident that has had a fix applied so that new orders should not be affected. Previous order remediation is still in-progress. We will wait to contain this instance of the problem until we confirm the next steps for order remediation.
UPDATE:
Prepress teams are currently reviewing the issue. Next update in 30 minutes.
New Incident: CTIM-1853
Plant: WND
Priority: Critical
Escalation sent to: Artwork Tech: Support for review.
Production in WND has reported black lines on rendered docs in multiple workflows. The symptoms are similar to the ongoing issue previously reported by Pixart. Artwork teams have already been working the issue.