National Pen

UPDATE:

A fix has been deployed to handle the issue moving forward. If there are stuck batches that show the same symptoms (unknown error returned to user when scanning the last step), there is a process in-place and those batches should be sent to the support team in the incident channel for remediation. We will keep this issue open for the next hour as production works through the backlog to see if it reoccurs.

UPDATE:

We currently have a workaround in place. Support teams will be able to unblock the machines and allow the Paperless teams more time to investigate the root cause. Next update in 1 hour.

UPDATE:

No additional updates at this time. Support teams continue to work the problem. Next update in 1 hour.

UPDATE:

Despite rolling back the previous deployment, the errors are still occurring. The support teams are looking at the possible causes for the issue on specific machines. We will update again in 30min or sooner if a fix is confirmed.

UPDATE:

Production confirms 7 machines currently down due to the issue. Support teams have been working with local IT to gather more details from the local workstation browser logs. Next update in 30min.

UPDATE:

The issue is ongoing. The service’s logs are not showing any timeouts, though production is still experiencing an ‘unknown error’ when scanning at certain machines. Next update in 1 hour or sooner.

UPDATE:

Production reports the issue is still occurring in multiple workflows despite an update made to the service to mitigate the issue. Support teams are investigating at highest priority. Next update in 30min or sooner if more details become available.

UPDATE:

Support teams are still reviewing the issue, which was first reported yesterday morning at 10:23am ET (May 26) but has had extended periods of stability and then reoccurs.

UPDATE:

Fulfiller has reported that two machines are now showing the error and are unable to scan batches. The support teams are currently investigating

UPDATE:

Team see that the batch is being scanned and is already in Production Completed state. Awaiting response from the fulfiller on this which may be expected later in the IST evening hours. Incident will be contained when we get confirmation from the fulfiller

UPDATE:

The right team is online and investigating the issue of batch scanning issue with highest priority. Next update in max 1 hour.

UPDATE:

Escalation has been sent to the appropriate team to investigate during their business hours and we are currently awaiting their response. CTIM will follow up in the next 1 hour and share an update unless an urgent escalation requires before then.