Reprocessing Methods for Integration Workflows

When an integration workflow halts because of an error, such as an authentication failure or an unavailable target system, events that fail or arrive during the halt can be held instead of discarded. Held events still represent real activity that has to reach the connected system, so leaving them unprocessed creates missing records and data gaps between systems.

Reprocessing methods let you recover these events once the cause of the halt is fixed, so the workflow can catch up without anyone re-entering data or re-triggering the original activity. Which method to use depends on whether the events are still valid, how many are affected, and whether they need individual review.

Note: This functionality is also referred to as the Dead Letter Queue (DLQ).

General Availability vs Public Preview Experiences

In the existing experience, Integration's Message Container shows only a count of events held, and the only actions are to reprocess or purge everything at once. Failed and pending events are held together, so a purge can remove pending events that still need to be processed, and nothing confirms what was affected.

For more information, see Message Container Existing Experience.

The OneTrust 202611.1.0 (Fall Release) Public Preview of Retry Queue Management identifies which events in the integration logs are in the Retry Queue, lets you select specific events to reprocess or purge, and confirms what changed after each action. This allows admins to recover failed events without losing pending ones.

For more information, see Retry Queue Public Preview Experience.

To enable the Message Container

  1. On the Integrations menu, click Integrations.

    The Integrations screen appears.

  2. Click on the name of an integration.

    The Integration Builder screen appears.

  3. Navigate to the Details tab.
  4. Provide a user email in the Notification Email field.
  5. Enable the Enable message container setting.


    The Default procedure for failed and pending messages setting field will appear.
  6. You have 2 options for reprocessing messages: Manual or Automatic.

Reprocessing Options

Manual Reprocessing

You can enable the workflow to require a manual reprocess or purge of messages from the Message Container. This procedure option is also the default selection for any workflows created via a Gallery Wizard.

Automatic Reprocessing

You can enable the workflow to automatically reprocess messages from the Message Container when it self-heals from a Halted state to an Active state. If a workflow is in a Halted state, OneTrust will periodically ping the workflow to check if it runs successfully. If the workflow is successful for three continuous messages, it will move from the Halted state back to an Active state.

About Long-Term Automatic Reprocessing

When 5 out of 10 messages fail, the workflow will halt and the admin is alerted via email. All failed and pending messages are preserved for re-processing in the Message Container (if enabled for the workflow).

In app-prod environments, when a failed message is received, we will attempt re-processing once every 15 minutes while the workflow is active. The following outcomes can result from this action:

  • Messages that have been attempted 15 times will be dropped from the Message Container and marked as expired.

  • For all other non-production environments, the scheduled frequency is 5 minutes and up to 5 attempts.

  • When workflows move from halted to an active state, failed and pending messages stored will be re-attempted.

Resuming vs. Deactivating Workflows

Once you have addressed halting errors in the logs, OneTrust recommends to resume the workflow instead of deactivating and reactivating.

Deactivation is recommended when a workflow is experiencing such persistent configuration issues where deactivation is necessary for more deliberate troubleshooting.

Message Container Existing Experience

When messages are held in the Message Container because they are failed or pending, you can choose to either reprocess or purge them before resuming the workflow. This article is intended to provide guidance to the possible user journey scenarios for how OneTrust customers might set up and manage the manual or automatic reprocessing methods for failed messages.

To manage reprocessing methods

You can opt to reprocess or purge messages for workflows that are held in a halted state.

  1. On the Integration Builder screen of a halted workflow, hover the Needs Attention icon to reveal the Needs Attention popover.
  2. The popover will state how many message are currently held in the container.

    Click View Details. The Message Container Details modal appears.



  3. Select either to reprocess messages or purge messages from the logs.
    Note:

    By default all failed or pending messages are saved to the Message Container.



  4. Click the Save button.

To reprocess individual messages

You can reprocess a single message from the failed logs to restart the integration. If you attempt to reprocess a queued message, the initial message can still process, which might produce duplicate results.

Reprocessing is not supported for workflows in the following statuses:

  • Success

  • Duplicate

  1. On the Integrations menu, click Integrations.

    The Integrations screen appears.

  2. Click on the name of an existing integration.

    The Integration Builder screen appears.

  3. Navigate to the Logs tab.
  4. Hover over the row of an event ID to reveal the Context Menu icon.
  5. Click the Context Menu, then select Reprocess Message.


    The Reprocess Message modal appears.

  6. Read the confirmation warning, then click the Confirm button. The individual message will be reprocessed.


Scenario 1: Save Messages in Queue

  1. On the Integrations menu, click Integrations. The Integrations screen appears.

  2. Click on the name of an existing integration. The Integration Builder screen appears.

  3. Navigate to the Details tab.

  4. Enable the Enable Hold and Reprocess Message when Workflow Halted setting. The Email Error Alerts and Default procedure for failed messages settings will appear.

  5. Provide an email for alerts.

  6. Change the default procedure for failed message to either:

    • Manually reprocess when workflow is resumed

    • Automatically reprocess when workflow self-heals

  7. Click the Save button.

  8. Move the workflow into a Halted state. Messages will save to Message Container whenever a message fails, and the designated user should receive the alert email.

  9. On the Integration Builder screen of a halted workflow, click the Needs Attention icon to reveal the Needs Attention popover.

  10. Click the View Details button. The Message Container Details modal appears.

  11. Here you can select to reprocess or purge the messages. This option can also be accessed from the Context Menu of the Integration Builder screen as you resume the workflow.

    Note:

    Reprocessing messages from the Message Container can take time, and you may not see immediate progress. To avoid message loss do not deactivate a workflow. Instead, use the methods described in step 11 instead.

Scenario 2: Select Purge Option

  1. On the Integrations menu, click Integrations. The Integrations screen appears.

  2. Click on the name of an integration. The Integration Builder screen appears.

  3. Navigate to the Details tab.

  4. Enable the Enable Message Container setting.

  5. Provide an email for alerts.

  6. Change the default procedure for failed message to either:

    • Manually reprocess when workflow is resumed

    • Automatically reprocess when workflow self-heals

  7. Click the Save button.

  8. Whenever the workflow next moves into a halted state, the designated user should receive the alert email.

  9. On the Integration Builder screen of a halted workflow, click the Needs Attention icon to reveal the Needs Attention popover.

  10. Click the View Details button. The Message Container Details modal appears.

  11. Select Purge Messages. The warning message will be erased. When the workflow is resumed, the module will not ask to reprocess or purge as the messages are not reprocessed.

Scenario 3: Do Not Save Messages in Queue

  1. On the Integrations menu, click Integrations. The Integrations screen appears.

  2. Click on the name of an integration. The Integration Builder screen appears.

  3. Navigate to the Details tab.

  4. Disable the Enable Message Container setting.

  5. Click the Save button.

  6. When the workflow halts, the Needs Attention popover warning should not appear.

  7. Resume the workflow.

  8. The workflow will resume. Messages will not automatically reprocess but individual failed messages can be manually retried. For more information, see To reprocess individual messages.

Retry Queue Public Preview Experience

Note: The Retry Queue feature is under Public Preview. Please contact OneTrust Support to request early access.

Retry Queue Handling Scenarios

Option 1: Resume the workflow, then review events

Use this option when the workflow supports time-sensitive activity and the held events can be sorted afterward.

Example: A workflow halts after 10 events. Five failed because of a credential error and five are pending. After the credential is fixed, resume the workflow, reprocess the five failed events, and leave the five pending events to run normally.

  1. Resolve the cause of the halt, such as an expired credential or an incorrect mapping.

  2. Resume the workflow

  3. Open the integrations logs and locate the events in the Retry Queue

  4. Select the eventes to reprocess, and then select Reprocess

  5. Select the events that should not run, and then select Purge.

  6. Review the confirmation to verify which events werea affected.

Option 2: Clear the queue, then resume workflow

Use this option when running the held events could cause incorrect or duplicate results in the connected system.

Example: A workflow is halted for several days. The held events describe changes that were later made manually in the target system. Purge the events before resuming so they do not overwrite current data.

  1. Resolve the cause of the halt.

  2. Open the Retry Queue before resuming

  3. Reprocess or purge all events, or a selected subset

  4. Review the confirmation ,and then resume the workflow.

Option Comparison

Evaluation

Resume first, then review events

Clear queue first, then resume

Best when

The cause of the halt is fixed, the queue contains both failed and pending events, and the workflow needs to start processing new events quickly.

The events are no longer current, the queue is large, or every event failed for the same reason.

Main benefit

New events resume right away while you handle held events at your own pace.

The workflow restarts with an empty queue, so old events cannot run alongside new ones.

Main risk

Reprocessed events may run after newer events

The workflow stays halted until the queue is cleared.

To inspect events in halted workflows

Users can identify halted workflows through configured email notifications or by checking statuses on the Integrations screen.

  1. On the Integrations menu, click Integrations.

    The Integrations screen appears.

  2. Select the halted workflow by searching or filtering the Status column for Halted.
    The Integration builder screen appears.
  3. An alert banner will appear across the top of the screen indicating the halted status.
    The Review events in the Retry Queue modal appears.
  4. The modal will state the number of events in the Retry Queue. Select one of the following options:

    Review Options

    Description

    Review events

    This option allows you to intentionally review the events and take action on them individually or in bulk before the integration is resumed.

    Resume and reprocess all

    This option resumes the integration and reprocesses all events after you have resolved the issue(s) that caused the integration to initially halt.

    Resume and purge all

    This option purges the events and resumes the integration.
  5. After you have chosen your option, click the Continue button.

To review events before resuming

This option is accessible when selecting the Review events option from the alert banner. The Retry Queue can be also accessed using the Review events option or from the Logs tab in the Integration builder screen.

  1. In the Logs tab, the table view will default to Retry queue.
  2. Select one or more events to display the management options:

    • Resume and reprocess

    • Resume and purge

  3. To bulk resume and reprocess or resume and purge for all messages, select the Context menu icon next to the buttons and make a selection. This bulk operation will override the number of selected messages and proceed with the operation for every message in this table.
  4. The other method of reprocessing a single message is through the Reprocess message option available in the Context menu of the given event row.
    The Reprocess message modal appears.
  5. The modal will declare the next steps, outcomes, and expectations for reprocess the message.
  6. Select the Confirm button to process with reprocessing.
  7. The status of the event will move into Queued then Pending.

To resume and reprocess all events

This option assumes that you have chosen to resume the integration before reviewing events for reprocessing. Resumed integrations will have this alert banner.

  1. On the Logs tab, select the Review events option from the alert banner.
    The Handle remaining events modal appears.
  2. The modal will state the number of events in the Retry Queue. Select Reprocess all.
  3. Select the Continue button.

To purge events and resume

This option assumes that you have chosen to resume the integration before reviewing events for purging. Resumed integrations will have this alert banner.

  1. On the Logs tab, select the Review events option from the alert banner.
    The Handle remaining events modal appears.
  2. The modal will state the number of events in the Retry Queue. Select Purge all.
  3. Select the Continue button.