Skip to main content

Workflow Event Retries & Failures

Audience​

Developers, QA, support, operations and service owners.

Summary​

When publication returns failure or throws, the processor increments the retry count, records the error and marks the outbox record Failed. Pending and Failed records remain eligible for later relay iterations.

Operational behavior​

The relay runs repeatedly in a hosted service and isolates iteration failures with warning logging. Each record is saved after its publication outcome. No verified maximum attempt count, exponential backoff, poison-message quarantine, dead-letter workflow or operator replay endpoint exists in this path.

A crash after broker publication but before the Processed update may cause a later duplicate. A permanently failing record remains eligible indefinitely.

Symptoms​

An eligible outbox record remains Failed, its retry count grows, or downstream effects are missing or duplicated.

Cause​

Publication returned failure or threw, the Processed status save failed after publication, or a permanently failing record has no terminal retry policy.

Resolution​

Inspect the stored error and relay warning, confirm transport availability and consumer ownership, and use only an approved recovery or replay procedure. No Workflow-specific operator replay endpoint is verified.

Validation​

Confirm that the record reaches Processed and that any downstream effect occurs once from the consumer's perspective. Retain the event identifier when checking for duplicate effects.

Requires Confirmation​

Retry budgets, backoff, alert thresholds, dead-letter ownership, manual recovery and duplicate tolerance require confirmation before production reliance.

Source References​

  • microservices/src/shared-kernel/Messaging/OutboxProcessor.cs
  • microservices/src/workflow-service/Infrastructure/Persistence.cs
  • microservices/src/workflow-service/Messaging/OutboxRelayHostedService.cs

See Also​

Keywords​

Workflow events, retry, failure recovery.

Revision Information​

  • Status: Draft
  • Last reviewed: 2026-07-20
  • Next review: 2026-10-20