Skip to main content

Troubleshooting Asset Outbox and Messaging

Summary​

This page covers the asset outbox and messaging relay. Outbox rows move through Pending, Processing, Processed, and Failed states. The relay polls every 10 seconds in batches of 50. Failed rows are retried indefinitely, with no ceiling, no backoff, no dead-letter queue, and no cleanup. A read-only diagnostics endpoint exposes row metadata for a requested number of rows. A pending backlog usually indicates the broker is unavailable. This page never instructs replay or row edits; those are out of scope.

Audience​

Support engineers, developers, and DevOps engineers investigating a growing pending backlog, rows stuck in Failed, or events that did not reach downstream consumers.

Symptoms​

  • The pending count grows and does not drain.
  • Rows remain in Failed and are retried repeatedly.
  • A downstream consumer did not receive an expected event.
  • Row states are unclear during an incident.

Cause​

Scope​

This page covers outbox row states, relay cadence, and read-only diagnostics. Downstream notification behavior is covered on the notifications page.

Likely causes​

  • The broker is unavailable, so Pending rows accumulate and do not progress.
  • A Failed row is being retried indefinitely because there is no ceiling, backoff, or dead-letter queue.
  • The relay cadence of 10 seconds in batches of 50 means large bursts drain gradually, which can look like a stall.
  • The event was written but not yet relayed at the moment of observation.

Resolution​

Safe diagnostics​

  • Read the outbox diagnostics endpoint to observe row metadata and state distribution; this is read-only.
  • Correlate a pending backlog with broker availability, which is the usual cause.
  • Account for the 10-second, batch-of-50 cadence when judging whether the backlog is draining.
  • Do not instruct replay, row edits, or cleanup; there is no cleanup mechanism and edits are out of scope.

Validation​

  • Confirm the pending count drains once the broker is available.
  • Confirm rows progress from Pending through Processing to Processed.
  • Confirm the diagnostics endpoint reflects the expected event rows for the operation under investigation.

Evidence to Collect​

  • The row state distribution from the diagnostics endpoint.
  • Whether the broker was available during the window.
  • The specific Asset event rows involved.
  • A correlation identifier and timestamp.

Escalation​

Route broker availability and delivery to the messaging owner. Route relay behavior and retry semantics to the asset or backend team. Any change to rows or replay must be handled by the owning team under change control.

Source References​

  • microservices/src/asset-service/Messaging/OutboxRelayHostedService.cs
  • microservices/src/asset-service/Api/AssetEndpoints.cs
  • microservices/scripts/smoke-asset.ps1

See Also​

Keywords​

outbox, relay, pending backlog, broker, failed rows

Revision Information​

  • Status: Draft
  • Last reviewed: 2026-07-17
  • Review cycle: Quarterly