Notification Operational Runbooks
Summary
Procedures below use only capabilities the reviewed source provides. No operational command, host or credential is reproduced. Several common operational needs have no supported procedure and are recorded as such.
Audience
DevOps engineers, support engineers, developers and QA.
Runtime operations
Verify the service is consuming
The health endpoint cannot answer this. Confirm the consumer's bound-to-queue information log line has been emitted since the last restart; it lists the subscribed event types. If the connect-failure warning is recurring instead, consumption has stopped.
A positive functional confirmation is to run the smoke script in broker mode, which publishes an event and polls for its processed row.
Investigate a notification that did not arrive
Query the audit trail filtered by event identifier, or by correlation identifier when the publisher supplied one. The recorded outcome names the cause. Follow the decision path in troubleshooting.
If the correlation filter returns nothing, remember that an event published without a correlation identifier is persisted with a null correlation and must be found by event identifier.
Re-drive a failed delivery
Use the retry action on the Notification Center's failed list, or the retry endpoint directly. Two things to expect, both verified:
- The operation is synchronous despite the console's toast saying the retry was queued.
- The console does not refresh afterwards, so the row remains listed until reloaded manually.
There is no supported bulk re-drive. Draining a large failed queue means repeating the single-message action.
Change notification content
Decide which implementation produces the notification first — they are separate stores with separate content.
For the legacy path, use the template administration screen: select the event key, edit, preview, save. A reset to the built-in default is available behind a confirmation.
For the extracted service there is no UI. Creating a new version or updating a row is possible only through the template API. Note that create produces a new version while update rewrites the existing row in place.
Change channels for an event
The per-event channel override map replaces the resolver default for a named event type. It is bound at startup, so a change requires a restart to take effect. There is no runtime reload and no UI.
Recover from a broker outage
No operator action is required to reconnect: the consumer retries indefinitely on a fixed interval and resumes when the broker returns. Messages published during the outage remain on the durable queue and are delivered on reconnection.
Two consequences to plan for. First, a backlog is processed at full prefetch on reconnection. Second, the staleness window applies — events older than the configured window are recorded as stale and skipped, so an outage longer than that window silently drops the oldest events. That window is the operational limit on tolerable outage length.
Verify end to end
The smoke script accepts a base URL, an optional connection string override, a mode selecting the direct or broker path, and optional gateway parameters. Direct mode requires no broker; broker mode does.
Operations with no supported procedure
| Need | Status |
|---|---|
| Bulk re-drive of the failed queue | Not implemented |
| Park or drain a poison message | Not implemented — no dead-letter destination and requeue is unbounded |
| Purge or archive old notifications, audit or ledger rows | Not implemented — no delete path exists anywhere |
| Manage recipient preferences | Not implemented at every layer |
| Alert on failure or on the consumer disconnecting | Not implemented |
| Query failures by tenant, channel or time window | Not implemented |
| Change a channel provider without a restart | Not implemented |
Classification
Partial — the listed procedures are source-backed; several routine operational needs have no supported path.
Requires confirmation
Escalation ownership, maintenance windows and whether the staleness window is tuned to the tolerable outage duration require confirmation.
Related Articles
See Also
Keywords
- Notification database
- Operational Runbooks
- Draft database documentation
Source References
microservices/src/notification-service/Api/NotificationEndpoints.csmicroservices/src/notification-service/Messaging/NotificationEventConsumer.csmicroservices/src/notification-service/Application/NotificationOptions.csUI/salary-ui/apps/client-hrms-portal/src/pages/admin/NotificationTemplateManagementPage.tsxUI/salary-ui/apps/client-hrms-portal/src/pages/admin/OperationsAdminPages.tsxmicroservices/scripts/smoke-notification.ps1microservices/src/notification-service/Application/NotificationDispatcher.cs
Revision Information
- Status: Draft
- Last reviewed: 2026-07-21
- Review cycle: Quarterly