| 1 | The Problem It Solves | 2 | Pattern Structure |
| 3 | When to Use | 4 | When Not to Use |
| 5 | Trade-offs | 6 | Implementation Approach |
| 7 | Anti-Patterns to Avoid | 8 | Cloud-Specific Implementations |
| 9 | References |
The Problem It Solves
A service that updates its database and then publishes an event to a message broker has a dual-write problem. If the database commits but the event publish fails, downstream consumers never hear about the change. If the event publishes but the database rollbacks, consumers process an event for a change that never happened. There is no atomic operation that spans a relational database and a message broker.
Pattern Structure
%%{init:{'theme':'base','themeVariables':{'fontSize':'14px','fontFamily':'IBM Plex Sans, system-ui, sans-serif','primaryColor':'#DBEAFE','primaryTextColor':'#1e3a5f','primaryBorderColor':'#2563EB','lineColor':'#374151','clusterBkg':'#F9FAFB','clusterBorder':'#D1D5DB','edgeLabelBackground':'#FFFFFF'},'flowchart':{'curve':'orthogonal','padding':30,'nodeSpacing':65,'rankSpacing':75,'useMaxWidth':true}}}%% flowchart TD START([Business Operation Requested]) START --> TX[Single Database Transaction\nUpdate business table\nInsert event into outbox table\nCommit — both or neither] TX --> RELAY{Message Relay\nPolling or CDC} RELAY -->|Polling publisher| POLL[Query outbox table\nWhere published = false\nBatch of unprocessed events] RELAY -->|Log tailing with CDC| CDC[Change Data Capture\nDebezium or DynamoDB Streams\nReads database commit log] POLL & CDC --> PUBLISH[Publish event to broker\nKafka, SQS, EventBridge\nAt-least-once delivery] PUBLISH --> MARK{Publish\nSucceeded?} MARK -->|Yes| DELETE[Mark outbox record published\nor delete from table] MARK -->|No| RETRY[Retry with backoff\nEvent remains in outbox\nPublished when broker available] DELETE --> DONE([Event Delivered — At-Least-Once]) RETRY --> PUBLISH style START fill:#4f8ef7,color:#fff style DONE fill:#10b981,color:#fff style RETRY fill:#fef3c7 style CDC fill:#e0f2fe
When to Use
- Any service that must update a database and publish a message in the same logical operation
- Microservices where losing an event causes observable data inconsistency across the system
- Systems that cannot tolerate the eventual consistency window of a retry-only approach
- Event-driven architectures where the event stream must be a faithful reflection of database state
When Not to Use
- Simple use cases where idempotent retry is sufficient and occasional duplicate processing is acceptable
- Services that only publish events and never update a local database in the same operation
- Greenfield systems where you can design the architecture to avoid the dual-write problem from the start
Trade-offs
| Benefit | Cost |
|---|---|
| Database commit is the single source of truth | Outbox table requires maintenance and cleanup |
| No distributed transaction required | At-least-once delivery — consumers must be idempotent |
| Broker unavailability does not cause data loss | Polling publisher adds database load at a fixed interval |
| Works with any relational database | CDC requires database log access and additional infrastructure |
Implementation Approach
Step 1 — Design the outbox table.
The outbox table lives in the same database as the business data. At minimum: an event ID, the event type, the event payload as JSON, a published flag, and a created-at timestamp. The event ID becomes the deduplication key for consumers.
Step 2 — Write to the outbox in the same transaction.
The service code inserts a row into the outbox table within the same database transaction as the business update. If the transaction rollbacks for any reason, the outbox row rolls back too. No orphaned events.
Step 3 — Choose a relay strategy.
Polling publisher: A background process queries the outbox table for unpublished rows on a fixed interval. Simple to implement. Adds read load to the database. Acceptable for most workloads up to a few thousand events per minute.
Change Data Capture (CDC): A tool like Debezium reads the database transaction log (binlog for MySQL, WAL for PostgreSQL) and publishes changes as events. Zero additional database load after the write. More complex infrastructure. Recommended for high-throughput systems.
Step 4 — Ensure consumer idempotency.
The relay delivers at-least-once. Network retries and relay restarts mean the same event can be published twice. Every consumer must check whether it has already processed a given event ID before acting on it.
Anti-Patterns to Avoid
Writing to the business table in one transaction, committing, and then writing the outbox row in a second transaction. The two transactions are independent. A crash between them leaves the business table updated with no outbox row — the event is silently lost.
Both writes must be in the same database transaction. The outbox row is inserted before the commit, not after.
Appending to the outbox table indefinitely without deleting or archiving published rows. Over months the table grows to millions of rows, queries slow down, and disk fills. The polling publisher degrades with table size.
Delete outbox rows after successful publishing or archive them to cold storage. Set a TTL cleanup job. Monitor outbox table row count as an operational metric.
Cloud-Specific Implementations
- AWS: DynamoDB Streams acts as a built-in outbox — every write to a DynamoDB table appears in the stream within milliseconds. A Lambda function consuming the stream publishes to EventBridge or SQS. See Event-Driven Architecture on AWS.
Flowchart
References
- Richardson, Chris — Microservices Patterns. Manning, 2018. Pattern: Transactional Outbox
- Debezium — Change Data Capture for databases. debezium.io
- AWS — DynamoDB Streams as an outbox implementation. docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams