Integration Patterns

Integration is where most distributed system failures are born. A synchronous call chain — Service A calls B, which calls C, which calls D — means that a 100ms delay in D becomes a 400ms delay in A, and a failure in D cascades to take down A, B, and C simultaneously. This is the core problem that integration patterns exist to solve.

Synchronous vs. Asynchronous is the fundamental integration choice. Synchronous (REST, gRPC) is appropriate when the caller needs an immediate answer — a payment authorization, a balance inquiry. Asynchronous (Kafka, SQS, RabbitMQ) is appropriate when the caller does not need an immediate answer — sending a confirmation email, updating a search index, notifying a downstream analytics system. The mistake most teams make is defaulting to synchronous everywhere because it feels simpler. It is only simpler until you need to trace why your p99 latency is 8 seconds.

The Saga Pattern solves the hardest integration problem: distributed transactions. In a monolith, you use a database transaction — either everything commits or nothing does. In a microservices architecture, you have multiple independent databases. If Order Service creates an order and Payment Service charges the card and Inventory Service reserves stock — what happens when Inventory fails after Payment succeeds? Sagas address this via compensating transactions. Each step publishes an event; if a later step fails, compensating events reverse the earlier steps. Uber uses saga orchestration for its trip lifecycle across 20+ services.

The Outbox Pattern solves a subtle but critical problem: how do you guarantee that when you save a record to your database AND publish an event to Kafka, both happen atomically? If the Kafka publish fails after the database commit, your event is lost. The Outbox Pattern writes the event to an outbox table in the same database transaction as the business record, then a separate process (Debezium, a polling publisher) reads the outbox and publishes to Kafka. Exactly-once semantics achieved without distributed transactions.

Event-Carried State Transfer is a pattern where events carry the full data payload needed by consumers — not just an ID. When an Order event carries customer name, shipping address, and line items, consumers can act without calling back to the Order service. This eliminates synchronous dependency chains between services and enables consumers to operate independently even when the Order service is down.

The Claim Check Pattern solves message bus performance: if you're streaming large payloads (PDFs, images, large JSON blobs) through a message bus, you'll saturate it quickly. Instead, store the payload in object storage (S3, Blob Storage) and put a reference token (the "claim check") in the message. Consumers redeem the claim check to fetch the actual payload. The message bus stays lean and fast.

Idempotency is the invisible glue that makes all of these patterns safe. At-least-once delivery means consumers will occasionally receive the same message twice. Every consumer must be designed to process duplicate messages without side effects — by checking a processed-event-ID table, using database upserts with deterministic keys, or by making the operation naturally idempotent (setting a status to 'confirmed' is idempotent; incrementing a counter is not).