Imagine a user places an order in an online store, and the order is successfully stored in the database:

Order #123 → Created

A few milliseconds later, your service tries to send an OrderCreated event to Kafka or RabbitMQ, and suddenly:

• The network goes down.

• The Broker becomes unavailable.

• The application crashes.

• A timeout occurs.

So what happens ?

The database says:

The order has been created.

But the other services say:

We have no idea about this order!

This is one of the classic problems in distributed systems and Microservices architecture: the Dual Write Problem.

This is where the Outbox Pattern comes in.

The Core Idea Behind the Outbox Pattern

The idea behind the Outbox Pattern is surprisingly simple.

Instead of writing to the database and Message Broker at the same time, we store the business data change and the corresponding message/event in the same database transaction.

We then delegate message publishing to a separate process.

This small change in architecture can make a significant difference to system reliability.

What Is the Outbox Pattern ?

The Outbox Pattern, or more precisely the Transactional Outbox Pattern, is a design pattern for reliably sending messages and events in distributed systems.

Instead of having the application directly publish an event to a Message Broker after modifying the database, the application first stores the event in an Outbox Table in the same database.

The most important point is this:

The business data change and the event stored in the Outbox must happen within the same database transaction.

A separate process then reads events from the Outbox and sends them to the Message Broker.

At a high level:

Why Do We Need the Outbox Pattern ?

To understand the Outbox Pattern, we first need to understand the problem it solves.

Suppose we have an Order Service that needs to perform two operations:

1. Store the order in the database.

2. Send the OrderCreated event to Kafka.

Our code might initially look like this:

BEGIN TRANSACTION
INSERT INTO orders (...)
COMMIT

publish(OrderCreated)

At first glance, this seems reasonable.

But there is a dangerous gap between COMMIT and publish().

What happens if the database commits successfully but the application crashes before publishing the event ?

Database
   │
   └── Order Created ✅

Message Broker
   │
   └── OrderCreated ❌

Now the system is in an inconsistent state:

• The internal business state has changed.

• The corresponding event was never published.

AWS also describes Transactional Outbox as a solution to the Dual Write problem: a situation where one logical operation needs to modify both a database and a messaging system, while failure in one of them can leave the systems inconsistent.

What Is the Dual Write Problem ?

A Dual Write occurs when one logical operation needs to be writtern into two independent systems.

For example:

Database
    +
Kafka

Or:

Database
    +
RabbitMQ

Or even:

Database
    +
External API

The problem is that we usually don't have a shared transaction between these two systems.

For example:

1. UPDATE database ✅
2. SEND message    ❌

Or the opposite:

1. SEND message    ✅
2. UPDATE database ❌

In the first case, the event is lost.

In the second case, an event has been published even though the corresponding business state does not actually exist.

Why Isn't a Normal Database Transaction Enough ?

You might say:

Why don't we simply put message publishing inside the transaction ?

The problem is that the database and Message Broker are two independent resources. To make the entire operation atomic, we would need some form of distributed transaction.

One classic approach is Two-Phase Commit, or 2PC.

In 2PC, a coordinator asks all participants:

Are you ready to commit your changes ?

Only when all participants agree, the coordinator instruct them to perform the final commit.

If one participant has a problem, the entire operation can be stopped.

This approach is theoretically attractive, but modern distributed architectures can introduce problems such as:

• Complexity

• Latency

• Coupling

• Failure handling

In addition, not every database or Message Broker supports distributed transactions in the same way.

The Outbox Pattern asks a different question:

Do we really need the database and Broker to participate in one transaction ?

The answer is:

No.

Instead, we can reliably record the critical operation in a local database transaction and defer message publishing until afterward.

How Does the Outbox Pattern Solve the Dual Write Problem ?

Instead of:

Database → Commit
Broker   → Publish

we use:

Database
├── Business Data
└── Outbox Event

For example:

BEGIN;
INSERT INTO orders (id, customer_id, status) VALUES (123, 456, 'CREATED');
INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload) VALUES ('event-789', 'Order', 123, 'OrderCreated', '{...}');
COMMIT;

If the transaction succeeds:

Order        ✅
Outbox Event ✅

If the transaction fails:

Order        ❌
Outbox Event ❌

The database guarantees that both changes are committed or rolled back together.

This is the key idea behind the Outbox Pattern.

What Is an Outbox Table ?

An Outbox is usually just a normal database table.

For example:

CREATE TABLE outbox (
    id              UUID PRIMARY KEY,
    aggregate_type  VARCHAR(100),
    aggregate_id    VARCHAR(100),
    event_type      VARCHAR(200),
    payload         JSONB,
    created_at      TIMESTAMP,
    processed_at    TIMESTAMP NULL
);

The actual schema can vary depending on your system's requirements.

Common fields include:

1. id

A unique identifier for the event.

This ID is extremely important for Deduplication and Idempotency.

2. aggregate_id

For example:

order_id = 123

This field can be important for maintaining event ordering in many architectures.

3. event_type

For example:

OrderCreated
OrderPaid
OrderCancelled

4. payload

The actual event data:

{
  "orderId": 123,
  "customerId": 456,
  "totalAmount": 250
}

5. created_at

The time at which the event was created.

6. processed_at

With a polling-based implementation, this field can be used to identify events that have already been processed.

How Does the Complete Outbox Flow Work ?

Let's look at the complete flow:

Step by step:

1. The Client sends a request to create an Order.

2. The Order Service starts a Transaction.

3. The Order is stored in the database.

4. The corresponding Event is stored in the Outbox.

5. The Transaction is committed.

6. An Outbox Publisher reads the event from the Outbox.

7. The Event is sent to the Message Broker.

8. Different Consumers receive the Event.

9. After successful publishing, the Outbox Event is cleaned up or marked as processed.

An important detail is that the application does not need to wait for the Broker for the original request to succeed.

This means event publishing can happen asynchronously.

A Real-World Example: Creating an Order

Suppose a user creates order 1001.

Step 1: Create the Order

POST /orders

The application starts a transaction.

Step 2: Store the Order

orders

id: 1001
status: CREATED

Step 3: Store the Event

Inside the same transaction:

outbox

id: event-abc
event_type: OrderCreated
aggregate_id: 1001

Step 4: Commit

If the commit succeeds:

Order  → persisted
Event  → persisted

Now, even if the application crashes immediately afterward, the event has not been lost.

Step 5: Publish the Event

The Publisher eventually sends:

Kafka → OrderCreated

Step 6: Consumers Process the Event

For example:

Inventory Service:

OrderCreated
      ↓
Reserve Stock

Notification Service:

OrderCreated
      ↓
Send Confirmation Email

Analytics Service:

OrderCreated
      ↓
Record Conversion

This is where the Outbox Pattern becomes especially useful for Event-Driven Architecture.

Does the Outbox Pattern Prevent Duplicate Messages ?

No.

This is one of the most important things to understand about the Outbox Pattern.

The Outbox Pattern can ensure that an event is not lost, but it does not necessarily guarantee that the event will be published exactly once.

Consider this sequence:

1. The Publisher reads the event.

2. The Publisher sends the event to Kafka.

3. Kafka successfully receives the message.

4. The Publisher crashes before marking the event as processed.

After the Publisher restarts:

Publisher
   ↓
"This event is still unprocessed"
   ↓
Publish Again

As a result, OrderCreated may reach the Consumer twice.

This behavior is normal and is associated with At-Least-Once Delivery.

AWS's Transactional Outbox guidance also discusses duplicate messages and the need for idempotent Consumers.

Idempotency: The Other Half of the Story

If you use the Outbox Pattern, you need to ask:

What happens if the same event reaches a Consumer twice ?

For example:

PaymentCompleted

Suppose the Consumer processes the event twice and each time performs:

$100 → Charge Customer

The customer could potentially be charged twice.

The Consumer therefore needs a way to detect duplicate events.

One common approach is to maintain a table such as:

processed_events

event_id
--------
event-123
event-456

Before processing:

Does event-123 exist ?

If it exists:

Ignore

If it doesn't:

Process
Insert event_id

Ideally, these operations should also happen inside a transaction.

This makes the Outbox Pattern and the Idempotent Consumer Pattern complementary solutions.

Does Outbox Mean Exactly-Once Delivery ?

This is where terminology matters.

You sometimes hear:

The Outbox Pattern provides Exactly-Once Delivery.

That statement is an oversimplification.

The Outbox Pattern by itself does not provide an exactly-once guarantee.

A more realistic architecture looks like this:

Producer
   ↓
Outbox
   ↓
At-Least-Once Publish
   ↓
Idempotent Consumer

This combination can provide much more reliable behavior in terms of the final business state.

However, if the Consumer calls an external API, guaranteeing exactly-once behavior across the entire chain becomes much more complicated.

The important point is:

Exactly-once is not simply a property of one component; it is an end-to-end property.

How Is an Outbox Event Published to the Broker ?

There are two common approaches:

1. Polling

2. Change Data Capture (CDC)

Approach 1: Polling Publisher

With polling, a worker periodically queries the Outbox table.

For example:

SELECT * FROM outbox WHERE processed_at IS NULL ORDER BY created_at LIMIT 100;

The workflow is:

Read
 ↓
Publish
 ↓
Mark as processed

The biggest advantage of polling is simplicity.

You don't need your system to work directly with the database's Transaction Log. Almost any database that supports queries can be used to build this type of architecture.

However, polling has a latency trade-off.

For example, if you poll every five seconds, an event created immediately after a polling cycle might have to wait almost five seconds before being detected.

If you poll more frequently:

100ms
50ms
10ms

you increase the load on the database.

So you have a trade-off:

Polling Frequency
       ↕
    Latency
       ↕
Database Load

Approach 2: Change Data Capture (CDC)

With Change Data Capture (CDC), instead of repeatedly querying the Outbox table, we track changes in the database's Transaction Log.

For example:

Application
     ↓
Database
     ↓
Transaction Log
     ↓
CDC Connector
     ↓
Message Broker

Different databases may use transaction logs such as:

WAL
Binlog
Redo Log

Tools such as Debezium can capture changes to an Outbox Table from the Transaction Log and transform them into events.

Debezium also provides a dedicated Outbox Event Router for this scenario.

Advantages of CDC

• Lower latency

• Less continuous polling

• Suitable for high throughput

• Better alignment with Transaction Log changes

Disadvantages of CDC

• More architectural complexity

• More infrastructure and monitoring requirements

• Dependency on database capabilities

• Higher operational complexity

Therefore, CDC is not necessarily better than polling. It is simply more appropriate for certain scales and latency requirements.

Polling or CDC: Which Should You Choose ?

If your system is still relatively new, polling can be a straightforward approach.

If event volume is high and low latency is particularly important, CDC can become a more suitable option.

The choice ultimately depends on factors such as:

• Event volume

• Latency requirements

• Database capabilities

• Operational complexity

• Infrastructure maturity

Does the Outbox Pattern Preserve Event Ordering ?

Not automatically, and not in every architecture.

Suppose we have:

OrderCreated
OrderPaid
OrderShipped

A Consumer may need to receive these events in exactly this order.

If it receives:

OrderShipped
OrderCreated
OrderPaid

the business logic could behave incorrectly.

Therefore, ordering needs to be designed intentionally.

For example:

aggregate_id = order-123

You can then route events belonging to the same Aggregate to the same Partition in the Message Broker.

With Kafka, a Partition Key can help ensure that events for a particular Aggregate are placed in the same Partition and processed in order within that Partition.

The important distinction is:

The Outbox Pattern and Message Ordering are related, but they are separate problems.

What Happens to the Outbox Table Over Time ?

Suppose your system generates:

1,000 events / second

That's approximately:

86,400,000 events / day

If you never delete or archive processed events, the Outbox table will grow rapidly.

This can lead to:

• Increased database size

• Larger database indexes

• Heavier backups

• Slower queries

• Higher storage costs

Therefore, an Outbox needs a Lifecycle Strategy.

For example:

Pending
   ↓
Published
   ↓
Retention Period
   ↓
Delete / Archive

Three Common Strategies

1. Delete

Delete events after they have been successfully published.

This is simple and inexpensive, but it makes historical replay more difficult.

2. Archive

Move events to another storage system after a certain period.

This can be more useful for:

• Auditing

• Debugging

• Historical analysis

3. Partitioning

Partition the table based on time.

For example:

outbox_2026_09_18
outbox_2026_09_19
outbox_2026_09_20

Old partitions can then be deleted or archived efficiently.

Advantages of the Outbox Pattern

1. Preventing Lost Events

The most important advantage of the Outbox Pattern is that when the Business Transaction commits, the Event is also stored.

Therefore, if the Publisher crashes immediately after the commit, the Event is not lost.

2. No Need for 2PC

The database and Broker do not need to participate in a shared Distributed Transaction.

3. Separation of Business Operations and Message Publishing

The Business Request does not need to wait for the Broker.

4. Suitable for Event-Driven Architecture

The Outbox Pattern is particularly useful for communication between Microservices.

5. Easier Retries

If the Broker is temporarily unavailable:

Outbox
   ↓
Retry
   ↓
Retry
   ↓
Success

The Event remains safely stored in the database.

Disadvantages of the Outbox Pattern

The Outbox Pattern solves one class of reliability problems, but it also introduces additional responsibilities.

1. Increased Complexity

Instead of having only a database, you may now have:

Database
+
Outbox
+
Publisher
+
Broker
+
Retry
+
Monitoring
+
Cleanup

2. Duplicate Events

The Publisher may send an Event more than once.

Therefore, Consumers should be designed to be idempotent.

3. Eventual Consistency

An Event does not necessarily reach every service immediately after the database transaction commits.

For a short period, you might have:

Order Service     → CREATED
Inventory Service → still knows nothing

This delay needs to be acceptable from a business perspective.

4. Outbox Growth

Without a proper lifecycle strategy, the Outbox can become a very large table.

5. New Failure Modes

You now need to handle problems such as:

Publisher Crash
Broker Down
Database Down
Poison Message
Retry Storm
Duplicate Event
Outbox Backlog

The Outbox Pattern therefore does not eliminate reliability problems.

Instead, it changes the types of failures your system needs to handle.

Outbox Pattern vs. 2PC

The Outbox Pattern takes a fundamentally different approach from distributed transactions.

Instead of trying to create one global transaction, it keeps the transaction within the boundaries of a single database.

Database Transaction
├── Business Data
└── Outbox Event

Message publishing happens afterward.

The goal is not to make the database and Message Broker participate in the same transaction, but to make the database operation itself reliable and then handle message delivery separately.

Outbox vs. Direct Event Publishing

The simple approach is:

UPDATE DB
   ↓
PUBLISH EVENT

Advantages

• Simple

• Low cost

• Low latency

Disadvantages

• Dual Write

• Potential Lost Event

• Failure Window

The Outbox approach is:

UPDATE DB
   +
INSERT OUTBOX
   ↓
COMMIT
   ↓
PUBLISH ASYNC

Advantages

• Higher reliability

• Easier retries

• No Lost Event in common failures between Commit and Publish

Costs

• Increased complexity

• Eventual Consistency

• Duplicate handling

• Operational overhead

Therefore, if an Event is not actually critical, the Outbox Pattern is not necessarily required.

For example, if Events are used only for non-critical analytics and losing a small percentage of them is acceptable, keeping the architecture simple may be more valuable.

Is the Outbox Pattern Only for Microservices ?

No.

A common misconception is that the Outbox Pattern is exclusively a Microservices pattern.

Even in a Monolithic Architecture, you may need to perform an external operation after changing the database.

For example:

Order Created
     ↓
Call a Webhook

Or:

User Registered
     ↓
Send Email

If the Email Provider is temporarily unavailable, you may not want to fail the user's registration.

Instead, you can store:

User
+
Email Event

inside the same transaction and send the email asynchronously afterward.

So the Outbox Pattern is better understood as a reliable solution for Asynchronous Messaging rather than something limited to Microservices.

When Is the Outbox Pattern Worth Using ?

The Outbox Pattern is particularly valuable when:

• The Event is important to the business.

• Losing an Event is unacceptable.

• The database and Broker are independent systems.

• The architecture is Event-Driven.

• Services communicate asynchronously.

• Retry and reliability are important.

• Immediate consistency between services is not required.

When Isn't the Outbox Pattern Necessary ?

If an Event is completely non-critical, adding an Outbox may introduce unnecessary complexity.

For example:

User clicked button

Suppose this Event is used only for approximate analytics.

If losing some Events is acceptable, direct publishing—or even an independent analytics system—may be a simpler solution.

The right architecture depends on the business consequences of failure.

Technical Considerations When Implementing Outbox

1. Use a Unique Event ID

Every Event should have a stable identifier:

event_id = UUID

This ID is essential for Deduplication.

2. Design Consumers to Be Idempotent

Suppose:

OrderPaid

is delivered twice.

The result should not be:

Payment = Payment × 2

The Consumer needs to recognize that the Event has already been processed.

3. Index the Outbox

If you use polling, queries such as:

WHERE processed_at IS NULL ORDER BY created_at

should be supported by appropriate indexes based on your workload and access patterns.

4. Consider Batch Processing

Instead of:

1 Event → 1 Query

you may be better off processing:

100 Events → 1 Batch

The AWS Batch Processing guidance provides useful background on batch processing.

However, very large batches can also increase:

• Lock duration

• Memory usage

• Retry costs

So the batch size should be chosen carefully.

5. Use Retry with Backoff

If the Broker is unavailable, you should not continuously retry every few milliseconds.

For example:

1s
2s
4s
8s
16s
...

Use exponential backoff together with appropriate retry limits and policies.

6. Don't Forget Dead-Letter Handling

Sometimes an Event will consistently fail.

If you retry indefinitely:

Event
 ↓
Fail
 ↓
Retry
 ↓
Fail
 ↓
Retry
 ↓
...

you can create a Retry Storm.

A problematic Event therefore needs an explicit failure-handling strategy, such as a Dead Letter mechanism.

An Important Note About Monitoring

One of the most useful Outbox metrics is the number of Events waiting to be published.

For example:

Outbox Pending Events = 12

might be completely normal.

But:

Outbox Pending Events = 2,000,000

could indicate that the Publisher has fallen significantly behind.

Useful metrics can include:

Outbox Backlog
Publish Latency
Publish Failure Rate
Retry Count
Oldest Unprocessed Event Age
Consumer Lag
Dead Letter Count

One particularly valuable metric is:

Oldest Unprocessed Event Age

For example:

Age = 45 minutes

Even if the total number of pending Events is relatively small, an old unprocessed Event could indicate an operational problem.

The Outbox Pattern and Eventual Consistency

The Outbox Pattern forces us to accept an important reality of distributed systems:

Not everything needs to be immediately consistent.

For example:

T0:
Order Created

T0 + 20ms:
Outbox Published

T0 + 50ms:
Inventory Updated

T0 + 100ms:
Notification Sent

For a short period, the system might look like this:

Order Service     → CREATED
Inventory Service → Previous State

This is not necessarily a bug.

If your business requirements allow this delay, Eventual Consistency can be a perfectly reasonable trade-off.

In Event-Driven Architectures, this is one of the fundamental design considerations.

Duplication, ordering, and idempotency all need to be deliberately designed.

Final Thoughts

The Outbox Pattern is a solution to one of the classic problems in distributed systems: the Dual Write Problem.

Instead of having an application simultaneously modify:

Database
+
Message Broker

we store the business state change and its corresponding Event in a single local database transaction:

Database
├── Business Data
└── Outbox Event

A separate process then transfers the Event to the Message Broker.

The main benefit is straightforward:

If the original transaction commits, the corresponding Event is also persisted and can be retried later.

But the Outbox Pattern is not the end of the story.

You still need to deal with:

• Duplicate messages

• Idempotency

• Ordering

• Eventual Consistency

• Retries

• Outbox Growth

• Monitoring

• Cleanup

In other words, the Outbox Pattern does not eliminate distributed-system complexity.

It moves that complexity into areas where it can be managed more explicitly.

A Question for Discussion

If losing an Event is unacceptable for your system, but a few seconds of Eventual Consistency is tolerable, would you introduce the additional complexity of the Outbox Pattern, retries, and idempotency ?

And at what point would you decide that architectural simplicity is more valuable than higher messaging reliability ?

Further Reading

• Transactional Outbox — Microservices.io

• AWS Prescriptive Guidance — Transactional Outbox

• Debezium — Outbox Event Router

• Microsoft Azure Architecture Center — Transactional Outbox

• The Outbox Pattern: A Love Letter to Eventual Consistency

Imagine your payment service is down. Normally, every purchase request follows a path like this:

User
 ↓
Order Service
 ↓
Payment Service
 ↓
Bank

Now the Payment Service is unavailable — or perhaps it’s simply extremely slow. A request that normally takes 200 ms now takes 10 seconds. If your system continues sending requests to it, what happens ?

• Request queues grow.

• Threads and connections remain occupied.

• Timeouts increase.

• Latency goes up.

And eventually, services that were perfectly healthy can start failing too.

In other words, one failing dependency can potentially bring down an entire system.

This is where the Circuit Breaker pattern comes in.

Instead of continuously calling a dependency that is already failing, the system temporarily stops sending requests to it and fails fast.

The goal isn’t to fix the broken service. The goal is to prevent its failure from spreading.

What Is the Circuit Breaker Pattern ?

The Circuit Breaker is a resilience pattern commonly used in distributed systems to protect services from repeatedly calling an unhealthy dependency.

The idea comes from electrical circuit breakers. When electrical current becomes dangerously high, a circuit breaker interrupts the circuit to prevent further damage.

Software can apply a similar idea:

Healthy dependency
↓
Requests flow
↓
Dependency starts failing
↓
Failure threshold reached
↓
Circuit opens
↓
Requests fail fast

A Circuit Breaker doesn’t repair the dependency.

Instead, it can:

• Stop unnecessary requests to a failing service

• Protect your own application’s resources

• Reduce additional load on the unhealthy dependency

• Help prevent cascading failures

• Give the dependency time to recover

• Allow the system to degrade gracefully when possible

A useful way to think about it is:

Circuit Breaker doesn’t eliminate failure. It prevents failure from becoming a larger system-wide failure.

Why Do Distributed Systems Need Circuit Breakers ?

In a simple monolithic application, you might call a function like:

calculatePrice()

If the function fails, handling the failure is usually relatively straightforward. But distributed systems are different.

A call to another service crosses a network boundary:

Service A
↓
Service B
↓
Service C
↓
External APIService A
↓
Service B
↓
Service C
↓
External API

Unlike a local function call, a remote call can:

• Time out

• Lose its connection

• Become extremely slow

• Return a 5xx error

• Be rate-limited

• Become completely unavailable

And the more important problem is that failure can propagate between services.

For example:

Payment Service
↓
starts failing
↓
Order Service waits
↓
Threads become occupied
↓
Requests start queuing
↓
Latency increases
↓
Error rate increases
↓
The entire service comes under pressure

This is a cascading failure.

The original problem may have started in one service, but the resulting resource exhaustion can spread to otherwise healthy parts of the system.

Circuit Breaker is one of the patterns that can help interrupt this chain.

Reliability, High Availability, and Circuit Breakers

Two concepts often appear in discussions about distributed systems:

Reliability: is about a system behaving dependably in the presence of failures and recovering appropriately when failures occur.

High Availability: is about keeping a system accessible to users even when parts of its infrastructure or dependencies experience problems.

A Circuit Breaker does not make an external service permanently available.

Instead, it can help isolate its failure:

Dependency Failure
↓
Circuit Breaker
↓
Failure Isolation
↓
Graceful Degradation
↓
System remains responsive

This is where the idea of failure isolation becomes important.

Instead of allowing one dependency’s failure to spread across the entire system, we try to limit its blast radius.

How Does a Circuit Breaker Work ?

A Circuit Breaker is commonly modeled as a state machine with three primary states:

• Closed

• Open

• Half-Open

CLOSED - Everything Is Normal

When the Circuit Breaker is CLOSED, requests are allowed to reach the dependency.

Client
↓
Circuit Breaker
↓
Payment Service

The Circuit Breaker monitors the results of those requests. For example:

100 requests
95 successes
5 failures

If the failure rate exceeds the configured threshold, the circuit transitions to OPEN.

But an important detail is that a failure threshold doesn’t necessarily mean:

Five requests failed consecutively.

The threshold can be based on different measurements, such as:

• Number of failures

• Failure percentage

• Number of timeouts

• Failures within a time window

• A combination of multiple metrics

For example:

50 failures in the last 100 requests

or:

Failure Rate > 30% during the last 30 seconds

The right threshold depends on the actual behavior of the dependency and the reliability objectives of your system. There is no universal magic number.

OPEN - Stop Calling the Failing Service

When the Circuit Breaker transitions to OPEN, its behavior changes completely.

Before:

Request
↓
Circuit Breaker
↓
Payment Service

After:

Request
↓
Circuit Breaker
↓
FAIL FAST

The request never reaches the Payment Service. This is the central idea behind the Circuit Breaker pattern. If we already have strong evidence that a dependency is unhealthy, why should every new request wait for another network timeout ?

Suppose the Payment Service has a 10-second timeout. Now imagine 1,000 concurrent requests all waiting for that timeout:

1,000 requests × 10 seconds

A significant amount of your application’s resources can remain occupied simply waiting for a dependency that is already failing.

With an open circuit:

Request
↓
Circuit = OPEN
↓
Immediate failure

The system can reject the request much earlier.

That can help:

• Free threads

• Preserve connections

• Reduce resource consumption

• Keep latency more predictable

• Prevent additional traffic from reaching the unhealthy dependency

Sometimes failing quickly is better than failing slowly.

But the circuit should not remain open forever. If it did, the system would never discover that the dependency has recovered.

After a configured period, the Circuit Breaker can transition to HALF-OPEN.

HALF-OPEN - Has the Service Recovered ?

This is one of the most subtle parts of the pattern.

Suppose the Payment Service was unavailable, but now appears to be healthy again. Should we immediately send all traffic back to it ?

Not necessarily. The service may have recovered only partially. For example, it might have just restarted and currently be capable of handling only a small amount of traffic.

Instead of immediately allowing thousands of requests through, the Circuit Breaker can allow a limited number of test requests. For example:

HALF-OPEN
↓
5 test requests
↓
Are they successful ?

If the requests succeed:

HALF-OPEN
↓
Successful requests
↓
CLOSED

The circuit returns to normal operation.

If the test requests fail:

HALF-OPEN
↓
Failure
↓
OPEN

The circuit opens again.

Why Does HALF-OPEN Allow Only a Few Requests ?

Because service recovery doesn’t necessarily mean full recovery. Imagine a Payment Service that was down and has just come back online. It might currently be capable of processing only:

20 requests / second

If your system suddenly sends:

10,000 requests

you could create another overload:

Recovery
↓
Traffic spike
↓
Overload
↓
Failure again

In other words, your system can accidentally overwhelm the dependency immediately after it recovers.

Half-Open provides a controlled way to test recovery:

Instead of:

0 requests
↓
10,000 requests

we can do:

0 requests
↓
A few test requests
↓
Evaluate results
↓
Gradually increase traffic

The exact number of test requests is implementation- and system-dependent. There is no universal value that works for every architecture.

A Real-World Scenario: Payment Service Failure

Let’s put everything together. Imagine an online store:

User
↓
Order Service
↓
Payment Service
↓
Bank Gateway

Under normal conditions:

Order Service
↓
Circuit Breaker
↓
Payment Service
↓
Bank

Everything is working normally.

Step 1: The Failure Starts

The Bank Gateway begins experiencing problems. The Payment Service starts timing out:

Payment Request
↓
Timeout

The Circuit Breaker records these failures.

Step 2: The Failure Threshold Is Reached

Suppose our policy is:

100 requests
40 failures

and our configured rule is:

Failure Rate > 30% → OPEN

The threshold has been exceeded. The Circuit Breaker opens.

Step 3: New Requests Fail Fast

Now a new request arrives:

User
↓
Order Service
↓
Circuit = OPEN
↓
Immediate Failure

The Order Service no longer calls the Payment Service.

This prevents the application from repeatedly waiting for a dependency that is already known to be unhealthy.

Step 4: The System Degrades Gracefully

Once the circuit is open, the system still has to decide what the user should experience. For example:

Payment is temporarily unavailable. Please try again in a few minutes.

Or the application might have another valid strategy:

• Keep the order in a PENDING state

• Put the operation into a queue

• Switch to another payment provider

• Return non-critical information from a cache

This is where fallback becomes important.

What Is a Fallback?

A fallback answers a simple question:

If the primary path fails, do we have a meaningful alternative ?

For example:

Primary Payment Provider
↓
FAILED
↓
Secondary Payment Provider

But a fallback is not always possible. For a payment operation, you cannot simply return Payment successful, when the payment was never actually processed.

The fallback must therefore be consistent with the business logic. A fallback is not a fake success. It is an alternative behavior that is safe and meaningful for that particular operation.

Circuit Breaker ≠ Retry

These two concepts are often confused. They solve different problems.

Retry says:

The request failed. Maybe trying again will succeed.

Circuit Breaker says:

This dependency has been failing repeatedly. Stop sending requests to it for now.

Retry can be useful for transient failures, such as:

• Temporary network glitch

• Temporary timeout

• Temporary throttling

But if the dependency is genuinely down, excessive retries can make the situation worse.

How Retries Can Cause a Cascading Failure

Suppose you have 1,000 requests and every failed request is retried three times. You might end up with:

1,000 original requests + 3,000 retries = 4,000 requests

That’s exactly what you don’t want when the dependency is already overloaded.

Now imagine:

A → B → C

Service C fails. Service B retries its requests to C. Service A retries its requests to B.

The result can look like this:

C fails
↓
B retries
↓
A retries
↓
More traffic
↓
C becomes even more overloaded
↓
More failures
↓
Cascading failure

This can lead to a retry storm.

So should we eliminate retries ?

No. The problem isn’t retrying itself. The problem is uncontrolled retries. A more resilient design might combine:

Request
↓
Timeout
↓
Retry
↓
Exponential Backoff + Jitter
↓
Circuit Breaker

The important point is that retries should generally be limited to failures where another attempt has a reasonable chance of succeeding.

For example, retrying a transient 503 Service Unavailable may make sense in some systems.

Retrying a 401 Unauthorized generally does not solve the underlying problem.

Exponential Backoff and Jitter

Imagine 10,000 requests fail at approximately the same time. If every client retries exactly one second later:

1 second
↓
10,000 requests

You’ve created another traffic spike.

With exponential backoff, the delay between retries increases:

1s
2s
4s
8s
…

This helps spread retries over a longer period.

But there’s still a problem. If every client follows exactly the same schedule, they can still retry at roughly the same moments. That’s where jitter helps. Instead of:

1s
2s
4s
8s

different clients might retry at slightly different times:

1.2s
1.8s
2.4s
3.1s
…

The goal is to spread retry traffic instead of allowing thousands of clients to synchronize their retries.

Idempotency: The Problem You Must Consider When Retrying

Retries introduce another important problem: duplicate side effects. Imagine:

POST /payments

The request reaches the Payment Service. The payment is successfully processed. But the response is lost because of a network failure. What does the client see?

Timeout

From the client’s perspective, it doesn’t know whether the payment succeeded. So it retries:

POST /payments

Now you potentially have:

Request #1 → Payment successful
↓
Response lost
↓
Retry
↓
Request #2 → Payment successful again

You may have charged the customer twice. This is where idempotency becomes critical. For example, the client could send an idempotency key:

Idempotency-Key: ABC123

The Payment Service can use that key to recognize that the same logical operation has already been processed.

Conceptually:

Retry
↓
Same Idempotency Key
↓
Same Logical Operation
↓
No Duplicate Side Effect

Therefore, whenever you introduce retries, ask:

Is this operation actually safe to retry ?

Important Circuit Breaker Configuration Parameters

Implementing a Circuit Breaker isn’t simply a matter of adding an if statement. Several parameters need to be defined.

1. Failure Threshold

How many failures should cause the circuit to open ? For example:

Failure Rate > 50%

A threshold that is too low may cause the circuit to open because of temporary failures.

A threshold that is too high may allow the dependency to cause significant damage before the circuit reacts.

2. Recovery Timeout

Once the circuit is open, how long should we wait before testing the dependency again ?

For example:

30 seconds

A timeout that is too short can produce this cycle:

OPEN
↓
HALF-OPEN
↓
Failure
↓
OPEN
↓
HALF-OPEN
↓
…

On the other hand, a timeout that is too long may prevent requests from reaching a dependency even after it has recovered.

3. Number of HALF-OPEN Requests

When the circuit enters HALF-OPEN, how many requests should be allowed through ?

For example:

1 request

or:

5 requests

or a limited percentage of traffic.

Allowing more requests can provide more information about recovery, but it also creates more load on the recovering dependency.

4. Which Failures Should Open the Circuit ?

Not every error should necessarily be treated as evidence that a dependency is unhealthy.

For example:

Timeout → likely relevant
503 → likely relevant
Connection error → likely relevant
401 → probably not
400 → probably not

If every 4xx error contributes to the Circuit Breaker threshold, you could incorrectly conclude that the entire dependency is unavailable.

Circuit Breaker Has a Cost Too

No resilience pattern is free. Circuit Breaker provides important benefits, but it also introduces complexity.

Potential benefits

• Helps prevent cascading failures

• Reduces load on an unhealthy dependency

• Enables fail-fast behavior

• Protects application resources

• Supports graceful degradation

• Improves system resilience

• Enables controlled recovery

Potential costs

• More application complexity

• Additional monitoring requirements

• Threshold tuning

• State management

• Fallback design

• More difficult debugging

• False positives

• False negatives

For example, if your threshold is too sensitive:

Temporary failure
↓
Circuit opens
↓
Requests rejected

even though the dependency wasn’t actually down. If the threshold is too permissive:

Dependency is failing
↓
Circuit remains CLOSED
↓
More failures
↓
More resource consumption

So configuring a Circuit Breaker is a trade-off, not a search for one perfect number.

Where Should the Circuit Breaker Live ?

A common placement is close to the remote call:

Order Service
↓
Circuit Breaker
↓
Payment Service

This keeps the decision close to the dependency being protected.

However, Circuit Breaking can also be implemented at other layers, depending on the architecture:

• Application layer

• API Gateway

• Service Mesh

• Sidecar

A Service Mesh for example, can provide resilience features without requiring every application to implement the same logic. But that introduces another trade-off. Moving more resilience behavior into infrastructure can simplify application code, while potentially making debugging and understanding system behavior more complicated.

Do You Always Need a Circuit Breaker ?

No.

Circuit Breaker is not automatically appropriate for every operation.

For example:

function calculateTax()

If this is a local, fast operation with no meaningful remote dependency, adding a Circuit Breaker may simply introduce unnecessary complexity.

Circuit Breakers become more interesting when dealing with things such as:

• Remote APIs

• External providers

• Database dependencies

• Microservices

• Third-party services

The key question isn’t:

Do we use microservices ?

The better question is:

What happens to our system when this dependency becomes slow, unavailable, or unreliable ?

Circuit Breaker Alone Is Not Resilience

One common misconception is Circuit Breaker = Resilience. It isn’t.

Circuit Breaker is only one tool in a broader resilience strategy.

A more complete design might look like:

     ┌─────────────┐
     │   Timeout   │
     └──────┬──────┘
            ↓
     ┌─────────────┐
     │    Retry    │
     └──────┬──────┘
            ↓
┌───────────────────────┐
│ Exponential Backoff   │
│       + Jitter        │
└───────────┬───────────┘
            ↓
    ┌─────────────┐
    │   Circuit   │
    │   Breaker   │
    └──────┬──────┘
           ↓
    ┌─────────────┐
    │  Fallback   │
    └─────────────┘

Depending on the system, other patterns and mechanisms may also be relevant:

• Bulkheads

• Rate limiting

• Queues

• Caching

• Health checks

• Observability

• Idempotency

Resilience usually comes from combining the right mechanisms, not from adding one pattern everywhere.

How Do You Know Your Circuit Breaker Is Working ?

A Circuit Breaker without observability is difficult to operate. At a minimum, you should be able to monitor metrics such as:

• Circuit state

• Failure rate

• Success rate

• Timeout rate

• Number of circuit openings

• Half-Open attempts

• Recovery time

• Fallback rate

• Request latency

State transitions can also be important monitoring events. For example:

Circuit: CLOSED → OPEN

This may indicate a significant problem with an external dependency.

Without monitoring, you may know that requests are failing — but not know why the Circuit Breaker opened, how often it opens, or whether it is actually helping.

The Goal Isn’t to Eliminate Errors

This is one of the most important ideas behind the pattern. A Circuit Breaker does not necessarily reduce the number of errors users see. In fact, after a circuit opens, you may see more immediate failures. But those failures can be much more controlled.

Without a Circuit Breaker:

1,000 requests
↓
1,000 timeouts
↓
10 seconds each
↓
Thread exhaustion
↓
Entire service becomes unhealthy

With a Circuit Breaker:

1,000 requests
↓
Circuit OPEN
↓
1,000 fast failures
↓
Dependency protected
↓
Main service remains responsive

In both cases, there are failures. But the second scenario prevents those failures from consuming resources indefinitely. That’s the key distinction.

The goal of a Circuit Breaker isn’t to make failure disappear. It’s to turn an uncontrolled failure into a controlled one.

And that is one of the fundamental ideas behind resilient distributed systems.

The State Machine in One Picture

Let’s summarize the state transitions:

CLOSED: Requests are allowed through.

CLOSED
↓
Requests flow normally

If failures exceed the configured threshold:

CLOSED → OPEN

OPEN: Requests fail fast without calling the dependency.

OPEN
↓
Fail Fast

After the configured recovery timeout:

OPEN → HALF-OPEN

HALF-OPEN: A limited number of test requests are allowed.

If recovery succeeds:

HALF-OPEN → CLOSED

If the dependency fails again:

HALF-OPEN → OPEN

So the complete lifecycle is:

Image by geeksforgeeks

The Bigger Lesson

It’s useful to remember the three states, But understanding why they exist is much more important.

When a dependency is failing, continuing to send more requests isn’t necessarily resilience. Sometimes the most resilient thing your system can do is stop making the problem worse.

Circuit Breaker, together with carefully designed timeouts, limited retries, exponential backoff, jitter, fallback strategies, and idempotency, can help prevent a localized dependency failure from turning into a much larger outage.

The central idea is simple:

When a dependency is unhealthy, protect your system first — and give the dependency a controlled chance to recover.

Final Takeaway

The Circuit Breaker pattern is a resilience mechanism for systems that depend on remote services or external resources where failures, timeouts, and temporary unavailability are possible.

Its three primary states are:

CLOSED
↓
Requests flow normally

OPEN 
↓ 
Requests fail fast

HALF-OPEN
↓
A limited number of requests test recovery

The most important lesson isn’t memorizing these states. It’s understanding the philosophy behind them:

A resilient system doesn’t just know how to handle failure. It knows when to stop making a failure worse.

A Final Thought Experiment

Imagine your Payment Service is completely down. Your system receives 10,000 payment requests per minute. Which behavior makes more sense ?

Option 1: Retry every request three times. Option 2: Open the Circuit Breaker after the failure rate crosses the configured threshold and fail fast. Option 3: Use a combination of:

The interesting part isn’t memorizing which pattern to use. The real engineering challenge is understanding how these mechanisms interact under failure. That’s where resilient distributed-system design begins.

Further Reading

• Microsoft Azure — Circuit Breaker Pattern

• Microsoft Azure — Transient Fault Handling

• Microsoft Azure — Retry Storm Antipattern

• AWS — Using the Circuit Breaker Pattern with Step Functions and DynamoDB

• AWS Builders’ Library — Making retries safe with idempotent APIs

• Martin Fowler — Circuit Breaker

• Circuit Breaker Design Pattern

• Resilient Distributed Systems

• Top Strategies to Improve Reliability in Distributed Systems

• How failure cascades in Distributed Systems

• Recovery in Distributed Systems

• Monolithic Architecture

• Finite-State Machine

• Service-Level Objective

• Retries Strategies in Distributed Systems

• Failure Detection and Recovery in Distributed Systems

• Graceful Degradation in Distributed Systems

• Recovering from Transient Errors

• Retry Storm Antipattern

• Understanding Retries, Exponential Backoffs, and Circuit Breakers in Distributed Systems

• Exponential Backoff And Jitter

• What Are False Positives and Negatives in Software Testing ?

• What is a Service Mesh ?

• Sidecar Design Pattern for Microservices

• Distributed Systems and Microservices: Guide to Streamline Software Architectures

• Bulkhead Pattern

• Rate Limiting in Distributed System

• Distributed Task Queue — Distributed Systems

• Health Checks and Graceful Degradation in Distributed Systems

• Observability in Distributed Systems

• Idempotency in Distributed Systems: When and Why It Matters

• Distributed Computing

If you’ve ever wondered what MySQL actually does when transactions run concurrently, this is where it gets interesting.

InnoDB gives you four transaction isolation levels:

• READ UNCOMMITTED

• READ COMMITTED

• REPEATABLE READ

• SERIALIZABLE

Most developers know their names, fewer can predict what will actually happen when two transactions execute at the same time.

That’s what we’re going to do here.

No abstract definitions first. No memorizing a table. We’ll make the transactions collide and see what MySQL does.

Why Do We Need Transactions ?

Consider a typical e-commerce purchase. When a customer buys a product, several database operations may need to happen together:

1. Check the product inventory.

2. Decrease the inventory.

3. Create the order.

4. Record the payment.

5. Update the order status.

What happens if the inventory is successfully decreased but creating the order fails ?

You could end up with a database where the product’s inventory has decreased, but no corresponding order exists.

That’s exactly the kind of problem transactions are designed to prevent.

A transaction lets us treat multiple database operations as a single logical unit:

Either all of the operations succeed, or the changes are rolled back.

For example:

START TRANSACTION;
UPDATE products SET stock = stock - 1 WHERE id = 10;
INSERT INTO orders (product_id, user_id) VALUES (10, 1001);
COMMIT;

If something goes wrong:

ROLLBACK;

Transactions are one of the foundations of ACID, and the I in ACID stands for Isolation. Isolation is where things get interesting.

What Is a Transaction Isolation Level ?

Imagine two transactions running at approximately the same time:

Transaction A                Transaction B
      |                            |
      |---- read data ------------>|
      |                            |
      |                       update data
      |                            |
      |<--- what can A see ? ------|

Both transactions may be reading and modifying the same data. The isolation level defines the rules for what each transaction is allowed to see while other transactions are running.

Those rules can dramatically change the behavior of your application.

For example:

• Can you see another transaction’s uncommitted changes ?

• Can the same query return a different value later ?

• Do you keep seeing the same snapshot ?

• Does your read lock the row ?

• Can another transactions modify the row while you’re making a decision ?

Let’s find out.

1. READ UNCOMMITTED - When Uncommitted Data Becomes Visible

READ UNCOMMITTED is the least restrictive isolation level. A transaction can potentially see changes made by another transaction before those changes have been committed. This is known as a dirty read.

Suppose the database contains:

stock = 10

Transaction A starts:

START TRANSACTION;
UPDATE products SET stock = 0 WHERE id = 10;

But Transaction A hasn’t committed yet.

Now Transaction B executes:

SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
START TRANSACTION;
SELECT stock FROM products WHERE id = 10;

Transaction B may see:

0

Wait. That 0 hasn't actually been committed.

Now Transaction A rolls everything back:

ROLLBACK;

The actual value goes back to:

10

Transaction B just observed a value that never became part of the committed database state. That’s a dirty read.

Why is this dangerous ?

Imagine applying the same idea to:

• payments

• bank balances

• inventory

• orders

• seat reservations

You could make a business decision based on data that ultimately disappears. A simple way to remember it:

READ UNCOMMITTED: you get more concurrency by giving up consistency.

2. READ COMMITTED - Only See What Has Been Committed

READ COMMITTED takes a more conservative approach. A transaction doesn’t see another transaction’s uncommitted changes through a normal consistent read. But there’s a catch.

The data you see can change while your transaction is still running.

Suppose:

Initial stock = 10

Transaction A starts:

SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
START TRANSACTION;
SELECT stock FROM products WHERE id = 10;

It sees:

10

Now Transaction B changes the value:

START TRANSACTION;
UPDATE products SET stock = 5 WHERE id = 10;
COMMIT;

Transaction A executes the same query again:

SELECT stock FROM products WHERE id = 10;

This time:

5

The same transaction saw 10 and later 5. That’s a Non-Repeatable Read.

But why this happens ? Because under READ COMMITTED, each consistent read can establish its own fresh snapshot.

When is READ COMMITTED useful ?

It’s a good fit for systems where seeing relatively fresh committed data is more important than maintaining one consistent snapshot throughout the entire transaction.

It also changes InnoDB’s locking behavior. For locking READ, UPDATE, and DELETE, InnoDB generally uses record locks rather than gap locks, except where gap locking is needed for foreign-key and duplicate-key checks.

A simple way to remember it:

READ COMMITTED: every consistent read sees committed data as of that read, so a later read can see changes committed by other transactions.

3. REPEATABLE READ - Your Transaction Gets a Snapshot

Now we reach InnoDB’s default isolation level: REPEATABLE READ.

Suppose:

stock = 10

Transaction A starts:

SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
START TRANSACTION;
SELECT stock FROM products WHERE id = 10;

It sees:

10

Now Transaction B changes the value:

START TRANSACTION;
UPDATE products SET stock = 5 WHERE id = 10;
COMMIT;

Transaction A runs the same query again:

SELECT stock FROM products WHERE id = 10;

And for a normal, non-locking consistent read, it can still see 10, not 5.

Why is that ? Because consistent reads in a REPEATABLE READ transaction use the transaction's snapshot.

So from Transaction A’s perspective, the database can effectively look like:

Transaction A’s view:

stock = 10

even though the current committed value is now:

stock = 5

4. SERIALIZABLE - Make Concurrency Behave More Like Sequential Execution

SERIALIZABLE is the strictest of the four isolation levels. Its goal is to provide behavior that is closer to transactions executing one after another rather than freely interleaving.

Conceptually:

Transaction A
     |
     | read/write
     |
     v
Transaction B
     |
     | waits
     v
Transaction A commits
     |
     v
Transaction B continues

In InnoDB, when auto-commit is disabled, plain SELECT statements under SERIALIZABLE are implicitly converted to locking reads using FOR SHARE.

This provides stronger consistency guarantees, but the additional locking can reduce concurrency.

So while SERIALIZABLE sounds like the safest choice, it isn't automatically the best choice. Stronger isolation comes with a cost.

A simple way to remember it:

SERIALIZABLE: strongest isolation, but concurrency becomes more expensive.

Comparing the Four Levels

Here’s the quick mental model:

But don’t treat this table as the whole story.

InnoDB’s behavior also involves:

• MVCC

• consistent reads

• locking reads

• record locks

• gap locks

• next-key locks

• transaction boundaries

• indexes

The interesting part isn’t memorizing the table. It’s being able to predict what happens when two transactions collide.

The Bigger Lesson

Here’s the part that matters most in real applications:

Choosing an isolation level doesn’t automatically make your concurrent code correct.

Isolation level determines the visibility and concurrency rules of transactions, But your SQL statements determine how you interact with the data.

Compare:

SELECT stock FROM products WHERE id = 10;

with:

SELECT stock FROM products WHERE id = 10 FOR UPDATE;

They may look almost identical, but they are not.

The first is a consistent read.

The second is a locking read.

And when two requests hit your application at almost exactly the same time, that difference can determine whether your system behaves correctly.

The Question You Should Ask

When debugging or designing concurrent database code, don’t ask only:

“Which isolation level am I using ?”

Ask this instead:

“What happens if two transactions execute these exact statements at the same time ?”

That’s the question that exposes race conditions.

Once you start thinking in terms of concurrent transactions instead of isolated SQL statements, MySQL’s transaction model becomes much easier to understand.

Final Takeaway

Transaction isolation isn’t just a list of four configuration values. It’s a set of rules governing what your transactions can see, when they can see it, and how they interact with other transactions.

If you remember only one thing from this article, remember this:

The real test of your database design isn’t what happens when one transaction runs. It’s what happens when two transactions run at the same time.

And when those two transactions are fighting over the last item in stock, the difference between a normal SELECT and SELECT ... FOR UPDATE suddenly becomes very important.

That’s where we’re going next.

Next: MySQL Overselling - Why SELECT Isn't Enough and How SELECT ... FOR UPDATE Solves It