Idempotence

TL;DR

Idempotent operations produce the same result regardless of how many times they’re executed. In distributed systems, idempotence enables safe retries, duplicate message handling, and exactly-once semantics without complex deduplication logic. Critical for building reliable APIs and message processing systems.

Visual Overview

NON-IDEMPOTENT OPERATION (Dangerous with retries)
┌────────────────────────────────────────────────┐
│  Operation: account.balance += 100 (INCREMENT) │
│                                                │
│  Execution 1: balance = 1000 + 100 = 1100 ✓    │
│  Network fails, client retries...              │
│  Execution 2: balance = 1100 + 100 = 1200 ✕   │
│  (WRONG! User charged twice)                   │
│                                                │
│  Problem: Multiple executions → different      │
│           results (unintended side effects)    │
└────────────────────────────────────────────────┘

IDEMPOTENT OPERATION (Safe with retries)
┌────────────────────────────────────────────────┐
│  Operation: account.balance = 1100 (SET)       │
│                                                │
│  Execution 1: balance = 1100 ✓                 │
│  Network fails, client retries...              │
│  Execution 2: balance = 1100 ✓                 │
│  (Same result! Safe)                           │
│                                                │
│  Property: Multiple executions → same result   │
└────────────────────────────────────────────────┘

IDEMPOTENT WITH REQUEST ID (Best Practice)
┌────────────────────────────────────────────────┐
│  Request: {                                    │
│    request_id: "txn_abc123",                   │
│    action: "deposit",                          │
│    amount: 100                                 │
│  }                                             │
│  ↓                                             │
│  Execution 1:                                  │
│  - Check: request_id processed? NO             │
│  - Execute: balance += 100 → 1100              │
│  - Store: request_id = "txn_abc123"            │
│  - Return: success ✓                           │
│  ↓                                             │
│  Network fails, client retries...              │
│  ↓                                             │
│  Execution 2:                                  │
│  - Check: request_id processed? YES            │
│  - Skip execution (already done)               │
│  - Return: success ✓ (same result)             │
│                                                │
│  Result: Safe retries, no duplicate processing │
└────────────────────────────────────────────────┘

HTTP METHOD IDEMPOTENCE
┌────────────────────────────────────────────────┐
│  GET /users/123                                │
│  → Idempotent ✓ (read, no side effects)        │
│                                                │
│  PUT /users/123 {"name": "Alice"}              │
│  → Idempotent ✓ (set to specific value)        │
│                                                │
│  DELETE /users/123                             │
│  → Idempotent ✓ (deleted or already deleted)   │
│                                                │
│  POST /users {"name": "Bob"}                   │
│  → NOT Idempotent ✕ (creates new resource)    │
│                                                │
│  POST /orders/123/pay                          │
│  → NOT Idempotent ✕ (charges money)           │
│  Unless: Use idempotency key                   │
└────────────────────────────────────────────────┘

Core Explanation

What is Idempotence?

Idempotence (from mathematics) means an operation produces the same result when applied multiple times:

f(f(x)) = f(x)

Examples:
- SET value = 42: Idempotent (repeated SET has same effect)
- INCREMENT value: NOT idempotent (repeated INCREMENT increases value)
- DELETE item: Idempotent (item deleted, stays deleted)
- CREATE item: NOT idempotent (creates duplicate items)

In Distributed Systems:

Idempotence enables safe retries when network failures or timeouts occur, eliminating the need to distinguish between:

• Request failed (retry needed)
• Request succeeded but response lost (retry causes duplicate)

Why Idempotence Matters

Problem: Network Timeouts

Scenario: Payment API call

Client sends: "Charge $100 to card"
  ↓
Network timeout (no response received)
  ↓
Did payment succeed or fail?

Without idempotence:
- Don't retry → User may not be charged (bad)
- Retry → User may be charged twice (worse!)

With idempotence:
- Retry safely → Guaranteed charged exactly once ✓

At-Least-Once + Idempotence = Exactly-Once

Message delivery guarantees:

At-Most-Once:
- Message delivered 0 or 1 times
- May lose messages
- Use case: Metrics (OK to lose)

At-Least-Once:
- Message delivered 1+ times
- No loss, but duplicates possible
- Use case: Most systems

Exactly-Once:
- Message delivered exactly 1 time
- Hard to implement (requires transactions)
- OR: At-least-once + idempotent processing ✓

Idempotent vs Non-Idempotent Operations

Naturally Idempotent Operations:

SET operations:
account.balance = 1100           ✓ Idempotent
user.email = "alice@example.com" ✓ Idempotent

DELETE operations:
DELETE FROM users WHERE id=123   ✓ Idempotent
(second delete: already gone)

Absolute updates:
UPDATE users SET status='active' ✓ Idempotent
WHERE id=123

Read operations:
SELECT * FROM users WHERE id=123 ✓ Idempotent
(no side effects)

NOT Idempotent (Require Special Handling):

INCREMENT operations:
account.balance += 100           ✕ Not idempotent
view_count++                     ✕ Not idempotent

CREATE operations:
INSERT INTO orders (id, amount)  ✕ Not idempotent
(creates duplicate rows)

Relative updates:
UPDATE users SET age = age + 1   ✕ Not idempotent

Implementing Idempotence

1. Unique Request IDs (Idempotency Keys)

API Request with idempotency key:

POST /api/payments
Headers:
  Idempotency-Key: req_abc123xyz

Body:
{
  "amount": 100,
  "currency": "USD",
  "card_id": "card_789"
}

Server-side implementation:
┌────────────────────────────────────────────┐
│  1. Extract idempotency key from header    │
│  2. Check if key exists in database:       │
│     - EXISTS: Return cached response ✓     │
│     - NOT EXISTS: Process request          │
│  3. Execute business logic                 │
│  4. Store key + response in database       │
│  5. Return response                        │
└────────────────────────────────────────────┘

Result:
- First request (key=req_abc123xyz): Process payment
- Retry (same key): Return cached result, no duplicate charge
- New request (key=req_def456uvw): Process new payment

2. Natural Idempotency (Design for It)

BAD (Non-Idempotent):
POST /orders
{
  "product": "laptop",
  "quantity": 1
}
→ Creates new order each time ✕

GOOD (Idempotent with client-generated ID):
PUT /orders/order_abc123
{
  "product": "laptop",
  "quantity": 1
}
→ Creates or updates order_abc123 ✓
→ Retry-safe

GOOD (Idempotent with unique constraint):
POST /orders
{
  "client_order_id": "order_abc123",  // Unique!
  "product": "laptop",
  "quantity": 1
}
→ Database unique constraint prevents duplicates ✓

3. Database-Level Idempotency

Using database constraints:

CREATE TABLE payments (
  id SERIAL PRIMARY KEY,
  request_id VARCHAR(255) UNIQUE,  -- Idempotency key
  amount DECIMAL(10, 2),
  status VARCHAR(50),
  created_at TIMESTAMP
);

Application code:
INSERT INTO payments (request_id, amount, status)
VALUES ('req_abc123', 100.00, 'completed')
ON CONFLICT (request_id) DO NOTHING;

Result:
- First insert: Creates payment
- Retry: Conflict detected, no duplicate payment ✓

4. State Machine Approach

Payment state machine:

States: PENDING → PROCESSING → COMPLETED
                           ↓
                        FAILED

Transitions are idempotent:
- PENDING → PROCESSING: OK
- PROCESSING → PROCESSING: OK (retry, same state)
- PROCESSING → COMPLETED: OK
- COMPLETED → COMPLETED: OK (already completed)

Implementation:
UPDATE payments
SET status = 'COMPLETED'
WHERE id = 123 AND status IN ('PENDING', 'PROCESSING')

→ Retrying "complete payment" is safe ✓

Idempotent Message Processing

Kafka Consumer Example:

Problem: Kafka guarantees at-least-once delivery
→ Messages may be processed multiple times

Solution: Idempotent consumer

Non-Idempotent Consumer (Bad):
┌────────────────────────────────────────────┐
│  consume message: "increment counter"      │
│  counter++  // ✕ Not idempotent           │
│  commit offset                             │
│                                            │
│  If crash before commit:                   │
│  → Reprocess message                       │
│  → counter++ again (duplicate)             │
└────────────────────────────────────────────┘

Idempotent Consumer (Good):
┌────────────────────────────────────────────┐
│  consume message: {                        │
│    message_id: "msg_123",                  │
│    action: "increment_counter"             │
│  }                                         │
│  ↓                                         │
│  if not processed(message_id):             │
│    counter++                               │
│    mark_processed(message_id)              │
│  ↓                                         │
│  commit offset                             │
│                                            │
│  If crash and reprocess:                   │
│  → Check: msg_123 processed? YES           │
│  → Skip increment ✓                        │
└────────────────────────────────────────────┘

Common Patterns

1. Stripe API Style

POST /v1/charges
Headers:
  Idempotency-Key: unique_key_here

Body:
{
  "amount": 2000,
  "currency": "usd"
}

Behavior:
- First request: Create charge, return 200
- Retry with same key: Return cached 200 (no new charge)
- Different key: Create new charge
- Key expires after 24 hours

2. AWS S3 Style

PUT /bucket/object.txt
Content: "Hello World"

Behavior:
- Uploading same object multiple times: Idempotent ✓
- Result always: object.txt contains "Hello World"
- Uses content-based addressing (ETag)

3. Database Upsert Style

INSERT INTO users (id, name, email)
VALUES (123, 'Alice', 'alice@example.com')
ON DUPLICATE KEY UPDATE
  name = VALUES(name),
  email = VALUES(email)

Behavior:
- First call: Insert new user
- Retry: Update user (same result)
- Idempotent ✓

Real Systems Using Idempotence

System	Idempotency Mechanism	Key Feature	Use Case
Stripe API	Idempotency-Key header	24-hour key expiration	Payment processing
AWS APIs	Client request token	Service-specific	CloudFormation, EC2
Kafka	Message offset + deduplication	Consumer-side	Stream processing
Kubernetes	Declarative desired state	Reconciliation loop	Container orchestration
HTTP PUT	Resource URI	REST semantics	RESTful APIs
Git	Content-addressable	SHA hashes	Version control

Case Study: Stripe Payments

Stripe Idempotency Implementation:

POST https://api.stripe.com/v1/charges
Headers:
  Authorization: Bearer sk_test_...
  Idempotency-Key: req_abc123

Body:
  amount=2000&currency=usd

First Request:
1. Server checks: key "req_abc123" exists? NO
2. Process payment → charge card
3. Store: {key: "req_abc123", response: {...}, ttl: 24h}
4. Return: 200 OK {id: "ch_789", amount: 2000}

Retry (network timeout):
1. Server checks: key "req_abc123" exists? YES
2. Fetch cached response
3. Return: 200 OK {id: "ch_789", amount: 2000}
   (Same charge ID, no duplicate payment)

Different Request:
Idempotency-Key: req_def456
→ New payment, different charge ID

Key Expiration:
- Keys expire after 24 hours
- After expiration, same key creates new charge

Case Study: Kafka Idempotent Producer

Kafka Producer Idempotence (since 0.11):

Properties config = new Properties();
config.put("enable.idempotence", "true");
config.put("acks", "all");
config.put("retries", Integer.MAX_VALUE);

How it works:
┌────────────────────────────────────────────┐
│  Producer assigns sequence numbers:        │
│  Message 1: {seq: 0, data: "msg1"}         │
│  Message 2: {seq: 1, data: "msg2"}         │
│  Message 3: {seq: 2, data: "msg3"}         │
│  ↓                                         │
│  Broker tracks: producer_id + seq number   │
│  ↓                                         │
│  If duplicate received:                    │
│  - Message seq=1 already written           │
│  - Discard duplicate, ACK success ✓        │
│  ↓                                         │
│  Result: Exactly-once delivery to topic    │
└────────────────────────────────────────────┘

Guarantees:
✓ No duplicate messages in partition
✓ Messages ordered within partition
✓ Safe retries (producer can retry forever)

When to Use Idempotence

✓ Perfect Use Cases

Payment Processing

Scenario: Credit card charges
Requirement: Never double-charge users
Solution: Idempotency keys for payment API
Benefit: Safe retries on network failures

Order Processing

Scenario: E-commerce order placement
Requirement: Same order submitted multiple times → single order
Solution: Client-generated order ID
Benefit: Prevent duplicate orders

Inventory Updates

Scenario: Deduct inventory on purchase
Requirement: Don't deduct twice on retry
Solution: Transaction ID + database constraint
Benefit: Accurate inventory counts

Message Processing

Scenario: Kafka consumer processing events
Requirement: Process each message exactly once
Solution: Message ID tracking
Benefit: At-least-once + idempotence = exactly-once

✕ When NOT to Use (or Use Carefully)

Intentional Duplicates

Example: User clicking "Add to Cart" multiple times
Intent: Add multiple items
Solution: Don't use idempotency for this use case

Time-Sensitive Operations

Example: Stock trading (buy at current price)
Problem: Price changes between retries
Solution: Idempotency key + timestamp validation

Analytics/Metrics

Example: Page view counters
Acceptable: Slight overcounting on retries
Alternative: Use approximate counters (HyperLogLog)

Interview Application

Common Interview Question

Q: “Design an API for a payment system. How would you handle network retries to prevent double-charging users?”

Strong Answer:

“I’d implement idempotent payment processing using idempotency keys:

API Design:

POST /api/v1/payments
Headers:
  Authorization: Bearer token
  Idempotency-Key: unique_request_id
Body:
  {
    "amount": 100.00,
    "currency": "USD",
    "payment_method_id": "pm_123"
  }

Server-Side Implementation:

1. Extract Idempotency Key:

   • Required header, client-generated UUID
   • Example: Idempotency-Key: req_a1b2c3d4

2. Check Idempotency Table:

   CREATE TABLE idempotency_keys (
     key VARCHAR(255) PRIMARY KEY,
     request_hash VARCHAR(255),
     response_status INT,
     response_body TEXT,
     created_at TIMESTAMP,
     INDEX idx_created (created_at)
   );
   

3. Processing Logic:

   BEGIN TRANSACTION
     SELECT * FROM idempotency_keys WHERE key = :key
   
     IF EXISTS:
       // Validate request unchanged (hash matches)
       IF request_hash matches:
         RETURN cached response ✓
       ELSE:
         RETURN 400 Bad Request (key reused with different request)
   
     ELSE:
       // First time seeing this key
       // Process payment
       charge = stripe.charges.create(...)
   
       // Store idempotency record
       INSERT INTO idempotency_keys (
         key, request_hash, response_status, response_body
       ) VALUES (:key, :hash, 200, :response)
   
       COMMIT TRANSACTION
       RETURN 200 OK {charge_id: ...}
   

Benefits:

• Safe Retries: Client can retry infinitely
• No Double-Charging: Same key → same result
• Request Validation: Hash ensures request unchanged

Key Management:

• TTL: Expire keys after 24 hours
• Cleanup: Periodic job removes old keys
• Monitoring: Alert on high duplicate rate

Edge Cases:

1. Concurrent Requests (same key):
   • Use database locking (SELECT FOR UPDATE)
   • First request processes, others wait
   • All return same result
2. Partial Failures:
   • Payment succeeded but idempotency insert failed
   • Solution: Store idempotency key in payment record
   • Recovery: Lookup by key in payments table
3. Key Reuse (malicious or accidental):
   • Validate request hash matches
   • Return 400 if different request with same key

Alternatives Considered:

• No idempotency: Unacceptable (double-charging risk)
• Request deduplication only: Insufficient (response needed)
• Distributed lock: More complex, chose DB-based approach

Real-World Example: Stripe uses this exact pattern with Idempotency-Key header”

Code Example

Idempotent Payment API

from flask import Flask, request, jsonify
import hashlib
import json
import uuid
from datetime import datetime, timedelta

app = Flask(__name__)

# Simple in-memory store (use database in production)
idempotency_store = {}
payments_store = {}

def compute_request_hash(request_data):
    """Compute hash of request body for validation"""
    return hashlib.sha256(
        json.dumps(request_data, sort_keys=True).encode()
    ).hexdigest()

@app.route('/api/v1/payments', methods=['POST'])
def create_payment():
    """Idempotent payment endpoint"""

    # Extract idempotency key
    idempotency_key = request.headers.get('Idempotency-Key')

    if not idempotency_key:
        return jsonify({'error': 'Idempotency-Key header required'}), 400

    # Get request data
    request_data = request.get_json()
    request_hash = compute_request_hash(request_data)

    # Check if we've seen this idempotency key before
    if idempotency_key in idempotency_store:
        stored = idempotency_store[idempotency_key]

        # Validate request unchanged
        if stored['request_hash'] != request_hash:
            return jsonify({
                'error': 'Idempotency key reused with different request'
            }), 400

        # Return cached response
        print(f"✓ Returning cached response for key: {idempotency_key}")
        return jsonify(stored['response']), stored['status_code']

    # First time seeing this key - process payment
    try:
        # Simulate payment processing
        payment_id = str(uuid.uuid4())

        # Create payment record
        payment = {
            'id': payment_id,
            'amount': request_data['amount'],
            'currency': request_data.get('currency', 'USD'),
            'status': 'succeeded',
            'created_at': datetime.now().isoformat()
        }

        payments_store[payment_id] = payment

        # Store idempotency record
        idempotency_store[idempotency_key] = {
            'request_hash': request_hash,
            'response': payment,
            'status_code': 200,
            'created_at': datetime.now()
        }

        print(f"✓ Processed payment: {payment_id} for key: {idempotency_key}")

        return jsonify(payment), 200

    except Exception as e:
        return jsonify({'error': str(e)}), 500

@app.route('/api/v1/payments/<payment_id>', methods=['GET'])
def get_payment(payment_id):
    """Retrieve payment (idempotent by nature)"""
    payment = payments_store.get(payment_id)

    if not payment:
        return jsonify({'error': 'Payment not found'}), 404

    return jsonify(payment), 200

# Cleanup old idempotency keys (run periodically)
def cleanup_old_keys():
    """Remove idempotency keys older than 24 hours"""
    cutoff = datetime.now() - timedelta(hours=24)

    keys_to_remove = [
        key for key, value in idempotency_store.items()
        if value['created_at'] < cutoff
    ]

    for key in keys_to_remove:
        del idempotency_store[key]

    print(f"Cleaned up {len(keys_to_remove)} old idempotency keys")

if __name__ == '__main__':
    # Example usage:
    # curl -X POST http://localhost:5000/api/v1/payments \
    #   -H "Content-Type: application/json" \
    #   -H "Idempotency-Key: req_abc123" \
    #   -d '{"amount": 100.00, "currency": "USD"}'

    app.run(debug=True, port=5000)

Idempotent Kafka Consumer

import json
from kafka import KafkaConsumer

class IdempotentConsumer:
    """Kafka consumer with idempotent message processing"""

    def __init__(self, topic, processed_messages_store):
        self.consumer = KafkaConsumer(
            topic,
            bootstrap_servers=['localhost:9092'],
            enable_auto_commit=False,  # Manual commit after processing
            value_deserializer=lambda m: json.loads(m.decode('utf-8'))
        )
        self.processed_messages = processed_messages_store

    def process_messages(self):
        """Process messages idempotently"""
        for message in self.consumer:
            # Extract message ID (required for idempotence)
            msg_data = message.value
            message_id = msg_data.get('message_id')

            if not message_id:
                print("Warning: Message without ID, processing anyway")
                self._process_message(msg_data)
                self.consumer.commit()
                continue

            # Check if already processed
            if message_id in self.processed_messages:
                print(f"✓ Message {message_id} already processed, skipping")
                self.consumer.commit()  # Commit offset to avoid reprocessing
                continue

            # Process message
            try:
                self._process_message(msg_data)

                # Mark as processed
                self.processed_messages.add(message_id)

                # Commit offset (atomic with marking processed)
                self.consumer.commit()

                print(f"✓ Processed message {message_id}")

            except Exception as e:
                print(f"Error processing message {message_id}: {e}")
                # Don't commit - will retry on restart

    def _process_message(self, msg_data):
        """Business logic (can be non-idempotent internally)"""
        # Example: Increment counter (non-idempotent operation)
        # But overall flow is idempotent due to message ID tracking
        action = msg_data.get('action')

        if action == 'increment_counter':
            counter_name = msg_data['counter']
            # Increment counter...
            print(f"Incrementing counter: {counter_name}")

        elif action == 'send_email':
            recipient = msg_data['recipient']
            # Send email...
            print(f"Sending email to: {recipient}")

# Usage
processed_messages = set()  # In production: Use Redis/DB
consumer = IdempotentConsumer('events', processed_messages)
consumer.process_messages()

Related Content

Prerequisites:

• Distributed Systems Basics - Foundation concepts

Related Concepts:

• Exactly-Once Semantics - Delivery guarantees
• Offset Management - Kafka consumer tracking
• Producer Acknowledgments - Message durability

Used In Systems:

• Stripe API: Idempotency keys for payments
• Kafka: Idempotent producer and consumer patterns
• REST APIs: HTTP PUT/DELETE idempotent semantics

Explained In Detail:

• Distributed Systems Deep Dive - Idempotence patterns

Quick Self-Check

• Can explain idempotence in 60 seconds?
• Know difference between idempotent and non-idempotent operations?
• Understand how idempotency keys work?
• Can implement idempotent API endpoint?
• Know how at-least-once + idempotence = exactly-once?
• Can design idempotent message processing?