# Delivery Semantics Deep Dive (/docs/deploying-agent-factories-in-the-cloud/82-event-driven-kafka/11-delivery-semantics) --- type: Document title: "Delivery Semantics Deep Dive" description: "Master at-most-once, at-least-once, and exactly-once delivery semantics in Kafka with practical decision frameworks and idempotent consumer patterns" status: stable order: 8211 chapter: 82 lesson: 11 duration_minutes: 50 generated: by: ag-archive/0.1.0 at: 2026-08-26T00:00:00Z ksor: audience: [public] approval: by: ag-archive/0.1.0 at: 2026-08-26T00:00:00Z trust_tier: unverified build_id: sha256:c8912c3008ae345a7fdc33ace656531ad74f37c065177057c390c7d759f1e970 source_commit: dad4758bb59919c7b9130444778ade062a2dce3e ksor_version: 0.0.47 --- Your Task API publishes a `task.created` event. The notification service consumes it and sends an email. But what happens if the consumer crashes after sending the email but before committing the offset? When it restarts, it re-reads the same event---and sends a second email. Your user receives duplicate notifications. Or worse: what if the consumer commits the offset before processing, crashes mid-processing, and the email never sends at all? These aren't edge cases. In production systems handling millions of events daily, they happen regularly. Network partitions, broker restarts, consumer crashes, and deployment rollouts all create situations where the relationship between message consumption and processing outcome becomes uncertain. This lesson examines the three delivery semantic guarantees Kafka can provide, the trade-offs each involves, and the practical patterns that make at-least-once---the most common choice---work reliably. By the end, you'll have a decision framework for choosing the right semantic for your use case and the implementation knowledge to build consumers that handle duplicates gracefully. ## The Three Delivery Semantics Every distributed messaging system must answer: how many times will a consumer see each message? The answer depends on where failures can occur and how the system handles them. ### At-Most-Once: Fast, But Messages Can Be Lost With at-most-once delivery, a message is delivered zero or one times. If anything fails, the message is lost rather than redelivered. **How it works:** ``` 1. Consumer receives message from Kafka 2. Consumer commits offset BEFORE processing 3. Consumer processes message 4. If processing fails: message is lost (offset already committed) ``` ```python from confluent_kafka import Consumer consumer = Consumer({ 'bootstrap.servers': 'localhost:30092', 'group.id': 'analytics-service', 'enable.auto.commit': True, # Commits before processing completes 'auto.commit.interval.ms': 1000 }) consumer.subscribe(['task-events']) while True: msg = consumer.poll(1.0) if msg is None: continue if msg.error(): continue # Offset already committed automatically # If we crash HERE, message is lost process_analytics(msg.value()) # May never complete ``` **Output:** ``` >>> # Consumer starts, processes some messages Processing analytics for task-123 Processing analytics for task-124 >>> # Consumer crashes during task-125 processing >>> # Consumer restarts >>> # task-125 is SKIPPED (offset was already committed) Processing analytics for task-126 ``` **When at-most-once is acceptable:** | Use Case | Why Loss Is Tolerable | |----------|----------------------| | Real-time metrics | Individual data points don't affect aggregate accuracy | | Activity logging | Missing log entries don't break business logic | | Click tracking | Statistical sampling is acceptable | | Sensor readings | Redundant sensors provide coverage | **When at-most-once is dangerous:** - Payment processing (lost payment = revenue loss) - Order fulfillment (lost order = angry customer) - Audit logging (missing audit = compliance violation) - Inventory updates (lost update = stock discrepancy) ### At-Least-Once: Safe, But Messages May Duplicate With at-least-once delivery, every message is delivered one or more times. If anything fails, the message is redelivered rather than lost. **How it works:** ``` 1. Consumer receives message from Kafka 2. Consumer processes message 3. Consumer commits offset AFTER processing 4. If crash before commit: message is redelivered (duplicate) ``` ```python from confluent_kafka import Consumer consumer = Consumer({ 'bootstrap.servers': 'localhost:30092', 'group.id': 'notification-service', 'enable.auto.commit': False, # Manual commit after processing 'auto.offset.reset': 'earliest' }) consumer.subscribe(['task-events']) while True: msg = consumer.poll(1.0) if msg is None: continue if msg.error(): continue # Process FIRST send_notification(msg.value()) # Commit AFTER processing succeeds # If we crash before this line, message is redelivered consumer.commit(message=msg) ``` **Output (normal operation):** ``` >>> # Normal processing Sending notification for task-123 Committed offset 42 Sending notification for task-124 Committed offset 43 ``` **Output (crash and recovery):** ``` >>> # Consumer processes task-125, sends notification Sending notification for task-125 >>> # CRASH before commit >>> # Consumer restarts from last committed offset (43) Sending notification for task-125 # DUPLICATE Committed offset 44 Sending notification for task-126 Committed offset 45 ``` **Why at-least-once is the most common choice:** 1. **No data loss**: Every message is guaranteed to be processed 2. **Simpler than exactly-once**: No transaction coordination required 3. **Duplicates are manageable**: Consumers can be made idempotent 4. **Lower latency**: No transaction overhead ### Exactly-Once: Correct, But Complex and Expensive With exactly-once delivery, every message is delivered exactly one time---no losses, no duplicates. This requires coordinating the consumer's processing with Kafka's offset commits atomically. **How it works:** ``` 1. Consumer receives message from Kafka 2. Consumer begins transaction 3. Consumer processes message (writing results to Kafka or transactional store) 4. Consumer commits offset AND processing results atomically 5. If anything fails: entire transaction rolls back, message is redelivered ``` ```python from confluent_kafka import Consumer, Producer consumer = Consumer({ 'bootstrap.servers': 'localhost:30092', 'group.id': 'order-processor', 'enable.auto.commit': False, 'isolation.level': 'read_committed' # Only read committed messages }) producer = Producer({ 'bootstrap.servers': 'localhost:30092', 'transactional.id': 'order-processor-1', 'enable.idempotence': True }) producer.init_transactions() consumer.subscribe(['orders']) while True: msg = consumer.poll(1.0) if msg is None: continue try: producer.begin_transaction() # Process order and produce result result = process_order(msg.value()) producer.produce('order-results', key=msg.key(), value=result) # Commit offset AND produced message atomically producer.send_offsets_to_transaction( consumer.position(consumer.assignment()), consumer.consumer_group_metadata() ) producer.commit_transaction() except Exception as e: producer.abort_transaction() # Message will be redelivered, but no partial results ``` **Output:** ``` >>> # Exactly-once processing Processing order-789 Transaction committed: order-789 processed, offset committed >>> # Even if crash occurs mid-transaction, no duplicates or losses ``` **The hidden costs of exactly-once:** | Cost | Impact | |------|--------| | **Latency** | Transaction coordination adds 10-50ms per message | | **Throughput** | Fewer messages per second due to coordination overhead | | **Complexity** | Transactional code is harder to write and debug | | **Failure modes** | Transaction timeouts, zombie fencing, coordinator failures | | **Resource usage** | More broker CPU and memory for transaction state | ## The Decision Matrix Which delivery semantic should you choose? The answer depends on your specific requirements: | Factor | At-Most-Once | At-Least-Once | Exactly-Once | |--------|-------------|---------------|--------------| | **Data loss acceptable?** | Yes | No | No | | **Duplicates acceptable?** | N/A | Yes (with idempotency) | No | | **Latency requirement** | Lowest | Low | Higher | | **Implementation complexity** | Simple | Moderate | High | | **Consumer can be idempotent?** | N/A | Yes = use this | N/A | ### Decision Framework Ask these questions in order: **1. Can you lose messages?** - Yes: Use at-most-once (simplest) - No: Continue to question 2 **2. Can your consumer handle duplicates?** - Yes (consumer is idempotent): Use at-least-once - No: Continue to question 3 **3. Can you make your consumer idempotent?** - Yes: Use at-least-once + idempotent consumer (recommended) - No: Use exactly-once (last resort) **Why idempotent consumers are usually better than exactly-once:** Exactly-once in Kafka only works when: - You're reading from Kafka AND writing to Kafka - Or you're using a transactional external store that supports Kafka transactions Most real-world consumers write to databases, call APIs, send emails, or update caches. These operations don't participate in Kafka transactions. Making these consumers idempotent is simpler and more reliable than attempting exactly-once semantics. ## Implementing Idempotent Consumers The key insight: if your consumer can safely process the same message multiple times with the same result, at-least-once becomes as good as exactly-once from a business logic perspective. ### Pattern 1: Deduplication with Idempotency Key Store processed event IDs and check before processing: ```python from confluent_kafka import Consumer import redis import json class IdempotentConsumer: def __init__(self, consumer_config: dict, redis_client: redis.Redis): self.consumer = Consumer(consumer_config) self.redis = redis_client self.ttl_seconds = 86400 * 7 # Keep IDs for 7 days def is_duplicate(self, event_id: str) -> bool: """Check if we've already processed this event.""" key = f"processed:{event_id}" return self.redis.exists(key) > 0 def mark_processed(self, event_id: str) -> None: """Mark event as processed with TTL.""" key = f"processed:{event_id}" self.redis.setex(key, self.ttl_seconds, "1") def process_with_deduplication(self, msg) -> bool: """Process message if not already processed.""" event = json.loads(msg.value().decode()) event_id = event.get('event_id') if not event_id: print(f"Warning: Message missing event_id, processing anyway") return self._do_process(event) if self.is_duplicate(event_id): print(f"Skipping duplicate: {event_id}") return True # Already processed successfully # Process the event success = self._do_process(event) if success: self.mark_processed(event_id) return success def _do_process(self, event: dict) -> bool: """Actual processing logic.""" # Your business logic here print(f"Processing: {event.get('event_id', 'unknown')}") return True ``` **Output:** ``` >>> # First delivery Processing: evt-123 >>> # Duplicate delivery (after crash/restart) Skipping duplicate: evt-123 >>> # New message Processing: evt-124 ``` ### Pattern 2: Database Upsert with Natural Key Use database constraints to prevent duplicates: ```python from confluent_kafka import Consumer import psycopg2 import json def process_task_event(msg, db_conn): """Process task event with upsert for idempotency.""" event = json.loads(msg.value().decode()) with db_conn.cursor() as cur: # Upsert: Insert or update based on task_id # If task_id already exists, this updates instead of duplicating cur.execute(""" INSERT INTO tasks (task_id, title, status, updated_at) VALUES (%(task_id)s, %(title)s, %(status)s, NOW()) ON CONFLICT (task_id) DO UPDATE SET title = EXCLUDED.title, status = EXCLUDED.status, updated_at = NOW() """, { 'task_id': event['data']['task_id'], 'title': event['data']['title'], 'status': event['data'].get('status', 'pending') }) db_conn.commit() print(f"Upserted task: {event['data']['task_id']}") return True ``` **Output:** ``` >>> # First delivery Upserted task: task-456 >>> # Duplicate delivery Upserted task: task-456 # Same result, no duplicate row >>> # Verify in database SELECT COUNT(*) FROM tasks WHERE task_id = 'task-456'; count: 1 ``` ### Pattern 3: Conditional Processing with Version Check For state changes, check current state before applying: ```python from confluent_kafka import Consumer import json def process_task_completion(msg, db_conn): """Only complete task if it's currently pending.""" event = json.loads(msg.value().decode()) task_id = event['data']['task_id'] with db_conn.cursor() as cur: # Only update if task is still pending cur.execute(""" UPDATE tasks SET status = 'completed', completed_at = NOW() WHERE task_id = %s AND status = 'pending' RETURNING task_id """, (task_id,)) result = cur.fetchone() db_conn.commit() if result: print(f"Completed task: {task_id}") return True else: print(f"Task already completed or not found: {task_id}") return True # Still success (idempotent) ``` **Output:** ``` >>> # First delivery Completed task: task-789 >>> # Duplicate delivery Task already completed or not found: task-789 >>> # Both commits succeed, no duplicate state change ``` ## Choosing Your Idempotency Strategy | Strategy | Best For | Complexity | Storage Requirement | |----------|----------|------------|---------------------| | **Deduplication key** | Any event type, external API calls | Low | Redis/cache for event IDs | | **Database upsert** | Create/update operations | Low | Natural key in database | | **Version/state check** | State transitions | Medium | State field in database | | **Outbox pattern** | Multi-system coordination | High | Outbox table + CDC | ### When Each Strategy Fits **Deduplication key (Pattern 1):** - External API calls (send email, charge payment) - Operations without natural idempotency - High-volume events where storage is cheap **Database upsert (Pattern 2):** - CRUD operations on entities with unique IDs - Simple inserts or updates - When the database is the source of truth **Version/state check (Pattern 3):** - State machine transitions - When duplicate transitions should be no-ops - Audit-sensitive operations ## Collaborating on Delivery Strategy When designing your event processing strategy, you might start with a simple question and discover the nuances through exploration. **Your initial question:** "My notification service sends emails when tasks are created. How do I prevent duplicate emails?" **Exploring the problem:** This is actually two separate concerns: 1. **Consumer reliability**: Ensuring you don't miss events 2. **Idempotency**: Ensuring you don't send duplicate emails For consumer reliability, use at-least-once (commit after processing). For idempotency, you need to track which emails you've already sent. **Discovering the implementation:** A simple approach uses Redis to track sent notifications: ```python def send_notification_idempotently(event_id: str, email: str, message: str) -> bool: """Send notification only if not already sent.""" key = f"notification:sent:{event_id}" # Check if already sent if redis_client.exists(key): return True # Already handled # Send the email success = email_service.send(email, message) if success: # Mark as sent with 7-day TTL redis_client.setex(key, 86400 * 7, "sent") return success ``` **Refining the approach:** But what if sending the email succeeds and Redis fails? You'd send the email again on the next attempt. For truly idempotent email sending, you might need: 1. Email service that accepts idempotency keys (like Stripe's API) 2. Or database transaction that commits email ID before sending 3. Or acceptance that rare duplicates are better than complex infrastructure **What emerged from this exploration:** - At-least-once + idempotent consumer is the standard pattern - Idempotency can be implemented at different layers - Perfect idempotency requires careful thought about failure modes - Sometimes "good enough" idempotency beats "perfect" complexity --- ## Reflect on Your Skill You built a `kafka-events` skill in Lesson 0. Test and improve it based on what you learned. ### Test Your Skill ``` Using my kafka-events skill, configure a consumer for exactly-once delivery semantics in a payment processing scenario. Does my skill explain idempotent consumers, transactional reads, and deduplication strategies? ``` ### Identify Gaps Ask yourself: - Did my skill explain the difference between at-least-once, at-most-once, and exactly-once? - Did it show when each delivery semantic is appropriate? ### Improve Your Skill If you found gaps: ``` My kafka-events skill is missing delivery semantics (at-least-once, exactly-once, idempotent consumers). Update it to include how to achieve exactly-once processing and when the trade-offs are worth it. ``` --- ## Try With AI Apply these delivery semantics to your own event processing scenarios. **Setup:** Open Claude Code or your preferred AI assistant in your Kafka project directory. --- **Prompt 1: Analyze Your Processing Requirements** ``` I'm building an order processing system with these events: - OrderCreated: Must create order in database - PaymentReceived: Must update order status and trigger fulfillment - OrderShipped: Must send shipping notification email For each event, help me determine: 1. Can the consumer be made idempotent? How? 2. What delivery semantic should I use? 3. What's the deduplication strategy? 4. What happens if processing fails halfway? Show me the consumer implementation for PaymentReceived with idempotency. ``` **What you're learning:** Mapping real business requirements to delivery semantics and implementing appropriate idempotency patterns for each event type. --- **Prompt 2: Design Idempotency for External API Calls** ``` My consumer calls an external payment API that doesn't support idempotency keys. The flow is: 1. Receive payment.requested event 2. Call external API to charge card 3. Publish payment.completed event 4. Commit offset If my consumer crashes after step 2 but before step 3, I'll charge the card twice when the message is redelivered. How do I make this idempotent? Consider: - I can't modify the external API - I have access to PostgreSQL and Redis - I need an audit trail of all payment attempts ``` **What you're learning:** Handling idempotency for external systems that don't natively support it---a common real-world challenge. --- **Prompt 3: Evaluate Exactly-Once vs Idempotent Consumer** ``` My team is debating whether to implement exactly-once semantics for our order processing pipeline. We currently use at-least-once with idempotent consumers. Our pipeline: 1. Read from 'orders' topic 2. Validate order 3. Write to 'validated-orders' topic 4. Write to 'inventory-reservations' topic Arguments for exactly-once: - "It's cleaner, no deduplication logic needed" - "Kafka supports it natively" Arguments against: - "More complexity" - "Higher latency" Help me analyze this decision. What questions should I ask? What are the hidden costs of each approach for THIS specific use case? ``` **What you're learning:** Critical evaluation of exactly-once vs at-least-once trade-offs for a specific architecture, not just theoretical understanding. --- **Safety Note:** Test your idempotency logic by deliberately causing failures (kill consumer mid-processing, simulate network partitions). Idempotency bugs often only surface under failure conditions that are hard to reproduce in development.