# Monitoring and Debugging Kafka (/docs/deploying-agent-factories-in-the-cloud/82-event-driven-kafka/19-monitoring-debugging)

---
type: Document
title: "Monitoring and Debugging Kafka"
description: "Master consumer lag diagnosis, use Kafka CLI tools for troubleshooting, and configure alerting thresholds for production reliability"
status: stable
order: 8219
chapter: 82
lesson: 19
duration_minutes: 50
generated:
  by: ag-archive/0.1.0
  at: 2026-08-26T00:00:00Z
ksor:
  audience: [public]
  approval:
    by: ag-archive/0.1.0
    at: 2026-08-26T00:00:00Z
trust_tier: unverified
build_id: sha256:c8912c3008ae345a7fdc33ace656531ad74f37c065177057c390c7d759f1e970
source_commit: dad4758bb59919c7b9130444778ade062a2dce3e
ksor_version: 0.0.47
---

It's 3 AM. Your pager goes off. The order processing system stopped sending confirmation emails two hours ago. You check the notification service logs—no errors. The service is running, polling Kafka, and processing messages. So where are the orders?

You discover the consumer group has 47,000 messages of lag on one partition. Those orders are sitting in Kafka, unprocessed. Your consumer has been processing, but slower than the incoming rate. For two hours, the gap widened silently until a customer complained.

This scenario illustrates why Kafka monitoring isn't optional—it's your early warning system. In this lesson, you'll learn to monitor consumer lag, inspect topics and consumer groups with CLI tools, diagnose common failures, and configure alerts that catch problems before customers do.

## Consumer Lag: The Most Important Metric

Consumer lag is the difference between where producers are writing (the log-end offset) and where your consumer has processed (the current offset). It tells you whether your consumer is keeping up with the production rate.

```
Partition 0:
  Log-end offset (latest):  10,000
  Consumer offset:           8,500
  LAG = 10,000 - 8,500 = 1,500 messages behind
```

**Why lag matters more than throughput:**

| Metric | What It Tells You |
|--------|-------------------|
| **Messages/second** | How fast you're processing right now |
| **Consumer lag** | Whether you're processing faster than producers write |
| **Lag trend** | Whether you're falling behind, catching up, or stable |

A consumer processing 1,000 msg/sec sounds fast—until you realize producers are writing 1,200 msg/sec. Your lag grows by 200 messages every second. Within an hour, you're 720,000 messages behind.

### Monitoring Lag with kafka-consumer-groups.sh

The primary tool for checking consumer lag is `kafka-consumer-groups.sh`. On a Strimzi cluster, you can execute it inside a Kafka pod:

```bash
# Check lag for a specific consumer group
kubectl exec -it task-events-kafka-0 -n kafka -- \
  /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group notification-service
```

**Output:**
```
GROUP                TOPIC         PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID
notification-service task-created  0          8500            10000           1500            consumer-1-abc123                              /10.244.0.15    consumer-1
notification-service task-created  1          9800            9800            0               consumer-1-abc123                              /10.244.0.15    consumer-1
notification-service task-created  2          7200            9500            2300            consumer-2-def456                              /10.244.0.16    consumer-2
```

**Reading this output:**

| Column | Meaning |
|--------|---------|
| `CURRENT-OFFSET` | Last committed offset for this partition |
| `LOG-END-OFFSET` | Latest message offset in the partition |
| `LAG` | Messages waiting to be processed |
| `CONSUMER-ID` | Which consumer instance owns this partition |
| `HOST` | IP address of the consumer |

From this output, you can see:
- Partition 1 is caught up (lag = 0)
- Partition 0 has 1,500 messages of lag
- Partition 2 has 2,300 messages of lag—the worst performer
- consumer-2 on partition 2 might be slower or handling more complex messages

### Interpreting Lag Patterns

Different lag patterns indicate different problems:

| Pattern | What It Means | Likely Cause |
|---------|---------------|--------------|
| **All partitions have similar, growing lag** | Overall throughput issue | Consumer processing too slow, need to scale or optimize |
| **One partition has much higher lag** | Partition-specific issue | Hot partition (uneven key distribution), slow message type, stuck consumer |
| **Lag spikes then recovers** | Transient issue | Consumer restart, rebalance, temporary slow processing |
| **Lag stays constant and low** | Healthy state | Consumer keeping pace with production |
| **Lag at 0 for all partitions** | Caught up | Healthy, or no messages being produced |

### Lag Alert Thresholds

Setting appropriate thresholds depends on your tolerance for processing delay:

```yaml
# Example alert thresholds for a notification service
alert_rules:
  # Warning: lag growing but not critical yet
  consumer_lag_warning:
    threshold: 1000
    duration: "5m"
    message: "Consumer lag above 1000 for 5 minutes"

  # Critical: significant delay, may miss SLAs
  consumer_lag_critical:
    threshold: 10000
    duration: "2m"
    message: "Consumer lag above 10000 - potential message loss risk"

  # Emergency: approaching retention limit
  consumer_lag_emergency:
    threshold: 100000
    duration: "1m"
    message: "Consumer lag near retention limit - data loss imminent"
```

**Rule of thumb:** Alert when lag exceeds what you can process in 1/3 of your retention period. If retention is 7 days and you process 10,000 msg/hour, alert around 50,000 lag.

## Inspecting Topics with kafka-topics.sh

When troubleshooting, you often need to understand the topic structure—how many partitions, replication factor, and configuration:

```bash
# List all topics
kubectl exec -it task-events-kafka-0 -n kafka -- \
  /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --list
```

**Output:**
```
__consumer_offsets
__strimzi-topic-operator-kstreams-topic-store-changelog
task-completed
task-created
task-updated
```

```bash
# Describe a specific topic
kubectl exec -it task-events-kafka-0 -n kafka -- \
  /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --topic task-created
```

**Output:**
```
Topic: task-created	TopicId: ABC123xyz	PartitionCount: 3	ReplicationFactor: 1	Configs: retention.ms=604800000,cleanup.policy=delete
	Topic: task-created	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
	Topic: task-created	Partition: 1	Leader: 0	Replicas: 0	Isr: 0
	Topic: task-created	Partition: 2	Leader: 0	Replicas: 0	Isr: 0
```

**Key information:**
- **PartitionCount**: 3 partitions (can run up to 3 parallel consumers)
- **ReplicationFactor**: 1 (dev setting—no fault tolerance)
- **Configs**: 7-day retention, delete cleanup policy
- **Leader**: Broker ID handling reads/writes for this partition
- **Isr**: In-Sync Replicas—brokers that have the latest data

### Checking Under-Replicated Partitions

Under-replicated partitions are partitions where one or more replicas have fallen behind the leader. This indicates broker health issues:

```bash
# Find under-replicated partitions
kubectl exec -it task-events-kafka-0 -n kafka -- \
  /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --under-replicated-partitions
```

**Healthy output (no problems):**
```
(empty - no under-replicated partitions)
```

**Unhealthy output:**
```
Topic: task-created	Partition: 0	Leader: 0	Replicas: 0,1,2	Isr: 0,1
Topic: task-created	Partition: 2	Leader: 1	Replicas: 0,1,2	Isr: 1,2
```

This shows:
- Partition 0: Broker 2 is not in ISR (expected 3 replicas, only 2 in sync)
- Partition 2: Broker 0 is not in ISR

**Diagnosing under-replication:**

| Symptom | Likely Cause | Fix |
|---------|--------------|-----|
| One broker missing from all ISRs | Broker down or slow | Check broker pod status, restart if needed |
| Random partitions under-replicated | Network issues | Check pod connectivity, cluster networking |
| All partitions under-replicated | Cluster-wide problem | Check all broker health, disk space, memory |

## Reading Messages with kafka-console-consumer.sh

When debugging, you often need to see what's actually in a topic:

```bash
# Read messages from beginning
kubectl exec -it task-events-kafka-0 -n kafka -- \
  /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic task-created \
  --from-beginning \
  --max-messages 5
```

**Output:**
```
{"id": "task-1", "title": "Buy groceries", "created_at": "2025-01-15T10:00:00Z"}
{"id": "task-2", "title": "Call dentist", "created_at": "2025-01-15T10:01:00Z"}
{"id": "task-3", "title": "Review PR", "created_at": "2025-01-15T10:02:00Z"}
{"id": "task-4", "title": "Deploy to staging", "created_at": "2025-01-15T10:03:00Z"}
{"id": "task-5", "title": "Write tests", "created_at": "2025-01-15T10:04:00Z"}
Processed a total of 5 messages
```

**Useful options:**

```bash
# Read from a specific partition
kubectl exec -it task-events-kafka-0 -n kafka -- \
  /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic task-created \
  --partition 2 \
  --offset 100 \
  --max-messages 3

# Include keys and metadata
kubectl exec -it task-events-kafka-0 -n kafka -- \
  /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic task-created \
  --from-beginning \
  --property print.key=true \
  --property print.timestamp=true \
  --max-messages 3
```

**Output with keys and timestamps:**
```
CreateTime:1705312800000	task-1	{"id": "task-1", "title": "Buy groceries"}
CreateTime:1705312860000	task-2	{"id": "task-2", "title": "Call dentist"}
CreateTime:1705312920000	task-3	{"id": "task-3", "title": "Review PR"}
```

## Common Errors and Troubleshooting

The Kafka ecosystem has specific error patterns. Understanding them speeds up debugging:

| Error | Cause | Fix |
|-------|-------|-----|
| `NOT_ENOUGH_REPLICAS` | ISR count below `min.insync.replicas` | Check broker health; ensure enough brokers are up |
| `COORDINATOR_NOT_AVAILABLE` | Consumer group coordinator not ready | Wait and retry; usually transient during startup |
| `REBALANCE_IN_PROGRESS` | Consumer group is rebalancing | Wait for completion; check for flapping consumers |
| `OFFSET_OUT_OF_RANGE` | Requested offset doesn't exist | Adjust `auto.offset.reset`; offset may have been deleted by retention |
| `UNKNOWN_TOPIC_OR_PARTITION` | Topic doesn't exist | Create topic first; check for typos in topic name |
| `REQUEST_TIMED_OUT` | Broker didn't respond in time | Check broker health, network, or increase timeout |
| `LEADER_NOT_AVAILABLE` | Partition has no leader | Wait for leader election; check broker health |

### Debugging a Slow Consumer

When a consumer is falling behind, use this systematic approach:

**Step 1: Confirm the lag**
```bash
kubectl exec -it task-events-kafka-0 -n kafka -- \
  /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group notification-service
```

**Step 2: Check if lag is growing**
```bash
# Run the describe command twice, 30 seconds apart
# Compare LAG values - if growing, consumer is too slow
```

**Step 3: Check partition distribution**
```
If one partition has much higher lag:
  - Check message key distribution (is one key getting all traffic?)
  - Check if messages on that partition are slower to process
  - Consider repartitioning or rebalancing
```

**Step 4: Check consumer performance**
```python
# Add timing to your consumer
import time

while True:
    msg = consumer.poll(1.0)
    if msg and not msg.error():
        start = time.time()
        process_message(msg)
        duration = time.time() - start

        if duration > 0.1:  # 100ms threshold
            print(f"SLOW: {duration:.2f}s for partition {msg.partition()}")
```

**Step 5: Scale if needed**
```bash
# Check current consumer count
kubectl get pods -l app=notification-service -n kafka

# Scale up if you have fewer consumers than partitions
kubectl scale deployment notification-service --replicas=3 -n kafka
```

## JMX Metrics for Production Monitoring

Kafka exposes detailed metrics via JMX (Java Management Extensions). In production, you'll export these to Prometheus or another monitoring system.

**Key broker metrics:**

| Metric | What It Measures | Alert Threshold |
|--------|------------------|-----------------|
| `kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions` | Count of under-replicated partitions | > 0 for 5 minutes |
| `kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec` | Incoming bytes/second | Depends on capacity |
| `kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec` | Incoming messages/second | Depends on capacity |
| `kafka.network:type=RequestMetrics,name=TotalTimeMs` | Request latency | 99th percentile > 500ms |
| `kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs` | Disk flush latency | > 100ms average |

**Key consumer metrics:**

| Metric | What It Measures | Alert Threshold |
|--------|------------------|-----------------|
| `kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=*,partition=*,name=records-lag` | Per-partition lag | > 10000 for 5 minutes |
| `kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,name=records-consumed-rate` | Consumption rate | Depends on expected rate |
| `kafka.consumer:type=consumer-coordinator-metrics,client-id=*,name=rebalance-latency-avg` | Average rebalance time | > 30 seconds |

### Strimzi Metrics with Prometheus

Strimzi provides built-in support for Prometheus metrics. Enable them in your Kafka resource:

```yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: task-events
spec:
  kafka:
    version: 4.1.1
    metadataVersion: 4.1-IV0
    # ... other config ...
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
```

Then create the metrics ConfigMap:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-metrics
data:
  kafka-metrics-config.yml: |
    lowercaseOutputName: true
    rules:
    - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
        clientId: "$3"
        topic: "$4"
        partition: "$5"
    - pattern: kafka.server<type=(.+), name=(.+)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
```

## Building an Alert Runbook

When alerts fire, you need clear steps. Here's a template runbook:

**Alert: Consumer Lag Critical (> 10,000)**

```
1. CONFIRM the alert
   $ kubectl exec -it task-events-kafka-0 -n kafka -- \
       /opt/kafka/bin/kafka-consumer-groups.sh \
       --bootstrap-server localhost:9092 \
       --describe --group <consumer-group>

2. IDENTIFY the pattern
   - All partitions lagging → Processing too slow overall
   - One partition lagging → Hot partition or stuck consumer

3. CHECK consumer health
   $ kubectl get pods -l app=<consumer-app> -n <namespace>
   $ kubectl logs <consumer-pod> --tail=100

4. CHECK for rebalancing
   Look for "Revoking" or "Assigned" in logs
   Frequent rebalancing = consumers timing out

5. SCALE if needed
   $ kubectl scale deployment <consumer-app> --replicas=<N>
   (only helps if partitions > consumers)

6. IF still lagging after 15 minutes
   - Check for slow external dependencies (DB, API calls)
   - Consider increasing max.poll.records for batch processing
   - Escalate if data loss risk (lag approaching retention)
```

**Alert: Under-Replicated Partitions > 0**

```
1. IDENTIFY which partitions
   $ kubectl exec -it task-events-kafka-0 -n kafka -- \
       /opt/kafka/bin/kafka-topics.sh \
       --bootstrap-server localhost:9092 \
       --describe --under-replicated-partitions

2. CHECK broker status
   $ kubectl get pods -l strimzi.io/cluster=task-events -n kafka
   Look for pods not in Running state

3. CHECK broker logs
   $ kubectl logs task-events-kafka-<N> -n kafka --tail=200
   Look for: disk errors, OOM, connection failures

4. IF broker pod is down
   $ kubectl describe pod task-events-kafka-<N> -n kafka
   Check Events section for failure reason

5. IF broker is slow
   - Check disk usage: df -h on broker
   - Check memory: possible GC pressure
   - Check network: latency between brokers
```

---

## Reflect on Your Skill

You built a `kafka-events` skill in Lesson 0. Test and improve it based on what you learned.

### Test Your Skill

```
Using my kafka-events skill, diagnose a consumer lag issue and identify which consumer is falling behind.
Does my skill show how to use kafka-consumer-groups.sh and interpret lag metrics?
```

### Identify Gaps

Ask yourself:
- Did my skill explain consumer lag metrics and what causes lag growth?
- Did it show how to use Kafka CLI tools for debugging (kafka-topics.sh, kafka-consumer-groups.sh)?

### Improve Your Skill

If you found gaps:

```
My kafka-events skill is missing monitoring and debugging patterns (consumer lag, offset inspection, CLI tools).
Update it to include how to diagnose and resolve common Kafka operational issues.
```

---

## Try With AI

**Setup:** You're on-call and receive an alert about your Kafka cluster.

**Prompt 1: Interpret monitoring output**

```
I'm debugging a Kafka consumer issue. Here's my kafka-consumer-groups output:

GROUP          TOPIC        PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
order-service  orders       0          45000           50000           5000
order-service  orders       1          49500           50000           500
order-service  orders       2          42000           50000           8000
order-service  orders       3          49800           50000           200

I have 2 consumer instances running. What patterns do you see, and what
should I check first? Walk me through a systematic diagnosis.
```

**What you're learning:** AI helps identify asymmetric lag patterns—partition 2 is significantly behind, suggesting either a hot partition, slow message processing, or an issue with the consumer assigned to it.

**Prompt 2: Build a troubleshooting checklist**

```
Create a troubleshooting checklist for this Kafka error:

"NOT_ENOUGH_REPLICAS: Messages are rejected because there are fewer in-sync
replicas than required: 2"

My cluster has 3 brokers and topics with replication.factor=3
and min.insync.replicas=2.

What are all the possible causes and how do I diagnose each one?
```

**What you're learning:** AI walks through ISR mechanics and helps you understand why this error occurs (at least one broker is not in sync), plus diagnostic steps for each scenario.

**Prompt 3: Design alerting for your system**

```
I'm setting up alerting for a Kafka-based event processing system. We have:

- 3 topics: orders (high priority), notifications (medium), analytics (low)
- SLA: orders must be processed within 5 minutes, others within 1 hour
- Retention: 7 days for all topics
- Traffic: orders 1000/min, notifications 5000/min, analytics 50000/min

Help me design alert thresholds for consumer lag on each topic.
Consider: SLA requirements, traffic rates, and what "critical" means for each.
```

**What you're learning:** AI collaborates on translating business SLAs into technical alert thresholds, showing how to differentiate alert severity based on topic priority and processing requirements.

**Safety note:** When running diagnostic commands on production Kafka clusters, use read-only commands (--describe, --list) rather than commands that modify state. Never reset consumer offsets or delete topics without understanding the implications.