Deploying Kafka with Strimzi | Agent Factory Archive

7 min read

You've built the mental model of Kafka architecture. Now it's time to run a real cluster.

In production Kubernetes environments, you don't manually configure Kafka brokers. You use an operator—a Kubernetes-native controller that understands Kafka's operational requirements and manages the cluster lifecycle automatically. The industry standard for Kafka on Kubernetes is Strimzi, a CNCF project with widespread adoption.

This lesson walks you through deploying Kafka on Docker Desktop Kubernetes using Strimzi. By the end, you'll have a running Kafka cluster that you can use throughout this chapter.

Docker Desktop Configuration

Before starting, configure Docker Desktop with enough resources for Kafka:

Open Docker Desktop → Settings → Resources
Configure based on your machine:

Your Machine RAM	Docker Desktop Memory	Docker Desktop CPUs	Swap
8GB	5GB	4	1GB
16GB	8GB	4	1GB
32GB+	12GB+	6+	2GB

Click Apply & Restart

Without this, Kafka pods will crash with OOMKilled or CrashLoopBackOff errors.

Prerequisites Check

Before proceeding, verify your environment is ready:

# Check Docker Desktop Kubernetes is running
kubectl cluster-info

Output:

Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

# Check Helm is installed
helm version

Output:

version.BuildInfo{Version:"v3.16.3", GitCommit:"...", GitTreeState:"clean", GoVersion:"go1.22.7"}

If either command fails, revisit Chapters 79-81 to set up Docker Desktop Kubernetes and Helm.

Understanding Operators and CRDs

Before diving into Strimzi, let's clarify two Kubernetes concepts you'll use throughout this lesson.

What's a CRD?

Kubernetes has built-in resources like Pods and Services. A Custom Resource Definition (CRD) extends Kubernetes with new resource types:

Built-in Resources	Custom Resources (after Strimzi)
`kubectl get pods`	`kubectl get kafka`
`kubectl get services`	`kubectl get kafkatopic`
`kubectl get deployments`	`kubectl get kafkauser`

CRDs let you manage Kafka the same way you manage any Kubernetes resource—with kubectl apply -f.

What's an Operator?

An operator is a Kubernetes-specific pattern: a program that watches your CRDs and takes action to make reality match your desired state.

You apply YAML → Operator sees it → Operator creates/manages resources
        ↑                                      |
        └──────── Continuous loop ─────────────┘

Think of it as a 24/7 expert encoded in software. A Kafka operator knows how to deploy, scale, upgrade, and heal Kafka clusters automatically. Without an operator, you'd write hundreds of lines of YAML and handle all operations manually.

What is Strimzi?

Strimzi is a CNCF open-source project that provides a Kafka operator for Kubernetes. When you install Strimzi, it:

Registers CRDs — Teaches Kubernetes what Kafka, KafkaTopic, KafkaUser resources are
Runs an operator — A controller that watches those CRDs and manages actual Kafka clusters

Instead of writing complex deployment manifests, you describe your Kafka cluster declaratively, and Strimzi handles the operational complexity.

Component	Role
Cluster Operator	Watches Kafka CRDs and manages broker lifecycle
Entity Operator	Contains Topic Operator and User Operator
Topic Operator	Syncs KafkaTopic CRDs to actual Kafka topics
User Operator	Manages KafkaUser CRDs and credentials

The operator pattern means you declare intent ("I want a 3-broker Kafka cluster") and Strimzi figures out how to achieve and maintain it.

Step 1: Install Strimzi Operator

Add the Strimzi Helm repository and install the operator:

# Add Strimzi Helm repository
helm repo add strimzi https://strimzi.io/charts/
helm repo update

Output:

"strimzi" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "strimzi" chart repository
Update Complete. Happy Helming!

# Create namespace for Kafka resources
kubectl create namespace kafka

Output:

namespace/kafka created

# Install Strimzi operator
helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator \
  --namespace kafka

Output:

NAME: strimzi-kafka-operator
LAST DEPLOYED: [timestamp]
NAMESPACE: kafka
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing strimzi-kafka-operator-0.49.1

Step 2: Verify Operator is Running

Wait for the operator pod to be ready:

# Watch operator pod status
kubectl get pods -n kafka -w

Output:

NAME                                        READY   STATUS    RESTARTS   AGE
strimzi-cluster-operator-6d4f5c5b9d-x7j2k   1/1     Running   0          45s

Press Ctrl+C once you see 1/1 Running. The operator is now watching for Kafka CRDs.

Understanding the Two-File Pattern

Before creating files, understand what we're about to build. Strimzi uses two CRDs that work together:

File	CRD Type	What It Defines
`kafka-nodepool.yaml`	KafkaNodePool	The machines — how many nodes, their roles, storage
`kafka-cluster.yaml`	Kafka	The cluster config — Kafka version, listeners, settings

Why two files? Think of it like building a car:

KafkaNodePool = "I want 1 engine that does both driving and steering" (hardware)
Kafka = "Here's how to configure the car" (software settings)

What happens when you apply each:

Step 3: kubectl apply -f kafka-nodepool.yaml
        └─ Strimzi: "Got it, storing node specification. Waiting for cluster..."
           (Nothing runs yet)

Step 4: kubectl apply -f kafka-cluster.yaml
        └─ Strimzi: "Now I have both! Creating:"
           → Pod: task-events-dual-role-0 (actual Kafka broker)
           → Service: task-events-kafka-bootstrap (client connection point)
           → Entity Operator (manages topics/users)

The files are linked by name—the NodePool references which Kafka cluster it belongs to:

# In kafka-nodepool.yaml
labels:
  strimzi.io/cluster: task-events  # ← Links to Kafka named "task-events"

# In kafka-cluster.yaml
metadata:
  name: task-events  # ← The name referenced above

Now let's create each file.

Step 3: Create KafkaNodePool (Dual-Role for Development)

Kafka in KRaft mode has two node roles: controllers (manage metadata) and brokers (handle messages). For development, we combine both roles in a single node to minimize resource usage.

Create a file named kafka-nodepool.yaml:

# kafka-nodepool.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: dual-role
  namespace: kafka
  labels:
    strimzi.io/cluster: task-events
spec:
  replicas: 1
  roles:
    - controller
    - broker
  storage:
    type: ephemeral  # Use persistent-claim for production
  resources:
    requests:
      memory: 512Mi   # Works on 8GB machines
      cpu: 200m
    limits:
      memory: 1Gi     # Increase to 2Gi for 16GB+ machines
      cpu: 500m

Key configuration points:

Field	Value	Purpose
`replicas`	1	Single node for development (use 3+ for production)
`roles`	controller, broker	Combined roles save resources in dev
`storage.type`	ephemeral	Data lost on restart (use persistent-claim for production)
`resources`	requests/limits	Memory and CPU allocation for stability
`strimzi.io/cluster`	task-events	Links this pool to the Kafka cluster

Apply the node pool:

kubectl apply -f kafka-nodepool.yaml

Output:

kafkanodepool.kafka.strimzi.io/dual-role created

Step 4: Create Kafka Cluster (KRaft Mode)

Now create the Kafka cluster that uses the node pool. Create a file named kafka-cluster.yaml:

# kafka-cluster.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: task-events
  namespace: kafka
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 4.1.1
    metadataVersion: 4.1-IV0
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: external
        port: 9094
        type: nodeport
        tls: false
        configuration:
          bootstrap:
            nodePort: 30092
          brokers:
            - broker: 0
              nodePort: 30093
              advertisedHost: localhost   # Required for Docker Desktop
              advertisedPort: 30093
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      auto.create.topics.enable: false  # Production best practice
  entityOperator:
    topicOperator: {}
    userOperator: {}

Key configuration points:

Field	Value	Purpose
`strimzi.io/kraft: enabled`	-	Uses KRaft mode (no ZooKeeper)
`strimzi.io/node-pools: enabled`	-	Uses KafkaNodePool for node config
`version`	4.1.1	Kafka version (requires Strimzi 0.49+)
`metadataVersion`	4.1-IV0	KRaft metadata format version
`listeners.plain`	port 9092, internal	For pod-to-pod communication inside K8s
`listeners.external`	nodeport 30092, advertisedHost: localhost	For your local machine to connect (Docker Desktop)
`replication.factor: 1`	-	Single replica for dev (use 3 for production)
`auto.create.topics.enable`	false	Prevents accidental topic creation
`entityOperator`	topicOperator, userOperator	Enable declarative topic/user management

Apply the cluster:

kubectl apply -f kafka-cluster.yaml

Output:

kafka.kafka.strimzi.io/task-events created

Step 5: Wait for Cluster Ready

The Strimzi operator will now create the Kafka pods. This takes 1-2 minutes on first deployment:

# Watch all Kafka-related pods
kubectl get pods -n kafka -w

Output (after ~90 seconds):

NAME                                        READY   STATUS    RESTARTS   AGE
strimzi-cluster-operator-6d4f5c5b9d-x7j2k   1/1     Running   0          5m
task-events-dual-role-0                     1/1     Running   0          90s
task-events-entity-operator-7f4d8b9c-2kj3l  2/2     Running   0          45s

Press Ctrl+C once all pods show Running.

Verify the Kafka cluster status:

kubectl get kafka -n kafka

Output:

NAME          DESIRED KAFKA REPLICAS   DESIRED ZK REPLICAS   READY   METADATA STATE   WARNINGS
task-events   1                                              True    KRaft

The READY: True and METADATA STATE: KRaft confirm your cluster is operational.

Step 6: Create Topics via KafkaTopic CRD

With the Entity Operator running, you can manage topics declaratively. Create a file named kafka-topic.yaml:

# kafka-topic.yaml
apiVersion: kafka.strimzi.io/v1
kind: KafkaTopic
metadata:
  name: task-created
  namespace: kafka
  labels:
    strimzi.io/cluster: task-events
spec:
  partitions: 3
  replicas: 1
  config:
    retention.ms: "604800000"  # 7 days
    cleanup.policy: delete

Key configuration points:

Field	Value	Purpose
`partitions`	3	Parallel processing units (scale consumers up to 3)
`replicas`	1	Single copy for dev (use 3 for production)
`retention.ms`	604800000	Keep messages for 7 days
`cleanup.policy`	delete	Remove old segments (vs "compact" for changelogs)

Apply the topic:

kubectl apply -f kafka-topic.yaml

Output:

kafkatopic.kafka.strimzi.io/task-created created

Verify the topic was created:

kubectl get kafkatopics -n kafka

Output:

NAME           CLUSTER       PARTITIONS   REPLICATION FACTOR   READY
task-created   task-events   3            1                    True

How Topic Operator Works

The Topic Operator watches for KafkaTopic resources and syncs them to the actual Kafka cluster:

┌─────────────────────────────────────────────────────────────┐
│  You apply YAML                                             │
│    └─ kubectl apply -f kafka-topic.yaml                     │
├─────────────────────────────────────────────────────────────┤
│  Kubernetes API stores KafkaTopic CR                        │
├─────────────────────────────────────────────────────────────┤
│  Topic Operator watches and detects new CR                  │
│    └─ Creates topic in Kafka cluster                        │
│    └─ Updates CR status (READY: True)                       │
├─────────────────────────────────────────────────────────────┤
│  Topic exists in Kafka, managed via GitOps                  │
└─────────────────────────────────────────────────────────────┘

This declarative approach means your topic configuration is version-controlled and reproducible across environments.

Verify Kafka is Accessible

Test connectivity by running a temporary pod with the Kafka CLI:

# Start a temporary pod with Kafka tools
kubectl run kafka-test -n kafka --rm -it \
  --image=quay.io/strimzi/kafka:0.49.1-kafka-4.1.1 \
  --restart=Never \
  -- bin/kafka-topics.sh --bootstrap-server task-events-kafka-bootstrap:9092 --list

Output:

task-created

This confirms:

The Kafka broker is accepting connections
The task-events-kafka-bootstrap service routes to the broker
The task-created topic exists

Understanding the Bootstrap Service

Strimzi creates a Kubernetes Service for client connections:

kubectl get svc -n kafka | grep bootstrap

Output:

task-events-kafka-bootstrap   ClusterIP   10.96.45.123   <none>   9091/TCP,9092/TCP,9093/TCP   5m

Clients connect to task-events-kafka-bootstrap:9092 (or port 9093 for TLS). This service load-balances across all broker pods, providing a stable endpoint regardless of pod restarts.

Development vs Production Configuration

The configuration in this lesson is optimized for learning on Docker Desktop. Production requires different settings:

Setting	Development	Production
Node pool replicas	1	3+ (controller: 3, broker: 3+)
Storage type	ephemeral	persistent-claim
Replication factor	1	3
min.insync.replicas	1	2
Separate controller pool	No (dual-role)	Yes

Lesson 18 covers production Strimzi configuration in detail.

Cleanup (Optional)

If you need to remove the Kafka cluster:

# Delete in reverse order
kubectl delete kafkatopic task-created -n kafka
kubectl delete kafka task-events -n kafka
kubectl delete kafkanodepool dual-role -n kafka
helm uninstall strimzi-kafka-operator -n kafka
kubectl delete namespace kafka

For now, keep the cluster running—you'll use it in the remaining lessons.

Reflect on Your Skill

You built a kafka-events skill in Lesson 0. Test and improve it based on what you learned.

Test Your Skill

Using my kafka-events skill, generate a Strimzi Kafka Custom Resource for a development cluster on Docker Desktop.
Does my skill produce valid Kafka CRDs with KRaft mode configuration?

Identify Gaps

Ask yourself:

Did my skill include Strimzi operator deployment patterns?
Did it explain the difference between KRaft mode and ZooKeeper mode?

Improve Your Skill

If you found gaps:

My kafka-events skill is missing Strimzi deployment patterns (Kafka CRDs, KRaft vs ZooKeeper).
Update it to include how to deploy production-ready Kafka on Kubernetes using Strimzi.

Try With AI

You've deployed Kafka using Strimzi's operator pattern. Now explore how this fits into broader infrastructure decisions.

Production Configuration Review

I just deployed Kafka on Docker Desktop Kubernetes using Strimzi with:
- Single node (dual-role: controller + broker)
- Ephemeral storage
- Replication factor 1

Review my configuration for production readiness:
1. What specific changes would I need for a production deployment?
2. For a 3-broker production cluster, what would the KafkaNodePool
   and Kafka CRDs look like?
3. What monitoring should I add to detect problems before they
   cause outages?

What you're learning: Production Kafka requires careful sizing and redundancy configuration. AI can help you understand the gap between development convenience and production reliability.

Debugging Operator Issues

My Strimzi operator is running but my Kafka cluster isn't becoming ready.
kubectl get kafka shows READY: False.

Help me debug:
1. What logs should I check first?
2. What are common reasons Kafka clusters fail to start?
3. How do I interpret Strimzi's status conditions?

What you're learning: Operators provide status information through CRD conditions. Understanding how to read operator status helps you debug declarative infrastructure.

Alternative Topic Configurations

I created a topic with partitions: 3 and replicas: 1.

Help me understand:
1. How do I decide how many partitions a topic needs?
2. What's the relationship between partitions and consumer scaling?
3. When should I use cleanup.policy: compact vs delete?

What you're learning: Topic configuration directly impacts performance and semantics. The right settings depend on your use case—high-throughput streaming vs change data capture vs event sourcing.

Safety note: When modifying production Kafka configurations, always test changes in a staging environment first. Incorrect replication or partition settings can cause data loss.

Core Concept

Strimzi provides Kubernetes-native Kafka deployment through Custom Resource Definitions (CRDs), allowing declarative cluster configuration that operators manage automatically, using KRaft mode for simplified architecture without ZooKeeper.

Key Mental Models

Operator Pattern: Declare intent ("I want a 3-broker cluster"), operator achieves and maintains it
KafkaNodePool: Define node roles (controller/broker) and resources separately from cluster config
Topic Operator: KafkaTopic CRDs automatically synced to actual Kafka topics (GitOps-friendly)
Bootstrap Service: Stable endpoint for client connections regardless of pod restarts

Critical Patterns

Helm install: helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator -n kafka
KafkaNodePool with dual-role for development (controller + broker in one node)
Kafka CRD with strimzi.io/kraft: enabled annotation for KRaft mode
KafkaTopic CRD for declarative, version-controlled topic management

AI Collaboration Keys

Review development configuration for production gaps (replicas, storage, replication factor)
Debug operator issues using CRD status conditions and operator logs
Understand partition count and cleanup policy trade-offs for different use cases

Common Mistakes

Using ephemeral storage in production (data lost on restart)
Replication factor of 1 in production (no fault tolerance)
Forgetting to wait for cluster Ready status before creating topics
Not understanding the Topic Operator sync mechanism

Connections

Builds on: Lesson 3 - Kafka mental model and architecture
Leads to: Lesson 5 - Your First Producer