Running an Apache Kafka cluster setup on your own hardware gives you full control over throughput, retention, and cost. This guide walks through a complete Apache Kafka cluster setup in KRaft mode using Docker Compose on a dedicated server.

Overview of the Kafka Cluster Setup Process

Apache Kafka is a distributed event-streaming platform built for high-throughput, durable, ordered log storage. ZooKeeper has been removed entirely from Kafka 4.0, and the cluster’s metadata is now managed internally through KRaft (Kafka Raft), which makes deployment lighter and failure recovery faster.

In this guide, you will deploy:

• Three nodes acting as both controller and broker, working together to agree on cluster metadata.
• One shared cluster ID, with each node’s storage set up to match it.
• Topics copied across all three brokers (replication factor 3) and split into partitions for parallel processing.
• Login-based security (SASL/PLAIN) plus access rules (ACLs) so each user can only produce or consume specific topics.
• A final test using the console producer and consumer to confirm everything works.

Prerequisites and Server Requirements

Before starting, prepare a dedicated server or three for a fully isolated multi-host cluster with the following minimum specs for a lightweight production setup:

• 4+ vCPUs and 8 GB+ RAM per broker.
• NVMe or SSD storage.
• Ubuntu 22.04 or 24.04 LTS.
• Docker Engine 24+ and the Docker Compose plugin.
• Open ports 9092–9094 and 19092–19094 between nodes if running on separate hosts.

Note: For testing, you can run all three broker containers on one server using Docker’s internal network. For production, put each broker on its own separate server. This way, if one server fails, the whole cluster doesn’t go down with it.

We assumed you have installed Docker and Docker Compose. Verify your installation:

docker --version
docker compose version

Create a working directory for the project:

mkdir -p ~/kafka-cluster/{data/broker1,data/broker2,data/broker3,secrets}
cd ~/kafka-cluster

1. Apache Kafka Cluster Setup: Node and Network Planning

For this Apache Kafka cluster setup, each node runs both the controller and broker roles (combined mode), which is fully supported in KRaft and simplifies operations for small-to-mid clusters. Each node needs:

• A unique node.id (1, 2, 3).
• A shared cluster.id generated once and reused everywhere.
• A controller.quorum.voters string listing all three controller endpoints.
• Separate the internal, controller, and external listener ports so brokers can communicate with each other and clients can reach them from outside Docker’s network.

First, generate the shared cluster ID:

docker run --rm apache/kafka:4.3.1 /opt/kafka/bin/kafka-storage.sh random-uuid

Copy the output; you will reuse it as CLUSTER_ID in the next step for the Compose file.

Example output:

q1Sh9_ISia_zwGINzRvyQ

2. Create Docker Compose File for the Kafka Cluster

The Compose file below is the backbone of this Apache Kafka cluster setup. Everything from replication to security is connected through the environment variables in this one file. It sets KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL for the Raft controller channel, defines a dedicated PLAIN_SASL_JAAS_CONFIG entry per listener, and mounts a shared secrets folder into every broker so the CLI tools can authenticate.

From the Kafka cluster directory, create the Docker Compose file:

nano docker-compose.yml

Add this content to the file; this defines a 3-broker, 3-controller KRaft cluster using the official apache/kafka image:

x-kafka-env: &kafka-env
  CLUSTER_ID: "q1Sh9_ISia_zwGINzRvyQ"
  KAFKA_PROCESS_ROLES: "broker,controller"
  KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "CONTROLLER:SASL_PLAINTEXT,INTERNAL:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT"
  KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka1:9093,2@kafka2:9093,3@kafka3:9093"
  KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
  KAFKA_SASL_ENABLED_MECHANISMS: "PLAIN"
  KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: "PLAIN"
  KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: "PLAIN"

  KAFKA_LISTENER_NAME_CONTROLLER_PLAIN_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret" user_admin="admin-secret";'
  KAFKA_LISTENER_NAME_INTERNAL_PLAIN_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret" user_admin="admin-secret" user_producer="producer-secret" user_consumer="consumer-secret";'
  KAFKA_LISTENER_NAME_EXTERNAL_PLAIN_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret" user_admin="admin-secret" user_producer="producer-secret" user_consumer="consumer-secret";'

  KAFKA_AUTHORIZER_CLASS_NAME: "org.apache.kafka.metadata.authorizer.StandardAuthorizer"
  KAFKA_SUPER_USERS: "User:admin"
  KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "false"
  KAFKA_DEFAULT_REPLICATION_FACTOR: 3
  KAFKA_MIN_INSYNC_REPLICAS: 2
  KAFKA_NUM_PARTITIONS: 6
  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
  KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
  KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2

x-kafka-common: &kafka-common
  image: apache/kafka:4.3.1
  networks: [kafka-net]

services:
  kafka1:
    <<: *kafka-common
    container_name: kafka1
    hostname: kafka1
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: 1
      KAFKA_LISTENERS: "INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:9094"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka1:9092,EXTERNAL://<DEDICATED_SERVER_IP>:19094"
    ports:
      - "19094:9094"
    volumes:
      - ./data/broker1:/var/lib/kafka/data
      - ./secrets:/etc/kafka/secrets:ro

  kafka2:
    <<: *kafka-common
    container_name: kafka2
    hostname: kafka2
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: 2
      KAFKA_LISTENERS: "INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:9094"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka2:9092,EXTERNAL://<DEDICATED_SERVER_IP>:29094"
    ports:
      - "29094:9094"
    volumes:
      - ./data/broker2:/var/lib/kafka/data
      - ./secrets:/etc/kafka/secrets:ro

  kafka3:
    <<: *kafka-common
    container_name: kafka3
    hostname: kafka3
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: 3
      KAFKA_LISTENERS: "INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:9094"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka3:9092,EXTERNAL://<DEDICATED_SERVER_IP>:39094"
    ports:
      - "39094:9094"
    volumes:
      - ./data/broker3:/var/lib/kafka/data
      - ./secrets:/etc/kafka/secrets:ro

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    depends_on: [kafka1, kafka2, kafka3]
    ports:
      - "8080:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: "dedicated-cluster"
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka1:9092,kafka2:9092,kafka3:9092"
      KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: "SASL_PLAINTEXT"
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: "PLAIN"
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";'
    networks: [kafka-net]

networks:
  kafka-net:
    driver: bridge

Replace <DEDICATED_SERVER_IP> with your server’s public or private IP in all three KAFKA_ADVERTISED_LISTENERS lines with your server’s actual IP address.

3. Launch the Kafka Cluster

This is the moment your Apache Kafka cluster setup actually comes alive, so watch the logs closely for formatting and quorum-election messages.

With the compose file and JAAS credentials in place, bring the cluster up:

docker compose up -d
docker compose ps
docker compose logs -f kafka1

Wait until all three logs show Kafka Server started. Since every node uses the same CLUSTER_ID, the official apache/kafka image automatically formats the storage on first boot; you don’t need to run kafka-storage.sh format manually.

Note: If you deploy Kafka directly on the host without Docker, you must run the format step yourself before starting each node:

KAFKA_CLUSTER_ID="q1Sh9_ISia_zwGINzRvyQ"
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

Confirm the secrets folder is mounted and visible inside the container:

docker exec -it kafka1 ls -la /etc/kafka/secrets/

4. Create Replicated Topics with Custom Partitions

With the Apache Kafka cluster setup running, the next step is defining how data is replicated and partitioned across brokers.

Replication and partitioning control how durable and fast your Apache Kafka cluster setup is. A replication factor of 3 means every partition’s data is copied to all three brokers, so the cluster keeps working even if two brokers fail. Setting min.insync.replicas=2 adds extra safety; at least two brokers must confirm a write before Kafka tells the producer it succeeded.

At this point, you must create an admin client config for authenticated CLI access. This file lives on your host and is readable inside every broker container through the ./secrets:/etc/kafka/secrets:ro mount already defined in the Compose file:

cat > secrets/admin.properties <<'EOF'
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
EOF

Now create a topic with 6 partitions and a replication factor of 3:

docker exec -it kafka1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka1:9092 \
  --command-config /etc/kafka/secrets/admin.properties \
  --create --topic events.orders \
  --partitions 6 --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000

List and describe topics to confirm partition distribution across brokers:

docker exec -it kafka1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka1:9092 --command-config /etc/kafka/secrets/admin.properties \
  --describe --topic events.orders

You should see six partitions, each with a leader and two in-sync replicas spread across kafka1, kafka2, and kafka3. This distribution is exactly what makes an Apache Kafka cluster setup flexible. If a broker goes down, Kafka automatically chooses a new leader for the partitions it hosted.

For AI and log-pipeline use cases, create a couple more topics reflecting real workloads:

docker exec -it kafka1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka1:9092 --command-config /etc/kafka/secrets/admin.properties \
  --create --topic ai.embeddings.jobs --partitions 12 --replication-factor 3

docker exec -it kafka1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka1:9092 --command-config /etc/kafka/secrets/admin.properties \
  --create --topic app.logs.raw --partitions 3 --replication-factor 3

More partitions mean more consumers can read from a topic at the same time. That’s why ai.embeddings.jobs uses 12 partitions; it lets multiple workers process embedding or vector database jobs in parallel, which helps if you’re scaling AI model-serving alongside this cluster.

5. Secure the Kafka Cluster with ACLs

Logging in isn’t the same as having permission. Without ACLs, anyone with valid SASL credentials could read or write to any topic. ACLs fix this by defining exactly what each user is allowed to do. Since StandardAuthorizer and allow.everyone.if.no.acl.found=false are already set in the Compose file, no one can touch a topic until you grant them access.

Grant a producer principal write access to events.orders:

docker exec -it kafka1 /opt/kafka/bin/kafka-acls.sh \
  --bootstrap-server kafka1:9092 --command-config /etc/kafka/secrets/admin.properties \
  --add --allow-principal User:producer \
  --operation WRITE --operation DESCRIBE --operation CREATE \
  --topic events.orders

Grant a consumer principal read access, plus group access for its consumer group:

docker exec -it kafka1 /opt/kafka/bin/kafka-acls.sh \
  --bootstrap-server kafka1:9092 --command-config /etc/kafka/secrets/admin.properties \
  --add --allow-principal User:consumer \
  --operation READ --operation DESCRIBE \
  --topic events.orders

docker exec -it kafka1 /opt/kafka/bin/kafka-acls.sh \
  --bootstrap-server kafka1:9092 --command-config /etc/kafka/secrets/admin.properties \
  --add --allow-principal User:consumer \
  --operation READ --group orders-consumer-group

List current ACLs to confirm:

docker exec -it kafka1 /opt/kafka/bin/kafka-acls.sh \
  --bootstrap-server kafka1:9092 --command-config /etc/kafka/secrets/admin.properties --list

This is one of the biggest differences you’ll manage in an Apache Kafka cluster setup compared to simpler brokers. Every producer and consumer should only get the exact permissions it needs, nothing more.

6. Test Kafka Cluster with Producer and Consumer

Create client config files matching each principal’s credentials:

cat > secrets/producer.properties <<'EOF'
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="producer" password="producer-secret";
EOF

cat > secrets/consumer.properties <<'EOF'
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="consumer" password="consumer-secret";
EOF

Since the ./secrets folder is mounted read-only into every broker container, these files are immediately available at /etc/kafka/secrets/ inside kafka1, kafka2, and kafka3.

Produce a few test messages:

docker exec -it kafka1 /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server kafka1:9092 --topic events.orders \
  --producer.config /etc/kafka/secrets/producer.properties

Type a few lines and press Enter after each, then press Ctrl+C to exit.

In a second terminal, consume them:

docker exec -it kafka2 /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server kafka2:9092 --topic events.orders \
  --group orders-consumer-group --from-beginning \
  --consumer.config /etc/kafka/secrets/consumer.properties

If you see the messages you typed appear on the consumer side, both authentication and ACL authorization are working correctly, and your Apache Kafka cluster setup is functioning.

As an extra check, kill one broker and confirm the topic is still readable and writable:

docker stop kafka2
docker exec -it kafka1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka1:9092 --command-config /etc/kafka/secrets/admin.properties \
  --describe --topic events.orders
docker start kafka2

This proves replication and leader failover actually work.

Kafka vs RabbitMQ for AI and Data Pipelines

Once your Apache Kafka cluster setup is live, the next practical question is whether every workload actually belongs on Kafka rather than a simpler queue.

Both are message brokers, but they’re built for different jobs; picking the wrong one can slow down your whole pipeline. Kafka keeps an ordered log you can replay, which is great when several consumers all need to read the same events independently, or when you need to reprocess old data to retrain a model. RabbitMQ is better at quickly routing tasks to the right place, where each message is handled once and then discarded.

Aspect	Kafka	RabbitMQ
How it stores messages	Keeps a log you can replay later	Deletes messages once they’re acknowledged
Speed	Handles millions of messages per second across a cluster	Fast, but each node handles less than Kafka
Message order	Guaranteed within each partition	Guaranteed per queue, trickier with multiple consumers
Best used for	Event streaming, logs, ML pipelines, audit trails	Task queues, request/response jobs, complex routing
How hard to run	Harder, partitions, replication, and quorum to manage	Easier, simpler setup and mental model
Replaying old data	Built-in, just use offsets	Not built-in, needs extra tools

If you mainly need background jobs or service-to-service requests, use RabbitMQ instead. Our durable queue guide with Docker Compose shows you how to set it up. Kafka and RabbitMQ are not direct replacements; choose the one that fits how your data moves.

Conclusion

At this point, you have a complete Apache Kafka cluster setup with KRaft, replication, partitions, SASL/PLAIN security, ACLs, and verified producer/consumer tests.

For real production traffic, run each broker on a separate dedicated server with NVMe storage, so replication and log persistence never compete with another tenant’s I/O, and you can scale brokers independently as event volume grows.

We hope you enjoy this guide.

For deeper details on KRaft configuration, controller quorums, and cluster operations, see the official Apache Kafka KRaft documentation.

FAQs