How to Set Up an Apache Kafka Cluster for Event Streaming on a Dedicated Server

Running an Apache Kafka cluster setup on your own hardware gives you full control over throughput, retention, and cost. This guide walks through a complete Apache Kafka cluster setup in KRaft mode using Docker Compose on a dedicated server.

Apache Kafka is a distributed event-streaming platform built for high-throughput, durable, ordered log storage. ZooKeeper has been removed entirely from Kafka 4.0, and the cluster's metadata is now managed internally through KRaft (Kafka Raft), which makes deployment lighter and failure recovery faster.

In this guide, you will deploy:

• Three nodes acting as both controller and broker, working together to agree on cluster metadata.
• One shared cluster ID, with each node's storage set up to match it.
• Topics copied across all three brokers (replication factor 3) and split into partitions for parallel processing.
• Login-based security (SASL/PLAIN) plus access rules (ACLs) so each user can only produce or consume specific topics.
• A final test using the console producer and consumer to confirm everything works.

Before starting, prepare a dedicated server or three for a fully isolated multi-host cluster with the following minimum specs for a lightweight production setup:

• 4+ vCPUs and 8 GB+ RAM per broker.
• NVMe or SSD storage.
• Ubuntu 22.04 or 24.04 LTS.
• Docker Engine 24+ and the Docker Compose plugin.
• Open ports 9092–9094 and 19092–19094 between nodes if running on separate hosts.

Note: For testing, you can run all three broker containers on one server using Docker's internal network. For production, put each broker on its own separate server. This way, if one server fails, the whole cluster doesn't go down with it.

We assumed you have installed Docker and Docker Compose. Verify your installation:

Create a working directory for the project:

For this Apache Kafka cluster setup, each node runs both the controller and broker roles (combined mode), which is fully supported in KRaft and simplifies operations for small-to-mid clusters. Each node needs:

• A unique node.id (1, 2, 3).
• A shared cluster.id generated once and reused everywhere.
• A controller.quorum.voters string listing all three controller endpoints.
• Separate the internal, controller, and external listener ports so brokers can communicate with each other and clients can reach them from outside Docker's network.

First, generate the shared cluster ID:

Copy the output; you will reuse it as CLUSTER_ID in the next step for the Compose file.

Example output:

The Compose file below is the backbone of this Apache Kafka cluster setup. Everything from replication to security is connected through the environment variables in this one file. It sets KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL for the Raft controller channel, defines a dedicated PLAIN_SASL_JAAS_CONFIG entry per listener, and mounts a shared secrets folder into every broker so the CLI tools can authenticate.

From the Kafka cluster directory, create the Docker Compose file:

Add this content to the file; this defines a 3-broker, 3-controller KRaft cluster using the official apache/kafka image:

Replace <DEDICATED_SERVER_IP> with your server's public or private IP in all three KAFKA_ADVERTISED_LISTENERS lines with your server's actual IP address.

This is the moment your Apache Kafka cluster setup actually comes alive, so watch the logs closely for formatting and quorum-election messages.

With the compose file and JAAS credentials in place, bring the cluster up:

Wait until all three logs show Kafka Server started. Since every node uses the same CLUSTER_ID, the official apache/kafka image automatically formats the storage on first boot; you don't need to run kafka-storage.sh format manually.

Note: If you deploy Kafka directly on the host without Docker, you must run the format step yourself before starting each node:

Confirm the secrets folder is mounted and visible inside the container:

With the Apache Kafka cluster setup running, the next step is defining how data is replicated and partitioned across brokers.

Replication and partitioning control how durable and fast your Apache Kafka cluster setup is. A replication factor of 3 means every partition's data is copied to all three brokers, so the cluster keeps working even if two brokers fail. Setting min.insync.replicas=2 adds extra safety; at least two brokers must confirm a write before Kafka tells the producer it succeeded.

At this point, you must create an admin client config for authenticated CLI access. This file lives on your host and is readable inside every broker container through the ./secrets:/etc/kafka/secrets:ro mount already defined in the Compose file:

Now create a topic with 6 partitions and a replication factor of 3:

List and describe topics to confirm partition distribution across brokers:

You should see six partitions, each with a leader and two in-sync replicas spread across kafka1, kafka2, and kafka3. This distribution is exactly what makes an Apache Kafka cluster setup flexible. If a broker goes down, Kafka automatically chooses a new leader for the partitions it hosted.

For AI and log-pipeline use cases, create a couple more topics reflecting real workloads:

More partitions mean more consumers can read from a topic at the same time. That's why ai.embeddings.jobs uses 12 partitions; it lets multiple workers process embedding or vector database jobs in parallel, which helps if you're scaling AI model-serving alongside this cluster.

Logging in isn't the same as having permission. Without ACLs, anyone with valid SASL credentials could read or write to any topic. ACLs fix this by defining exactly what each user is allowed to do. Since StandardAuthorizer and allow.everyone.if.no.acl.found=false are already set in the Compose file, no one can touch a topic until you grant them access.

Grant a producer principal write access to events.orders:

Grant a consumer principal read access, plus group access for its consumer group:

List current ACLs to confirm:

This is one of the biggest differences you'll manage in an Apache Kafka cluster setup compared to simpler brokers. Every producer and consumer should only get the exact permissions it needs, nothing more.

Create client config files matching each principal's credentials:

Since the ./secrets folder is mounted read-only into every broker container, these files are immediately available at /etc/kafka/secrets/ inside kafka1, kafka2, and kafka3.

Produce a few test messages:

Type a few lines and press Enter after each, then press Ctrl+C to exit.

In a second terminal, consume them:

If you see the messages you typed appear on the consumer side, both authentication and ACL authorization are working correctly, and your Apache Kafka cluster setup is functioning.

As an extra check, kill one broker and confirm the topic is still readable and writable:

This proves replication and leader failover actually work.

Once your Apache Kafka cluster setup is live, the next practical question is whether every workload actually belongs on Kafka rather than a simpler queue.

Both are message brokers, but they're built for different jobs; picking the wrong one can slow down your whole pipeline. Kafka keeps an ordered log you can replay, which is great when several consumers all need to read the same events independently, or when you need to reprocess old data to retrain a model. RabbitMQ is better at quickly routing tasks to the right place, where each message is handled once and then discarded.

If you mainly need background jobs or service-to-service requests, use RabbitMQ instead. Our durable queue guide with Docker Compose shows you how to set it up. Kafka and RabbitMQ are not direct replacements; choose the one that fits how your data moves.

At this point, you have a complete Apache Kafka cluster setup with KRaft, replication, partitions, SASL/PLAIN security, ACLs, and verified producer/consumer tests.

For real production traffic, run each broker on a separate dedicated server with NVMe storage, so replication and log persistence never compete with another tenant's I/O, and you can scale brokers independently as event volume grows.

We hope you enjoy this guide.

For deeper details on KRaft configuration, controller quorums, and cluster operations, see the official Apache Kafka KRaft documentation.