Kafka Brokers and Cluster
Okay in the last post we have talked about kafka topics. What holds the topics and the partitions.
The answer is a kafka broker
Okay, then what is a kafka broker, simply put it is a machine that runs kafka i.e a single kafka server. Now the official definition,
Kafka broker is a logical process, not a phsyical entity. It can be run in a physical machine or a virtual machine, or in a container. Though some people, might use the term broker and node interchangeably. But, as far as the definition goes, a node is a typically a physical entity, a machine or a VM, but a broker is a process. We can run multiple brokers in one physical node also.
Kafka broker is more precisely described as a Message Broker which is responsible for mediating the conversation between different computer systems, guaranteeing delivery of the message to the correct parties.
On the other Hand, A Kafka cluster consists of one or more servers (Kafka brokers) running Kafka. In simple terms the Kafka Custer is the combination of multiple brokers and distribute the data on multiple instances. Kafka clusters consists of the following components
Zookeeper ( Broker management )
Kafka Brokers( kafka nodes )
Producers ( which produce data into the kafka )
Consumers ( which consume data from kafka)
In the kafka cluster, each broker will only have some kind of data but not all of it. Because kafka is distributed, the data is distributed among all brokers. The Kafka brokers are stateless in nature. Hence, the Kafka broker uses the zookeeper for maintaining their state in the cluster. The single Kafka broker will handle the hundreds or thousands of reads and writes per second
A good number of brokers to start with is 3, but some big clusters have over 100 brokers. When we create a topic kafka will create and distribute the topic and its partition across all the brokers. This will be done automatically by the kafka cluster and distribution will be uniform.
#100DaysofCode #Day21
