Replication Factor in Kafka
Last time we have seen about the brokers and clusters. In short Kafka broker is an independent process ( it can be a vm or physical machine) that runs kafka and Kafka cluster is a group of kafka brokers.
We can have any number of brokers in kafka cluster. It can be 3 or it can be 100 it doesn’t matter. But the point is Kafka is a distributed system. When there is a distributed system and when it process millions of messages as kafka really does, we need to have replication.
So what is Replication?
Copying the same data over multiple nodes in distributed systems is critical to keep the system up and keep on serving data queries even during failures. Following are other reason behind data replication in a distributed system :
- Higher Availability: To ensure the availability of the distributed system( System keeps on working even if one or fewer nodes fail)
- Reduced Latency: Replication assists in reducing the latency of data queries (By keeping data geographically closer to a user. For example, CDN(Content Delivery Networks) keeps a copy of replicated data closer to the user. Ever thought how Netflix streams videos with such short latencies!)
- Read Scalability: Read queries can be served from replicated copies of the same data (this increase overall throughput of queries)
- Network Interruption: System works even under network faults
Replication in Kafka:
In Kafka, replication happens at the partition granularity i.e. copies of the partition are maintained at multiple broker instances. Replication factor defines the number of copies of the partition that needs to be kept.
A Topic when created should have a replication factor. It is usually decided between 2 and 3. In which 2 being a bit risky and 3 is better.
A replication factor of 2 means that there will be two copies for every partition.
In the above scenario the replication factor decided is 2 and there will be exactly 2 copies of particular partition in the cluster. If Broker 1 goes down, Topic 1’s partition 0 will be accessed from Broker 2. Similarly Topic 1’s partition 3 will also be present in Broker 3.
Partition Leader & In Sync Replicas:
At any time only one broker can be leader for a given partition. In kafka compared to some databases, the leader can only send as well as receive data for that partition. All the other replicas are called as In Sync Replicas or simply followers of the partition.
In-sync replicas are the subset of all the replicas for a partition having same messages as the leader.
Now let’s see what happens when a broker goes down. If for some reason lets say Broker 2 goes down. The access to partition 1 is now lost since broker 2 was the leader for partition 1. What happens now is Kafka automatically selects one of the in-sync replicas (in our case there is only 1 replica) and makes them the leader. Now when broker 2 comes back online, it can try to become the leader again.
#100DaysOfCode #Day23
