Kafka Producers

Producers live by their meaning, they produce data into the kafka stream. A producer is the one which publishes or writes data to the topics within different partitions.

Its kind of magical that producers automatically knows which broker and partition to write the data ( which we will discuss in detail while discussing kafka broker discovery). The user does not require to specify the broker and the partition. Which removes a lot of difficulty.

The producer will also do an amazing thing, It will load balance across all partitions (i.e. The load will not be concentrated on a single partition )

Producer Acknowledgements:

Producer can choose to receive acknowledgements of data write into kafka partitions.

Ack -> Confirmation of data reception

There are three ways to receive data acknowledgement

1. (acks=0) Producer will send data into the kafka stream and don’t wait for acknowledgement. Here the data loss will be high because of the missing acknowledgements.
2. (acks=1) Producer will send data into the kafka stream and wait for leader reception. Minimal data loss since we are ensuring the data is available in leader.
3. (acks = all) Producer will send data into the kafka stream and wait for all the copies to confirm the reception. There will be no data loss since all the replicas are ensured that they have the data.

acks = 0

acks = 1

acks = ALL

Since we have addressed the concern of receiving acknowledgement for data reception. We move on to how data is added to partitions in kafka. As we saw before producer distributes data across the partitions of kafka cluster. But how?

Enter Producer Message Keys

Apache Kafka enables the concept of the key to send the messages in a specific order. The key enables the producer with two choices, i.e., either to send data to each partition (automatically) or send data to a specific partition only. Sending data to some specific partitions is possible with the message keys. If the producers apply key over the data, that data will always be sent to the same partition always. But, if the producer does not apply the key while writing the data, it will be sent in a round-robin manner. This process is called load balancing. In Kafka, load balancing is done when the producer writes data to the Kafka topic without specifying any key, Kafka distributes little-little bit data to each partition.

We can ensure this guarantee of all messages of same key goes to same partitions thanks to hashing.

#100DaysofCode #day24