Kafka Levers
Learn how to tune the levers provided by Kafka.
We'll cover the following
Kafka provides many levers to adjust the way it operates depending on the application’s needs.
Tuning levers
The levers should be tuned carefully depending on requirements around availability, durability, and performance. For example, the user can control:
- The replication factor of a topic
- The minimum size of the ISR set (
min.insync.replicas
) - The number of replicas from the ISR set that needs to acknowledge a record before it’s committed (acks)
Trade-offs
Let’s discuss some of the trade-offs of Kafka one can make using the above values.
-
Setting
min.insync.replicas
to a majority quorum (e.g. and to would allow one to enforce stricter durability guarantees, while also achieving good availability. Let’s assume , so there are replicas per partition and . This would mean up to node failures can be tolerated with zero data loss, and the cluster is still available for writes and reads. -
Setting
min.insync.replicas
equal to and to would provide even stronger durability guarantees at the expense of lower availability. In our previous example of , this would mean that up to node failures can now be tolerated with zero data loss. However, a single node failure makes the cluster unavailable for writes. -
Setting to can provide better performance at the expense of weaker durability and consistency guarantees. For example, records will be considered committed and acknowledged as soon as the leader has stored them locally without waiting for any of the followers to catch up. However, in case of a leader’s failure and election of a new leader, records that the previous leader acknowledged but did not make it to the new leader yet will be lost.
What tuning levers are available in Kafka, and how do they impact system behavior? Write your answer with proper reasoning in the AI assessment widget below.
Get hands-on with 1400+ tech skills courses.