Kafka Levers

Learn how to tune the levers provided by Kafka.

We'll cover the following

Kafka provides many levers to adjust the way it operates depending on the application’s needs.

Tuning levers

The levers should be tuned carefully depending on requirements around availability, durability, and performance. For example, the user can control:

  • The replication factor of a topic
  • The minimum size of the ISR set (min.insync.replicas)
  • The number of replicas from the ISR set that needs to acknowledge a record before it’s committed (acks)

Trade-offs

Let’s discuss some of the trade-offs of Kafka one can make using the above values.

  • Setting min.insync.replicas to a majority quorum (e.g. (replicationfactor/2)+1)(replication factor / 2) + 1) and acksacks to allall would allow one to enforce stricter durability guarantees, while also achieving good availability. Let’s assume replicationfactor=5replication factor = 5, so there are 55 replicas per partition and min.insync.replicas=3min.insync.replicas = 3. This would mean up to 22 node failures can be tolerated with zero data loss, and the cluster is still available for writes and reads.

  • Setting min.insync.replicas equal to replicationfactorreplication factor and acksacks to allall would provide even stronger durability guarantees at the expense of lower availability. In our previous example of replicationfactor=5replication factor = 5, this would mean that up to 44 node failures can now be tolerated with zero data loss. However, a single node failure makes the cluster unavailable for writes.

  • Setting acksacks to 11 can provide better performance at the expense of weaker durability and consistency guarantees. For example, records will be considered committed and acknowledged as soon as the leader has stored them locally without waiting for any of the followers to catch up. However, in case of a leader’s failure and election of a new leader, records that the previous leader acknowledged but did not make it to the new leader yet will be lost.

What tuning levers are available in Kafka, and how do they impact system behavior? Write your answer with proper reasoning in the AI assessment widget below. 

What tuning levers are available in Kafka?

Get hands-on with 1400+ tech skills courses.