Building Scalable Data Pipelines with Kafka

This course brings any Big Data newbie up to speed on one of the most widely used open-source platform to build data pipelines - Apache Kafka.

Beginner

62 Lessons

Certificate of Completion

This course brings any Big Data newbie up to speed on one of the most widely used open-source platform to build data pipelines - Apache Kafka.

AI-POWERED

Explanations

AI-POWERED

Explanations

This course includes

8 Playgrounds

This course includes

8 Playgrounds

Course Overview

If you’re interested in Big Data, then Apache Kafka is a must-know tool. What started as an internal LinkedIn project to streamline data transmission and propagation among services has quickly grown to become a mainstay platform for building highly scalable data pipelines. Meet Apache Kafka - the ubiquitous tool to build pipelines for diverse use cases ranging from chronologically tracking user-activity on a website to implementing publish-subscribe feeds. This course introduces you to Kafka theory and ...Show More

What You'll Learn

Learn the theory behind Kafka

Interact with a Kafka cluster running in the browser-terminal

What You'll Learn

Learn the theory behind Kafka

Course Content

Basics

Introduction

Characteristics of Distributed Systems

Messaging Patterns

Commit Log

Components of Kafka Partitions Partition Rebalancer

Kafka Producer

Producer Sending Messages Producer Configurations Producer Serialization

Kafka Consumer

Consumer and Consumer Groups Kafka Consumer Consumer Configurations Commits and Offsets Offset Commit Configuration Handling Rebalances Stopping a Consumer Consumer Deserializer Running a Single Consumer

Kafka Internals

Replication Kafka Controller Request Processing Partition Allocation Data Storage Reliability in Kafka Reliable Producers and Consumers

Conclusion

Kafka vs Other Messaging Systems

Appendix

3 Lessons

Reference: Replication

14 Lessons

Reference: Partitioning

4 Lessons

Reference: Transactions

9 Lessons

10.

Reference: Issues in Distributed Systems

4 Lessons

Course Author

Trusted by 1.4 million developers working at companies

Anthony Walker

@_webarchitect_

Emma Bostian 🐞

@EmmaBostian

Evan Dunbar

ML Engineer

Carlos Matias La Borde

Software Developer

Souvik Kundu

Front-end Developer

Vinay Krishnaiah

Software Developer

Eric Downs

Musician/Entrepeneur

Kenan Eyvazov

DevOps Engineer

Anthony Walker

@_webarchitect_

Emma Bostian 🐞

@EmmaBostian

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI-Powered Mock Interviews

Put your skills to the test in a simulated interview setting. Receive personalized feedback based on your performance. Available in Premium & Premium Plus plans.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!