Successfully reported this slideshow.
Your SlideShare is downloading. ×

Learning Apache Kafka Theory 101 w/ My Little Ponies

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 21 Ad
Advertisement

More Related Content

Similar to Learning Apache Kafka Theory 101 w/ My Little Ponies (20)

Recently uploaded (20)

Advertisement

Learning Apache Kafka Theory 101 w/ My Little Ponies

  1. 1. Apache Kafka: Theory 101 … with My little Ponies! Alejandra Quetzalli alejandra@shecodesnow.org
  2. 2. Agenda • What is Apache Kafka? • What can I use event streaming for? • Where is Apache Kafka used? • Kafka Theory 101 Overview And maybe a pony or two.
  3. 3. What is Apache Kafka?
  4. 4. What is Apache Kafka? • It is an 🙌 open-source 🙌 event streaming platform. • Simply put, it is a way of moving data between systems. • (i.e. between applications, and servers)
  5. 5. 3 key capabilities for event streaming: • To publish (write) and subscribe to (read) streams of events, including continuous import/export of data from other systems. • To store streams of events durably and reliably for as long as a dev needs. • To process streams of events as they occur or retrospectively.
  6. 6. What does one use event streaming for?
  7. 7. What can I use event streaming for? • To process payments and financial transactions in real-time • stock exchanges • banks • insurances • To track and monitor cars, trucks, fleets, and shipments in real-time • logistics • automotive industry • To continuously capture and analyze sensor data from IoT devices/equipment • factories • wind parks
  8. 8. Event streaming – cont’d… • To collect and immediately react to customer interactions and orders • retail • hotel and travel industry • mobile applications • To connect, store, and make available data produced by different divisions of a company. • To serve as the foundation for data platforms, event-driven architectures, and microservices.
  9. 9. Where is Apache Kafka used in the world?
  10. 10. Where is Apache Kafka used? • Netflix is using Kafka to apply recommendations in real-time while you're watching TV shows. • Uber uses Kafka to gather user, taxi and trip data in real-time to compute and forecast demand and pricing in real-time. • LinkedIn uses Kafka to prevent spam in their platform, collect user interactions and make better connection recommendations.
  11. 11. Concepts & Theory 101
  12. 12. T is for Topics and Twilight Sparkle • A particular stream of data. • A topic is identified by its name. • Topics are split into partitions. • Each partition is ordered. • Each message within a partition gets an incremental id, called offset. • Central protagonist of the show • Most intellectual member of the Mane Six • Her cutie mark represents her talent for magic and her love for books and knowledge. 0 1 2 3 4 0 1 2 3 0 1 2 3 4 5 6 Partition 0 Partition 1 Partition 2
  13. 13. Topic Replication Topic Replication is the process to offer fail-over capability for a topic. • If one broker (holds topics and partitions) is down, the other broker can serve the data. • This replication factor defines the number of copies of a topic in a Kafka cluster (made up of multiple Kafka brokers). • Kafka stores messages in topics that are partitioned and replicated across multiple brokers in a cluster. Broker 2 Broker 1 Partition 0 Topic A Partition 1 Topic B Partition 0 Topic A
  14. 14. B is for Brokers and Babs Seed • Holds topics and partitions. • Each broker is identified with its ID (integer). • Each broker contains certain topic partitions. • After connecting to any broker, you're connected to an entire cluster. • Apple Bloom's cousin from Manehattan. • Former member of the Cutie Mark Crusaders
  15. 15. P is for Partitions and Pinkie Pie • An ordered, immutable record sequence. • Once data is written to a partition, it can't be changed (immutable). • Data is assigned randomly to a partition, unless a key is provided. • At any one time, only ONE broker can be a leader for a given partition. • That leader can receive and serve data for a partition. • The other brokers will just be passive replicas and synchronize the data. • Each partition is going to have one leader, and multiple ISR (in-sync replica). • Baker at Sugarcube Corner. • Toothless pet alligator, Gummy. • Represents the element of laughter.
  16. 16. P is also for Producers and Pound Cake • How do we get data in Kafka? • Producers write data to topics (which are made up of partitions). • Producers automatically know to which broker and partition to write to. • Producers Message Keys: producers can choose to send a key with a message. (i.e. string, number, .etc) • If a key is sent, all messages for that key will always go to the same partition. • A key is sent if you need message ordering for a specific field (i.e. truck_id). • Parents are surprised to find out that Pound Cake is a male Pegasus, and his twin foal sister (Pumpkin Cake) is a female unicorn… even though their parents are both Earth ponies.
  17. 17. C is for Consumers and Princess Celestia • How do we read data in Kafka? • They read data from a topic (identified by name). • Consumers know which broker to read from. • In case of broker failures, consumers know how to recover. • Data is read in order within each partition. • Consumer Groups: consumers read data in consumer groups. • Each consumer within a group will read directly from exclusive partitions. • If you have more consumers than partitions, some will be inactive. • Most magical pony. • Responsible for raising the sun to create light in Equestria. • Over 1,000 years old!
  18. 18. Z is for Zookeeper and Zecora • Keeps track of status of the Kafka cluster nodes. • Also keeps track of Kafka topics and partitions. • Currently, Apache Kafka® uses Apache ZooKeeper™ to store its metadata (i.e. location of partitions, the configuration of topics). ⚠️ Apache Kafka is removing the Apache ZooKeeper Dependency. ⚠️ • In 2019, they outlined a plan to break this dependency and bring metadata management back into Kafka itself. • Female zebra shaman and herbalist. • Always speaks in rhyme.
  19. 19. What next?
  20. 20. What now, developer? • To get hands-on experience, follow the Quickstart. • To learn more about Apache Kafka, check out the developer docs. • Books and academic papers!
  21. 21. Gracias Alejandra Quetzalli alejandra@shecodesnow.org twitter.com/QuetzalliAle linkedin.com/in/alejandra-quetzalli

×