Kafka Summit SF 2017 - Shopify Flash Sales with Apache Kafka
•
0 likes•2,385 views
This document discusses how Shopify uses Apache Kafka in their systems architecture. It describes how Kafka provides reliable asynchronous messaging that allows Shopify to collect logs and events and feed them to their data lake. It outlines how Kafka provides operational decoupling and allows them to deploy application or Kafka changes independently. It then discusses two specific use cases: using Kafka as part of a logs/events pipeline and using it to enable active-active Elasticsearch replication across multiple data centers.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Kafka Summit SF 2017 - Shopify Flash Sales with Apache Kafka
Similar to Kafka Summit SF 2017 - Shopify Flash Sales with Apache Kafka (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
Kafka Summit SF 2017 - Shopify Flash Sales with Apache Kafka
- 1. 1 Shopify Flash Sales and Apache Kafka
- 2. Agenda ● Shopify ● Flash Sales ● Kafka in Shopify Stack ● Architecture Evolution ● Kafka in the big picture ● Kafka and SysvMQ ● Kafka Architecture Today ● Use cases
- 3. $whoami ● Senior Production Engineer at Shopify ● Production Engineering / Data Delivery ● Visiting Lecturer @ Concordia University
- 7. 7
- 8. Shopify ● Complete e-commerce solution ● Multi-tenant platform ● Hosted online Stores ● Fully customizable ● Multiple Sale Channels: Web, Mobile, Social, Amazon, eBay ● Online and in-store ● Small to Enterprise
- 9. 500K Merchants $40B TOTAL SALES ON SHOPIFY 175 Countries 2000 Employees
- 13. ● Limited Sale ● By Time or Quantity ● Massive Traffic in a Short Time ● Celebrities ● Special Events ● Super Bowl Flash Sales
- 14. ● Mostly not scheduled! ● Recently more frequent ● Faster, Shorter ● Millions of dollars each! ● Impact all our infrastructure Flash Sales
- 16. Flash Sale
- 18. Shopify Stack, Simplified! Nginx LB Rails MySQL Memcached Redis Unicorn Resque Kafka Data Land Elasticsearch
- 19. Scaling Up 1 Shop 500K Shops 2004 2017
- 20. Scaling Up 1 Shop 500K Shops 2004 20172005 2007 Multi-tenant Flash Sales 2013 2015 2016 Database Sharding Multi-DC Multi-DC Active-Active
- 21. Why Kafka? ● Reliable Asynchronous Messaging ● Collect logs and events ● Feed to Data Land ● Container Friendly ● Use our technology stack: Ruby, Go ● Sarama: Kafka Go Client
- 22. Application Constraints ● Fully decouple Rails Application & Kafka ● Buffer messages outside Rails memory space ● Send retry outside Rails application
- 23. Operational Constraints ● Operational decoupling ● Zero impact during Kafka cluster maintenance ● Kafka cluster maintenance with zero data loss ● Deploy Rails App or Kafka changes with zero dependencies
- 24. Container Kafka In Shopify Stack Rails Kafka Data Land Kafka Producer
- 25. Host Container Kafka In Shopify Stack Rails Kafka Data Land SysV Message Queue Producer
- 26. System V Message Queue ● System V IPC Message Queues ● Kernel Persistence ● Queue Size Depends on OS and message size ● JSON BLOB
- 27. SysV Message Queue Limits localhost$ ipcs -l ------ Messages Limits -------- max queues system wide = 32000 max size of message (bytes) = 8192 default max size of queue (bytes) = 16384 ------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 131072 max total shared memory (kbytes) = 18014398442373116 min seg size (bytes) = 1 ------ Semaphore Limits -------- max number of arrays = 32000 max semaphores per array = 32000 max semaphores system wide = 1024000000 max ops per semop call = 500 semaphore max value = 32767
- 28. SysV Message Queue Limits kernel.msgmni=32 # maximum number of message queues kernel.msgmax=1000000 # maximum message size in bytes kernel.msgmnb=2000000 # maximum total size of all messages in a queue
- 29. SysV Message Queue Limits default['kafka']['max_message_size'] = 1024 * 1024 default['sysctl']['kernel.msgmax'] = node[:kafka][:max_message_size]
- 30. ● Topic: Destination Kafka Topic ● Key: Message key ● Value: Message payload SysV MQ Message Format Topic Bytesize Topic Key Bytesize Key Value Bytesize Value Enqueue Timestamp Retry Attempt
- 32. SysV Message Queue def produce(topic, key, payload) queue.send(Sysv.encode(topic, key, payload), SysVMQ::IPC_NOWAIT) rescue Errno::EAGAIN, Errno::EINVAL => exception raise EventProductionFailure.new(exception.message) end
- 34. SysV MQ Advantages ● OS level buffer owned by Kernel ● ̴2 hours fully operational for Kafka maintenance ● No data loss during this period ● Tested in production with 100K+ RPM ● Multiple containers share the same Sysvmq on same host ● Container friendly
- 35. Use Case 1: Logs/Events Pipeline ASH Nginx LB Memcached Redis Kafka Rails Apps Resque Jobs Aggregate Cluster Agg. Kafka MirrorMaker
- 36. Data Land Agg. Kafka ASH MirrorMaker ASH Kafka CHI Kafka Cloud Kafka CL. MirrorMaker CHI MirrorMaker HDFS Presto
- 37. Use Case 2: Elasticsearch Multi-DC Replication ● Shopify Core Running from 2 DCs ● Active-Passive ES Configuration ● Take ES Snapshots periodically ● If one DC down ○ switch to other DC ○ Restore ES Backups ● Long process!
- 38. Use Case 2: Elasticsearch Multi-DC Replication ● Our target is Active-Active Configuration ● Index data in local ES cluster ● Replicate ES documents Using Kafka ● ES cluster in both DCs have all indexed documents
- 41. 4 1 Q/A