Scaling
•
0 likes•564 views
Scaling: a naïve approach A look at how to scale an existing monolythic system, and how companies such as Disqus and Eventbrite have done it.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Scaling
Similar to Scaling (20)
More from Òscar Vilaplana
Recently uploaded
Recently uploaded (20)
Scaling
- 3. PEOPLE Care Focus Automate & Test. Shared brain Finish & DRY.
- 4. TECH Design to clone Separate pieces API Offload everything Measure
- 10. TECH • Design to clone • Separate pieces • API • Offload everything • Measure
- 11. TYPES OF TASKS • Realtime • ASAP • When you have time } Async!
- 12. INSTAGRAM’S FEED • Redis queue per follower. • New media: push to queues • Small chained tasks
- 13. INSTAGRAM’S FEED harro wouter orestis siebejan oscar Schedule next batch
- 14. SMALL TASKS • 10k followers per task • < 2s • Finer-grained load balancing • Lower penalty of failure/reload
- 15. CELERY: REDIS • Good: Fast • Bad: • Polling for task distribution • Messy non-synchronous replication • Memory limits task capacity
- 16. CELERY: BEANSTALK • Good: • Fast • Push to consumers • Writes to disk • Bad: • No replication • Only useful for Celery
- 17. CELERY: RABBITMQ • Fast • Writes to disk • Low-maintenance synchronous replication • Excellent Celery compatibility • Supports other use cases
- 21. RESERVATIONS • UI • Room locking • Room availability • Registration manager • Email, PDF invoice • Payment • Login • …
- 22. WE DON’T DO THIS def do_everything(request): hotel_id = request.GET.hotel_id room_number = request.GET.room_number with room_mutex(hotel_id, room_number): room = (session.query(Room) .filter(Room.hotel_id == hotel_id) .filter(Room.room_number == room_number).one()) if not room.available: return Response("Room not available”, template=room_template) reservation = Reservation(client=request.client, room=room) session.add(reservation) room.available = False price = # price_calculation payment = Payment(reservation=reservation, price=price) session.add(payment) session.commit() url = payment.get_psp_url() return Redirect(url)
- 23. BUT WE DO THIS • Frontend UI • Locking rooms • Calculating room availability • Temporarily locking rooms • Payment processing • Mail • PDF invoice generation
- 24. BUT WE CAN SCALE!
- 25. SCALE DB: HARD • Slaves • Master- Master? • Sharding?
- 26. SCALING
- 27. MINOR SCALE
- 28. MAJOR SCALE
- 30. FRONTEND Everything Frontend External payment providers User Everything Frontend Master Read slaves
- 31. SPLIT • Responsibility • Stateful/stateless • Type of system
- 32. TYPES OF SYSTEMS • Unique (mutex, datastore) • Multiple
- 33. TYPES OF TASKS • Realtime • ASAP • When you have time } Async!
- 36. CLONABILITY
- 37. CLONABILITY
- 40. WHAT’S IN AN EASY STEP As little change as possible. Reuse. Unintrusive. Measure. Go on the right direction.
- 46. INITIAL SYSTEM (MODIFIED) Everything Frontend Sales Sync
- 47. INITIAL SYSTEM (MODIFIED) Sales Backend
- 50. WHEN? • Difficult. • Measure everything. • Find patterns. • Define thresholds. • Design: address as risk. • Don’t overenigneer — Don’t ignore.
- 51. EVENTBRITE • 2012: $600M ticket sales • Accumulated: $1B
- 52. TECHNOLOGY • Monitoring: nagios, ganglia, pingdom • Email: offloaded to StrongMail • Load-balanced read slave pool • Feature flags • Automated server configuration and release with Puppet and Jenkins
- 53. TECHNOLOGY • Feature flags • Develop on Vagrant • Celery + RabbitMQ • Virtual customer queue • Big data for reporting, fraud, spam, event recommendations
- 54. TECHNOLOGY • Hadoop • Cassandra • HBase • Hive • Separated into independent services
- 55. TIPS • Instrument and monitor everything • Lean
- 56. HOW BIG? • 2Gb/day database transactions • 3.5Tb/day social data analyzed • 15Gb/day logs
- 57. ORDER PROCESSOR • Pub/sub queue with Cassandra and Zookeeper
- 58. PUBLISHING Publisher Get queue lock+last batch id Create new batch “process orders 10, 11, 12” Store batch id, release lock
- 59. SUBSCRIBING Subscriber Get my latest processed batch id Store result Update my latest processed batch id
- 60. SCALING STORAGE • Move to NoSQL • Aggressively move queries to slaves • Different indexes per slave • Better hardware • Most optimal tables for large and highly-utilized datasets
- 61. EMAIL ADDRESSES • Users have many email addresses. • Lookup by email, join to users table
- 62. FIRST ATTEMPT CREATE TABLE `user_emails` ( `id` int NOT NULL AUTO_INCREMENT, `email_address` varchar(255) NOT NULL, ... --other columns about the user `user_id` int, --foreign key to users KEY (`email_address`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
- 63. FIRST ATTEMPT
- 64. LOOKUP
- 65. CAN IT BE IMPROVED?
- 66. INDEX VS PK • InnoDB: B+trees, O(log n) • Known user id: index on email not needed. • Small win on lookup: O(1) • Big win on not storing the index.
- 67. INNODB INDEXES
- 68. HASH TABLE
- 69. DISQUS • >165K messages per second • <10ms latency • 1.3B unique visitors • 10B page views • 500M users in discussions • 3M communitios • 25M comments
- 70. ORIGINAL REALTIME BACKEND • Python + gevent • NginxPushStream • Network IO: great • CPU: choking at peaks • <15ms latency
- 71. CURRENT REALTIME BACKEND • Go • Handles all users • Normal load: 3200 connections/machine/sec • <10ms latency • Only 10%-20% CPU
- 72. Workers CURRENT REALTIME BACKEND Subscribed to results Push result to user NginxPushStream
- 73. TESTING • Test with real traffic • Measure everything
- 74. LESSONS • Do work once, distribute results. • Most likely to fail: your code. Don’t reinvent. Keep team small. • End-to-end ACKs are expensive. Avoid. • Understand use cases when load testing. • Tune architecture to scale.
- 75. LEARN MORE • Instagram • Braintree • highscalability.com • VelocityConf (youtube, nov 2014 @ bcn?)