Advertisement

Scaling

Python Geek at Paylogic
Jun. 14, 2014
Advertisement

More Related Content

Advertisement
Advertisement

Scaling

  1. SCALING Òscar Vilaplana @grimborg http://oscarvilaplana.cat
  2. WHAT’S THIS ABOUT? People Technology Tools
  3. PEOPLE Care Focus Automate & Test. Shared brain Finish & DRY.
  4. TECH Design to clone Separate pieces API Offload everything Measure
  5. VIRTUAL QUEUE Queue Instance Queue Instance Queue Instance Queue Instance
  6. VIRTUAL QUEUE Queue Instance Queue Instance Queue Instance
  7. VIRTUAL QUEUE Queue Instance Queue Instance Queue Instance
  8. VIRTUAL QUEUE Queue Instance Queue Instance Queue Instance Queue Instance
  9. TECH • Design to clone • Separate pieces • API • Offload everything • Measure
  10. TYPES OF TASKS • Realtime • ASAP • When you have time } Async!
  11. INSTAGRAM’S FEED • Redis queue per follower. • New media: push to queues • Small chained tasks
  12. INSTAGRAM’S FEED harro wouter orestis siebejan oscar Schedule
 next
 batch
  13. SMALL TASKS • 10k followers per task • < 2s • Finer-grained load balancing • Lower penalty of failure/reload
  14. CELERY: REDIS • Good: Fast • Bad: • Polling for task distribution • Messy non-synchronous replication • Memory limits task capacity
  15. CELERY: BEANSTALK • Good: • Fast • Push to consumers • Writes to disk • Bad: • No replication • Only useful for Celery
  16. CELERY: RABBITMQ • Fast • Writes to disk • Low-maintenance synchronous replication • Excellent Celery compatibility • Supports other use cases
  17. RESERVATIONS • UI • Room locking • Room availability • Registration manager • Email, PDF invoice • Payment • Login • …
  18. WE DON’T DO THIS def do_everything(request): hotel_id = request.GET.hotel_id room_number = request.GET.room_number with room_mutex(hotel_id, room_number): room = (session.query(Room) .filter(Room.hotel_id == hotel_id) .filter(Room.room_number == room_number).one()) if not room.available: return Response("Room not available”, template=room_template) reservation = Reservation(client=request.client, room=room) session.add(reservation) room.available = False price = # price_calculation payment = Payment(reservation=reservation, price=price) session.add(payment) session.commit() url = payment.get_psp_url() return Redirect(url)
  19. BUT WE DO THIS • Frontend UI • Locking rooms • Calculating room availability • Temporarily locking rooms • Payment processing • Mail • PDF invoice generation
  20. BUT WE CAN SCALE!
  21. SCALE DB: HARD • Slaves • Master- Master? • Sharding?
  22. SCALING
  23. MINOR SCALE
  24. MAJOR SCALE
  25. FRONTEND Everything Frontend External
 payment
 providers User Everything Frontend Master Read slaves
  26. SPLIT • Responsibility • Stateful/stateless • Type of system
  27. TYPES OF SYSTEMS • Unique (mutex, datastore) • Multiple
  28. TYPES OF TASKS • Realtime • ASAP • When you have time } Async!
  29. SPLIT THIS Everything Frontend External
 payment
 providers User Everything Frontend Master Read slaves
  30. AUTONOMOUS SYSTEMS Payment External
 payment
 providers Locking Invoice
 PDF Mailer UI Reservations Manager User Session
 Storage Datawarehouse Reporting Configuration Payout
  31. CLONABILITY
  32. CLONABILITY
  33. CLONABILITY Frontend
  34. CLONABILITY Everything Frontend External
 payment
 providers User Everything Frontend Master Read slaves
  35. WHAT’S IN AN EASY STEP As little change as possible. Reuse. Unintrusive. Measure. Go on the right direction.
  36. SMALL STEPS PROBLEMS? ! Oversells Configuration Reporting Payout Everything Frontend Everything Frontend Everything Frontend Everything Frontend Everything Frontend Everything Frontend
  37. SMALL STEPS PROBLEMS? ! Oversells Configuration Reporting Payout SessionsRoom Availability Lock Read Everything Frontend Everything Frontend Everything Frontend Everything Frontend Everything Frontend Everything Frontend
  38. ISOLATED SYSTEM Best technology Decoupled API Testable
  39. SMALL STEPS PROBLEMS? ! Oversells Configuration Reporting Payout Sessions Everything Frontend Config Backend Settings Everything Frontend Everything Frontend Everything Frontend Everything Frontend Everything Frontend Everything Frontend
  40. INITIAL SYSTEM Everything Frontend
  41. INITIAL SYSTEM (MODIFIED) Everything Frontend Sales Sync
  42. INITIAL SYSTEM (MODIFIED) Sales Backend
  43. SMALL STEPS PROBLEMS? ! Oversells Configuration Reporting Payout Sessions Everything Frontend Sales Backend Sales Main DB Everything Frontend Everything Frontend Everything Frontend Everything Frontend Everything Frontend Everything Frontend
  44. SMALL STEPS PROBLEMS? ! Oversells Configuration Reporting Payout SessionsSession Storage Everything Frontend Everything Frontend Everything Frontend Everything Frontend Everything Frontend Everything Frontend
  45. WHEN? • Difficult. • Measure everything. • Find patterns. • Define thresholds. • Design: address as risk. • Don’t overenigneer — Don’t ignore.
  46. EVENTBRITE • 2012: $600M ticket sales • Accumulated: $1B
  47. TECHNOLOGY • Monitoring: nagios, ganglia, pingdom • Email: offloaded to StrongMail • Load-balanced read slave pool • Feature flags • Automated server configuration and release with Puppet and Jenkins
  48. TECHNOLOGY • Feature flags • Develop on Vagrant • Celery + RabbitMQ • Virtual customer queue • Big data for reporting, fraud, spam, event recommendations
  49. TECHNOLOGY • Hadoop • Cassandra • HBase • Hive • Separated into independent services
  50. TIPS • Instrument and monitor everything • Lean
  51. HOW BIG? • 2Gb/day database transactions • 3.5Tb/day social data analyzed • 15Gb/day logs
  52. ORDER PROCESSOR • Pub/sub queue with Cassandra and Zookeeper
  53. PUBLISHING Publisher Get queue lock+last batch id Create new batch “process orders 10, 11, 12” Store batch id, release lock
  54. SUBSCRIBING Subscriber Get my latest processed batch id Store result Update my latest processed batch id
  55. SCALING STORAGE • Move to NoSQL • Aggressively move queries to slaves • Different indexes per slave • Better hardware • Most optimal tables for large and highly-utilized datasets
  56. EMAIL ADDRESSES • Users have many email addresses. • Lookup by email, join to users table
  57. FIRST ATTEMPT CREATE TABLE `user_emails` ( `id` int NOT NULL AUTO_INCREMENT, `email_address` varchar(255) NOT NULL, ... --other columns about the user `user_id` int, --foreign key to users KEY (`email_address`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
  58. FIRST ATTEMPT
  59. LOOKUP
  60. CAN IT BE IMPROVED?
  61. INDEX VS PK • InnoDB: B+trees, O(log n) • Known user id: index on email not needed. • Small win on lookup: O(1) • Big win on not storing the index.
  62. INNODB INDEXES
  63. HASH TABLE
  64. DISQUS • >165K messages per second • <10ms latency • 1.3B unique visitors • 10B page views • 500M users in discussions • 3M communitios • 25M comments
  65. ORIGINAL REALTIME BACKEND • Python + gevent • NginxPushStream • Network IO: great • CPU: choking at peaks • <15ms latency
  66. CURRENT REALTIME BACKEND • Go • Handles all users • Normal load:
 3200 connections/machine/sec • <10ms latency • Only 10%-20% CPU
  67. Workers CURRENT REALTIME BACKEND Subscribed to results Push result to user NginxPushStream
  68. TESTING • Test with real traffic • Measure everything
  69. LESSONS • Do work once, distribute results. • Most likely to fail: your code. Don’t reinvent. Keep team small. • End-to-end ACKs are expensive. Avoid. • Understand use cases when load testing. • Tune architecture to scale.
  70. LEARN MORE • Instagram • Braintree • highscalability.com • VelocityConf (youtube, nov 2014 @ bcn?)
  71. QUESTIONS? ANSWERS? THANKS! Òscar Vilaplana @grimborg http://oscarvilaplana.cat
Advertisement