SCALING,WHAT YOUNEED TOKNOWQUENTIN ADAM @WAXZCECLEVER CLOUD CEO2013
WHEN YOUNEED TOSCALETHERE ARE 2 WAYS
GROWING AND GROWINGUNTIL YOU EXPLODE ORBECOME WEIRD
OR SPLIT THE WORK ANDMAKE YOUR SOFTWAREWORK AS A TEAM
Build an army of fat appYOU CAN DO BOTH
SO WE NEED TO BE ABLETO DISPATCH THE WORKSCALE OUT•  Many workersdoing the samething•  No SPOF•  Growing is moreeasy•  Int...
SO, HOW TOSCALEOUT ?JUST SOME FACTS
SPLIT PROCESS ANDSTORAGEStorage• Databases• Files• Sessions• Events• …Code• Can be replicated• Stateless• Process
Picking one instance or another doesn’t matterSTATELESSNESS IS THE KEY
CONSIDER MORETHINGS AS DATA•  User account•  Users data•  Files•  Sessions•  Events
CHOOSEDATASTOREWISELYYOU CAN SHOULD USE MANY DATASTORES
DATASTORE CHOICESARE DRIVEN BY USAGEMakedecisionsbased onneedsDo I needatomicity ofrequests ?Do I needconcurrentaccess ?Do...
•  Not a big volume•  DB have to managedata TTL•  Data model : K/V•  Multiple writes at thesame time•  High availabilityI ...
•  Not a big volumeIt’s OK, PG can handlesmall quantity of data•  DB have to managedata TTLNo, I have to do itmanually•  D...
•  Not a big volumeIt’s OK, redis canhandle small quantityof data•  DB have to managedata TTLYes Redis can do it•  Data mo...
•  Not a big volumeIt’s OK, redis canhandle small quantityof data•  DB have to managedata TTLYes redis can do it•  Data mo...
USE ONLINEDATABASE / BEREADY TO TESTIN JUST A FEWMINUTESNO NEED TO TRASH YOUR COMPUTER
DON’T BE THAT GUY
DO NOT USE ATECHNOLOGY BECAUSEYOU <3 IT OR BECAUSEIT’S HYPE : USE ITBECAUSE IT FITS YOURNEEDSBALANCE YOUR LEARNING CURVE W...
DO NOT CREATE MONSTERS
COMMON MISTAKES
DO NOT USEMEMORY ASDATABASELIKE : SHARED / GLOBAL VARIABLE,CACHE “IN THE CODE”, INTENSIVESESSION USAGE…
DO NOT USE A VARIABLEFOR MORE THAN ONEREQUEST
2 + 2 = 4FOR SAME INPUT, SAME OUTPUT
And data will be lostCODE WILL FAIL
DO NOT USE FILESYSTEM AS DATASTOREFile system are POSIX compliant•  POSIX is ACID•  POSIX is powerful but is bottleneck•  ...
CAREFUL USE OFDARK MAGIC
SPLIT THE CODE :MODULES•  Smallest codebase•  Deploy asservice foreach other•  Focus on besttechnology fora problem
USE EVENT BROKER TOMODULARIZE YOUR APP•  AMQP•  Celery•  0MQ•  Redis•  JMS•  Some case : hadoop, akka…•  …CRON is not an e...
MAKE HARDCOMPUTATION ASYNC
ALWAYS USE AREVERSE PROXYY U NOT USE ONE ?
USE PROCESSDEPLOYMENT
KEEP CALM UNDERFIRE
TRACK BUG & GETMETRICS
Quentin ADAMTwitter : @waxzceTHX FOR LISTENING& QUESTIONS TIMEcleverc l o u d
Dotscale2013 : How to scale ?
Upcoming SlideShare
Loading in...5
×

Dotscale2013 : How to scale ?

1,800

Published on

Quentin ADAM slides on dotScale, about How to scale ?

Published in: Technology

Dotscale2013 : How to scale ?

  1. 1. SCALING,WHAT YOUNEED TOKNOWQUENTIN ADAM @WAXZCECLEVER CLOUD CEO2013
  2. 2. WHEN YOUNEED TOSCALETHERE ARE 2 WAYS
  3. 3. GROWING AND GROWINGUNTIL YOU EXPLODE ORBECOME WEIRD
  4. 4. OR SPLIT THE WORK ANDMAKE YOUR SOFTWAREWORK AS A TEAM
  5. 5. Build an army of fat appYOU CAN DO BOTH
  6. 6. SO WE NEED TO BE ABLETO DISPATCH THE WORKSCALE OUT•  Many workersdoing the samething•  No SPOF•  Growing is moreeasy•  Introduce bestpracticeSCALE UP•  1 Fat instance•  1 Fat application•  SPOF (single pointof failure)•  Hard to maintain•  Always has a limit•  Short termmeaning
  7. 7. SO, HOW TOSCALEOUT ?JUST SOME FACTS
  8. 8. SPLIT PROCESS ANDSTORAGEStorage• Databases• Files• Sessions• Events• …Code• Can be replicated• Stateless• Process
  9. 9. Picking one instance or another doesn’t matterSTATELESSNESS IS THE KEY
  10. 10. CONSIDER MORETHINGS AS DATA•  User account•  Users data•  Files•  Sessions•  Events
  11. 11. CHOOSEDATASTOREWISELYYOU CAN SHOULD USE MANY DATASTORES
  12. 12. DATASTORE CHOICESARE DRIVEN BY USAGEMakedecisionsbased onneedsDo I needatomicity ofrequests ?Do I needconcurrentaccess ?Do I mostlyread orwrite ?Do I needrelational ?Do I needbig storagecapacity ?Do I needhighavailability ?
  13. 13. •  Not a big volume•  DB have to managedata TTL•  Data model : K/V•  Multiple writes at thesame time•  High availabilityI need to store sessionsQUICK EXAMPLE
  14. 14. •  Not a big volumeIt’s OK, PG can handlesmall quantity of data•  DB have to managedata TTLNo, I have to do itmanually•  Data model : K/VNo, PG is relational(mainly)•  Multiple writes at thesame timeNo, PG is Atomic•  High availabilityPG is awesome ;-) Useof PG bouncer orsimilar allow goodclusteringI need to store sessionsQUICK EXAMPLE
  15. 15. •  Not a big volumeIt’s OK, redis canhandle small quantityof data•  DB have to managedata TTLYes Redis can do it•  Data model : K/V Yes•  Multiple writes at thesame timeNo, redis is pseudoAtomic (master/slave)•  High availabilityRedis is great, butcauterization is rude…I need to store sessionsQUICK EXAMPLE
  16. 16. •  Not a big volumeIt’s OK, redis canhandle small quantityof data•  DB have to managedata TTLYes redis can do it•  Data model : K/V Yes•  Multiple writes at thesame timeOK, this is possiblewith memcachedprotocol•  High availabilityClustering is built in, nodowntime JI need to store sessionsQUICK EXAMPLE
  17. 17. USE ONLINEDATABASE / BEREADY TO TESTIN JUST A FEWMINUTESNO NEED TO TRASH YOUR COMPUTER
  18. 18. DON’T BE THAT GUY
  19. 19. DO NOT USE ATECHNOLOGY BECAUSEYOU <3 IT OR BECAUSEIT’S HYPE : USE ITBECAUSE IT FITS YOURNEEDSBALANCE YOUR LEARNING CURVE WITHTHE TIME SAVED
  20. 20. DO NOT CREATE MONSTERS
  21. 21. COMMON MISTAKES
  22. 22. DO NOT USEMEMORY ASDATABASELIKE : SHARED / GLOBAL VARIABLE,CACHE “IN THE CODE”, INTENSIVESESSION USAGE…
  23. 23. DO NOT USE A VARIABLEFOR MORE THAN ONEREQUEST
  24. 24. 2 + 2 = 4FOR SAME INPUT, SAME OUTPUT
  25. 25. And data will be lostCODE WILL FAIL
  26. 26. DO NOT USE FILESYSTEM AS DATASTOREFile system are POSIX compliant•  POSIX is ACID•  POSIX is powerful but is bottleneck•  File System is nightmare of ops•  File System is create coupling (host provider/OS/language)•  Free SPOF multi tenant File System is a unicornSTORE IN DATABASE, OR DATASTORE LIKE S3 (AWS)DEDICATED TO FILE MANAGEMENT
  27. 27. CAREFUL USE OFDARK MAGIC
  28. 28. SPLIT THE CODE :MODULES•  Smallest codebase•  Deploy asservice foreach other•  Focus on besttechnology fora problem
  29. 29. USE EVENT BROKER TOMODULARIZE YOUR APP•  AMQP•  Celery•  0MQ•  Redis•  JMS•  Some case : hadoop, akka…•  …CRON is not an event queue
  30. 30. MAKE HARDCOMPUTATION ASYNC
  31. 31. ALWAYS USE AREVERSE PROXYY U NOT USE ONE ?
  32. 32. USE PROCESSDEPLOYMENT
  33. 33. KEEP CALM UNDERFIRE
  34. 34. TRACK BUG & GETMETRICS
  35. 35. Quentin ADAMTwitter : @waxzceTHX FOR LISTENING& QUESTIONS TIMEcleverc l o u d

×