Erlang and
                       Scalability
            Percona Performance 2009



Jan Henry Nystrom
henry@erlang-consu...
Introduction
•     Scalability Killers
•     Design Decisions – Language and Yours
•     Thinking Scalable/Parallel
•     ...
Scalability Killers
• Synchronization
• Resource contention




Erlang and Scalability   Percona Performance Conference © ...
Scalability Killers

• Synchronization




Erlang and Scalability   Percona Performance Conference © 2009 -2009, Erlang Tr...
Design Decisions
                                               No sharing

• Processes
• Encapsulation
• No implicit sync...
Design Decisions
                         No implicit synchronization

•     Spawn always succeed
•     Sending always suc...
Design Decisions




Erlang and Scalability   Percona Performance Conference © 2009 -2009, Erlang Training and Consulting ...
Design Decisions
              Concurrency oriented programming




Erlang and Scalability   Percona Performance Conferenc...
Design Decisions
              Concurrency oriented programming

• Concurrency support an integral part of the language


...
Design Decisions
              Concurrency oriented programming

• Concurrency support an integral part of the language
• ...
Design Decisions
              Concurrency oriented programming

• Concurrency support an integral part of the language
• ...
Design Decisions
              Concurrency oriented programming

•     Concurrency support an integral part of the languag...
Design Decisions
              Concurrency oriented programming

•     Concurrency support an integral part of the languag...
Design Decisions
              Concurrency oriented programming

•     Concurrency support an integral part of the languag...
Design Decisions
              Concurrency oriented programming

•     Concurrency support an integral part of the languag...
Design Decisions
              Concurrency oriented programming

•     Concurrency support an integral part of the languag...
Design Decisions
              Concurrency oriented programming

•     Concurrency support an integral part of the languag...
Thinking Scalable/Parallel


      0




Erlang and Scalability   Percona Performance Conference © 2009 -2009, Erlang Trai...
Thinking Scalable/Parallel
                         List length: Obviously Linear

      4:




                         B...
Thinking Scalable/Parallel
 List length: O(logN) with sufficient processors




Erlang and Scalability   Percona Performan...
Thinking Scalable/Parallel
 List length: O(logN) with sufficient processors

          4


           2                   ...
Thinking Scalable/Parallel
                                 In the Erlang setting

•     Do not introduce unneeded synchro...
Code for the Correct Case




Erlang and Scalability   Percona Performance Conference © 2009 -2009, Erlang Training and Co...
Code for the Correct Case


                                    request
        set timer
                                ...
Code for the Correct Case


                                     request
         set timer
                              ...
Code for the Correct Case


                                     request
         set timer
                              ...
Code for the Correct Case




Erlang and Scalability   Percona Performance Conference © 2009 -2009, Erlang Training and Co...
Code for the Correct Case


                                    request
        set timer
                                ...
Code for the Correct Case


                                    request
        set timer
                                ...
Rules of Thumb
• Rule 1 - All independent tasks should be processes
• Rule 2 - Do not invent concurrency that is not there...
Rules of Thumb
• Rule 1 - All independent tasks should be processes
• Rule 2 - Do not invent concurrency that is not there...
Rules of Thumb
• Rule 1 - All independent tasks should be processes
• Rule 2 - Do not invent concurrency that is not there...
Rules of Thumb
• Rule 1 - All independent tasks should be processes
• Rule 2 - Do not invent concurrency that is not there...
Scalability in the small: SMP
                                  Erlang SMP ”Credo”


 SMP should be transparent to the pro...
Scalability in the small: SMP
                                                                         ”Big bang” benchmar...
Scalability in the small: SMP
   Case Study: Telephony Gateway Controller

                                               ...
Scalability in the small: SMP
   Case Study: Telephony Gateway Controller
                          IS/GCP                ...
Scalability in the small: SMP
   Case Study: Telephony Gateway Controller
                          IS/GCP                ...
Scalability in the small: SMP
                                                   Speedup on 4 Hyper Threaded Pentium4
    ...
Scalability in the small: SMP
                                                    Erlang VM
non-SMP VM
                   ...
Scalability in the small: SMP
                                                       Erlang VM
Current SMP VM
            ...
Scalability in the small: SMP
                                              Erlang VM
New SMP VM
  OTP R13                ...
Scalability in the small: SMP
                                         Multiple
                                        ru...
Scalability in the small: SMP
                                  Shift in Bottlenecks

• All scalable Erlang systems were s...
Scalability in the small: SMP
                          Death by a thousand cuts

• Many requests that generate short spik...
Questions




                                                   ???


Erlang and Scalability   Percona Performance Confer...
Upcoming SlideShare
Loading in …5
×

High Performance Erlang

3,955 views

Published on

Published in: Technology, News & Politics
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,955
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
97
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

High Performance Erlang

  1. 1. Erlang and Scalability Percona Performance 2009 Jan Henry Nystrom henry@erlang-consulting.com 2007 Course Introduction Course Title @ Course Author
  2. 2. Introduction • Scalability Killers • Design Decisions – Language and Yours • Thinking Scalable/Parallel • Code for the correct case • Rules of Thumb • Scalability in the small: SMP Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 2
  3. 3. Scalability Killers • Synchronization • Resource contention Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 3
  4. 4. Scalability Killers • Synchronization Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 3
  5. 5. Design Decisions No sharing • Processes • Encapsulation • No implicit synchronization Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 4
  6. 6. Design Decisions No implicit synchronization • Spawn always succeed • Sending always succeed • Random access message buffer • Fire and forget unless you need the synchronization Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 5
  7. 7. Design Decisions Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6
  8. 8. Design Decisions Concurrency oriented programming Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6
  9. 9. Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6
  10. 10. Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6
  11. 11. Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support • Sets the focus firmly on the concurrent tasks Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6
  12. 12. Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support • Sets the focus firmly on the concurrent tasks • Code for the correct case Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6
  13. 13. Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support • Sets the focus firmly on the concurrent tasks • Code for the correct case • Clear Code Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6
  14. 14. Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support • Sets the focus firmly on the concurrent tasks • Code for the correct case • Clear Code Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6
  15. 15. Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support • Sets the focus firmly on the concurrent tasks • Code for the correct case • Clear Code Clarity is King! Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6
  16. 16. Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support • Sets the focus firmly on the concurrent tasks • Code for the correct case • Clear Code Clarity is King! Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6
  17. 17. Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support • Sets the focus firmly on the concurrent tasks • Code for the correct case • Clear Code Clarity is King! I rather try to get clear code correct than correct code clear Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6
  18. 18. Thinking Scalable/Parallel 0 Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 7
  19. 19. Thinking Scalable/Parallel List length: Obviously Linear 4: But not when you have n processors? Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 7
  20. 20. Thinking Scalable/Parallel List length: O(logN) with sufficient processors Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 8
  21. 21. Thinking Scalable/Parallel List length: O(logN) with sufficient processors 4 2 2 1 1 1 1 Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 8
  22. 22. Thinking Scalable/Parallel In the Erlang setting • Do not introduce unneeded synchronization • Remember processes are cheap • Do not introduce unneeded synchronization • A terminated process is all garbage • Do not introduce unneeded synchronization Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 9
  23. 23. Code for the Correct Case Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 10
  24. 24. Code for the Correct Case request set timer request set timer request set timer Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 10
  25. 25. Code for the Correct Case request set timer request set timer request set timer r answe release timer check r answe release timer check er answ release timer check Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 10
  26. 26. Code for the Correct Case request set timer request set timer request set timer r answe release timer check r answe release timer check er answ release timer check Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 10
  27. 27. Code for the Correct Case Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 11
  28. 28. Code for the Correct Case request set timer request request Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 11
  29. 29. Code for the Correct Case request set timer request request nswer a Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 11
  30. 30. Rules of Thumb • Rule 1 - All independent tasks should be processes • Rule 2 - Do not invent concurrency that is not there! f() g() h() Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 12
  31. 31. Rules of Thumb • Rule 1 - All independent tasks should be processes • Rule 2 - Do not invent concurrency that is not there! f() g() h() Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 12
  32. 32. Rules of Thumb • Rule 1 - All independent tasks should be processes • Rule 2 - Do not invent concurrency that is not there! f() g() h(g(f())) h() Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 12
  33. 33. Rules of Thumb • Rule 1 - All independent tasks should be processes • Rule 2 - Do not invent concurrency that is not there! f() g() h(g(f())) h(g(f())) h(g(f())) h(g(f())) h() Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 12
  34. 34. Scalability in the small: SMP Erlang SMP ”Credo” SMP should be transparent to the programmer in much the same way as Erlang Distribution • You shouldn’t have to think about it ...but sometimes you must • Use SMP mainly for stuff that you’d make concurrent anyway • Erlang uses concurrency as a structuring principle • Model for the natural concurrency in your problem Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 13
  35. 35. Scalability in the small: SMP ”Big bang” benchmark on Sunfire T2000 • Erlang on multicore 1 scheduler • SMP prototype ‘97, First OTP release May ‘06. • Mid -06 benchmark mimicking call handling (axdmark) on the (experimental) SMP emulator. Observed speedup/core: 0.95 • First Ericsson product (TGC) released on SMP Erlang in Q207. 16 schedulers Simultaneous processes Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 14
  36. 36. Scalability in the small: SMP Case Study: Telephony Gateway Controller AXE TGC • Mediates between legacy telephony and multimedia networks. • Hugely complex state machines GW GW GW • + massive concurrency. • Developed in Erlang. • Multicore version shipped to customer Q207. • Porting from 1-core PPC to 2-core Intel took < 1 man-year (including testing). Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 15
  37. 37. Scalability in the small: SMP Case Study: Telephony Gateway Controller IS/GCP IS/GEP IS/GEP AXD AXD Traffic scenario CPB5 CPB6 1slot/board Dual core Dual core One core Two cores running running 2slots/board 2slots/board POTS-POTS / 4.3X call/sec X call/sec 2.3X call/sec 0.4X 2.1X call/sec AGW call/sec OTP R11_3 beta One core used +patches ISUP-ISUP /Inter 13X call/sec 3.6X call/sec 7.7X call/sec 1.55X 7.6X call/sec MGW call/sec OTP R11_3 beta One core used +patches ISUP-ISUP /Intra 5.5X call/sec 26X call/sec 3.17X 14X call/sec MGW call/sec Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 16
  38. 38. Scalability in the small: SMP Case Study: Telephony Gateway Controller IS/GCP IS/GEP IS/GEP AXD AXD Traffic scenario CPB5 CPB6 1slot/board Dual core Dual core One core Two cores running running 2slots/board 2slots/board POTS-POTS / 4.3X call/sec X call/sec 2.3X call/sec 0.4X 2.1X call/sec AGW call/sec OTP R11_3 beta One core used +patches ISUP-ISUP /Inter 13X call/sec 3.6X call/sec 7.7X call/sec 1.55X 7.6X call/sec MGW call/sec OTP R11_3 beta One core used +patches ISUP-ISUP /Intra 5.5X call/sec 26X call/sec 3.17X 14X call/sec MGW call/sec Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 16
  39. 39. Scalability in the small: SMP Speedup on 4 Hyper Threaded Pentium4 5.00 3.96 3.79 3.75 3.63 Speddup 3.11 2.73 2.50 2.05 1.92 1.25 1.00 0 1 2 3 4 5 6 7 8 # Schedulers • Chatty • 1000 processes created • Each process randomly sends req/recieves ack from all other processes Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 17
  40. 40. Scalability in the small: SMP Erlang VM non-SMP VM run queue Scheduler Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 18
  41. 41. Scalability in the small: SMP Erlang VM Current SMP VM run queue OTP R11/R12 Scheduler #1 Scheduler #2 Scheduler #N Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 19
  42. 42. Scalability in the small: SMP Erlang VM New SMP VM OTP R13 Scheduler #1 run queue Released 21th April migration Scheduler #2 run queue logic Scheduler #N run queue Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 20
  43. 43. Scalability in the small: SMP Multiple run queues Speedup: Ca 0.43 * N @ 32 cores Memory allocation locks dominate... Single run queue • Speedup of ”Big Bang” on a Tilera Tile64 chip (R13A) • 1000 processes, all talking to each other Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 21
  44. 44. Scalability in the small: SMP Shift in Bottlenecks • All scalable Erlang systems were stress tested  for CPU usage  for network usage • With SMP hardware we must stress test for memory usage • In the typical SMP system, the bottleneck has shifted from the CPU to the memory Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 22
  45. 45. Scalability in the small: SMP Death by a thousand cuts • Many requests that generate short spikes in memory usage • Limit or serialize those requests • More on this in coming paper from CTO Ulf Wiger loop(State) -> receive {request, typeA, Data} -> Data1 = allocate_lots_of_memory(Data), a_server ! {request, typeA, self()}, receive {answer, … Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 23
  46. 46. Questions ??? Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 24

×