High Performance Erlang

Erlang and Scalability Percona Performance 2009 Jan Henry Nystrom henry@erlang-consulting.com 2007 Course Introduction Course Title @ Course Author

Introduction • Scalability Killers • Design Decisions – Language and Yours • Thinking Scalable/Parallel • Code for the correct case • Rules of Thumb • Scalability in the small: SMP Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 2

Design Decisions No implicit synchronization • Spawn always succeed • Sending always succeed • Random access message buffer • Fire and forget unless you need the synchronization Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 5

Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6

Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6

Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support • Sets the focus firmly on the concurrent tasks Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6

Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support • Sets the focus firmly on the concurrent tasks • Code for the correct case Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6

Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support • Sets the focus firmly on the concurrent tasks • Code for the correct case • Clear Code Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6

Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support • Sets the focus firmly on the concurrent tasks • Code for the correct case • Clear Code Clarity is King! Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6

Design Decisions Concurrency oriented programming • Concurrency support an integral part of the language • Distribution support • Sets the focus firmly on the concurrent tasks • Code for the correct case • Clear Code Clarity is King! I rather try to get clear code correct than correct code clear Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 6

Thinking Scalable/Parallel In the Erlang setting • Do not introduce unneeded synchronization • Remember processes are cheap • Do not introduce unneeded synchronization • A terminated process is all garbage • Do not introduce unneeded synchronization Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 9

Code for the Correct Case request set timer request set timer request set timer r answe release timer check r answe release timer check er answ release timer check Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 10

Rules of Thumb • Rule 1 - All independent tasks should be processes • Rule 2 - Do not invent concurrency that is not there! f() g() h() Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 12

Rules of Thumb • Rule 1 - All independent tasks should be processes • Rule 2 - Do not invent concurrency that is not there! f() g() h(g(f())) h() Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 12

Rules of Thumb • Rule 1 - All independent tasks should be processes • Rule 2 - Do not invent concurrency that is not there! f() g() h(g(f())) h(g(f())) h(g(f())) h(g(f())) h() Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 12

Scalability in the small: SMP Erlang SMP ”Credo” SMP should be transparent to the programmer in much the same way as Erlang Distribution • You shouldn’t have to think about it ...but sometimes you must • Use SMP mainly for stuff that you’d make concurrent anyway • Erlang uses concurrency as a structuring principle • Model for the natural concurrency in your problem Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 13

Scalability in the small: SMP ”Big bang” benchmark on Sunfire T2000 • Erlang on multicore 1 scheduler • SMP prototype ‘97, First OTP release May ‘06. • Mid -06 benchmark mimicking call handling (axdmark) on the (experimental) SMP emulator. Observed speedup/core: 0.95 • First Ericsson product (TGC) released on SMP Erlang in Q207. 16 schedulers Simultaneous processes Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 14

Scalability in the small: SMP Case Study: Telephony Gateway Controller AXE TGC • Mediates between legacy telephony and multimedia networks. • Hugely complex state machines GW GW GW • + massive concurrency. • Developed in Erlang. • Multicore version shipped to customer Q207. • Porting from 1-core PPC to 2-core Intel took < 1 man-year (including testing). Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 15

Scalability in the small: SMP Case Study: Telephony Gateway Controller IS/GCP IS/GEP IS/GEP AXD AXD Traffic scenario CPB5 CPB6 1slot/board Dual core Dual core One core Two cores running running 2slots/board 2slots/board POTS-POTS / 4.3X call/sec X call/sec 2.3X call/sec 0.4X 2.1X call/sec AGW call/sec OTP R11_3 beta One core used +patches ISUP-ISUP /Inter 13X call/sec 3.6X call/sec 7.7X call/sec 1.55X 7.6X call/sec MGW call/sec OTP R11_3 beta One core used +patches ISUP-ISUP /Intra 5.5X call/sec 26X call/sec 3.17X 14X call/sec MGW call/sec Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 16

Scalability in the small: SMP Speedup on 4 Hyper Threaded Pentium4 5.00 3.96 3.79 3.75 3.63 Speddup 3.11 2.73 2.50 2.05 1.92 1.25 1.00 0 1 2 3 4 5 6 7 8 # Schedulers • Chatty • 1000 processes created • Each process randomly sends req/recieves ack from all other processes Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 17

Scalability in the small: SMP Erlang VM Current SMP VM run queue OTP R11/R12 Scheduler #1 Scheduler #2 Scheduler #N Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 19

Scalability in the small: SMP Erlang VM New SMP VM OTP R13 Scheduler #1 run queue Released 21th April migration Scheduler #2 run queue logic Scheduler #N run queue Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 20

Scalability in the small: SMP Multiple run queues Speedup: Ca 0.43 * N @ 32 cores Memory allocation locks dominate... Single run queue • Speedup of ”Big Bang” on a Tilera Tile64 chip (R13A) • 1000 processes, all talking to each other Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 21

Scalability in the small: SMP Shift in Bottlenecks • All scalable Erlang systems were stress tested  for CPU usage  for network usage • With SMP hardware we must stress test for memory usage • In the typical SMP system, the bottleneck has shifted from the CPU to the memory Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 22

Scalability in the small: SMP Death by a thousand cuts • Many requests that generate short spikes in memory usage • Limit or serialize those requests • More on this in coming paper from CTO Ulf Wiger loop(State) -> receive {request, typeA, Data} -> Data1 = allocate_lots_of_memory(Data), a_server ! {request, typeA, self()}, receive {answer, … Erlang and Scalability Percona Performance Conference © 2009 -2009, Erlang Training and Consulting 23

High Performance Erlang

0 comments

Notes on slide 1

Favorites, Groups & Events