Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Real world erlang

1,811 views

Published on

  • Be the first to comment

Real world erlang

  1. 1. Real World Erlang VOYAGE GROUP @ajiyoshi
  2. 2. Advertisement System HERE!
  3. 3. RTB• Real Time Bidding• real time auction for each ad impression to decide price
  4. 4. Ad Request Ad ServerBrowser for Publisher (SSP)
  5. 5. Bid Request Ad ServerBrowser for Publisher (SSP) Ad Server for Advertiser (DSP)
  6. 6. Bid 10 Ad Server 20Browser for Publisher (SSP) Ad Server 30 for Advertiser (DSP) 15
  7. 7. Auction 10 Ad Server 20 Browser for Publisher (SSP) Ad Server 30 for Advertiser (DSP) 15 ※usually “second price auction”2nd highest bid price will be the contract price
  8. 8. Display winner Ad 10 Ad Server 20Browser for Publisher (SSP) Ad Server 30 for Advertiser (DSP) 15
  9. 9. Problem• How will you design the system?
  10. 10. Requirements• stable• fast! fast! fast! • < 50ms (if possible) • don’t block in network IO • timeout (IMPORTANT!)
  11. 11. Non blocking IO• multi process, multi thread• IO multiplexing • select epoll libeio etc.• IO multiplexing + event driven • libev etc.
  12. 12. TIMEOUT
  13. 13. Network timeout (curl)• CURLOPT_CONNECTTIMEOUT• CURLOPT_CONNECTTIMEOUT_MS• CURLOPT_TIMEOUT• CURLOPT_TIMEOUT_MS
  14. 14. Network timeout (curl)• CURLOPT_TIMEOUT_MS • “cURL 関数の実行にかけられる最大のミ リ秒数。 システムの標準の名前解決を 使うように libcurl をビルドしている場合 は、 接続のタイムアウトは秒単位の精 度となり、最小のタイムアウトは 1 秒と なります。
  15. 15. Timeout• timeout for each connection • C socket API has “connect()” and “recv()” • need to manage each timeout • what we need is “round trip” timeout• timeout for total auction process • hard to implement in some languages like C • watch dog thread for each processes? • multi processes + signal?
  16. 16. My options• write in C with libev, libeio and so on• use node.js (it uses libev libeio)• light weight process(like Erlang)
  17. 17. Write in C• pros • Fast!
  18. 18. Write in C• cons • self memory management in the HELL • multi thread in C in the HELL • timeout in the HELL
  19. 19. use node.js• pros • abstracted event loop and IO multiplexing • auto memory management
  20. 20. use node.js• cons • callback HELL • single threaded • timeout for whole process in the HELL • NOT stable • many upgrades
  21. 21. Erlang• pros • with lightweight process • stable • easy to write networking system • fitting perfectly for the requirements
  22. 22. Erlang• cons • few Erlang programmers
  23. 23. Erlang
  24. 24. Erlang performancehttp://dsas.blog.klab.org/archives/51993306.html proc#/sec of TCP echo
  25. 25. Observations“しかし、CPU負荷をモニタリングしていると、thread版はほんの少し速いだけなのにCPUを200% 使いきっており、CPU負荷のうちでも sys が多い状況になっていました。これは、ネイティブスレッドの コンテキストスイッチの負荷だと思います。なので、最速TCPサーバーの条件とは、基本的にネイティブスレッドではなく軽量なユーザーランド スレッドかイベント駆動方式で接続の多重化を行いつつ、なおかつ複数コアを利用するために コア数程度のネイティブスレッドかプロセスを利用するという物になると思います。”
  26. 26. Let’s go!• implement RTB daemon in Erlang • other system is not Erlang • receive request from other system, return auction result• 120msec timeout in each process
  27. 27. Performance• about 3 Billion bids/month• about 110 Million bids/day• about 5000 bids/sec (peak time)
  28. 28. Timeouttotal ad deliver time msec→ # (logarithmic)
  29. 29. stable• no memory leak• no bottle neck• no crash (including VM)
  30. 30. Requirements(again)• stable→OK• fast! fast! fast!→OK • no network blocking →OK • by lightweight process • timeout(IMPORTANT!)→OK • by lightweight process
  31. 31. Real world Erlang• Erlang is awsome• very fit to handling many many little network IOs• stable• easy to write

×