Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[ScalaMatsuri] グリー初のscalaプロダクト!チャットサービス公開までの苦労と工夫

4,478 views

Published on

ScalaMatsuriでの講演資料です。
http://scalamatsuri.org/

<セッション内容>
グリーでは今年6月にGREEチャットというサービスをリリースしました。
本発表では、GREEチャットのバックエンドをScalaで開発したときの利点や苦労した点についてお話させて頂きます。
トピックは以下の通りです。

-弊社で初めてScalaのプロダクトを導入する際の苦労
-数十万DAUでもリアルタイム性を維持するための工夫
-AkkaやFinagleなどのOSSライブラリを使った非同期処理、並列処理

グリー株式会社 長谷川 貴之&尾崎 俊

Published in: Technology
  • Be the first to comment

[ScalaMatsuri] グリー初のscalaプロダクト!チャットサービス公開までの苦労と工夫

  1. 1. The Trial and Error in Releasing GREE Chat Shun Ozaki, Takayuki Hasegawa Copyright © GREE, Inc. All Rights Reserved. Scala Matsuri2014 B-6 GREE's First Scala Product
  2. 2. Copyright © GREE, Inc. All Rights Reserved. Self Introduction • Shun Ozaki • @wozaki • Joined in April, 2013 • Android Application • Takayuki Hasegawa • @hase1031 • Joined in April, 2013 • NLP, Machine Learning 1/56
  3. 3. Copyright © GREE, Inc. All Rights Reserved. GREE Chat • Released at June 2 2/56
  4. 4. How to build our system for hundreds of thousands daily users Copyright © GREE, Inc. All Rights Reserved. Agenda 3/56
  5. 5. Goal Share our knowledge we got through the development of GREE Chat Copyright © GREE, Inc. All Rights Reserved. • Why Scala? • Team development • Decision of frameworks • Obstacles & workarounds 4/56
  6. 6. Copyright © GREE, Inc. All Rights Reserved. Outline • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary
  7. 7. • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Copyright © GREE, Inc. All Rights Reserved. Outline
  8. 8. Decide the Language based on the Requirement Copyright © GREE, Inc. All Rights Reserved.
  9. 9. Requirements • Hundreds of thousands daily users • Real-time response Copyright © GREE, Inc. All Rights Reserved.  Connect with streaming • Run on a small number of servers  Utilize server resource effectively • Maintain for over 5 years 8/56
  10. 10. Copyright © GREE, Inc. All Rights Reserved. ◎ GREE uses PHP so heavily • Many libraries and know-how in GREE △ Streaming • #connections = #processes △ Single thread, multi process • Overhead of spawning an OS process △ Maintainability • Dynamic typing 9/56
  11. 11. ◎Compatible with concurrent programming ◎One process, many connections ◎High maintainability • Static typing • Functional programming (no side effects) ◎Stimulate new technology learning in GREE • Open to new languages & technologies! Copyright © GREE, Inc. All Rights Reserved. Scala! 10/56
  12. 12. • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Copyright © GREE, Inc. All Rights Reserved. Outline
  13. 13. After adopting Scala … Copyright © GREE, Inc. All Rights Reserved.
  14. 14. Only two Scala programmers in a team of seven Copyright © GREE, Inc. All Rights Reserved. Shortage of Scala experts 2/ 7 13/56
  15. 15. Only two Scala programmers in a team of seven We must build up our skill! Copyright © GREE, Inc. All Rights Reserved. Shortage of Scala experts 2/ 7 14/56
  16. 16. Copyright © GREE, Inc. All Rights Reserved. Learning of Scala 1. Self study 2. Study club 3. Pair programming 15/56
  17. 17. Copyright © GREE, Inc. All Rights Reserved. 1. Self Study Understand the basic syntax
  18. 18. Copyright © GREE, Inc. All Rights Reserved. 1. Self Study Books 17/56
  19. 19. Copyright © GREE, Inc. All Rights Reserved. 1. Self Study Documents by Twitter https://twitter.github.io/scala_school/ http://twitter.github.io/effectivescala/ 18/56
  20. 20. Copyright © GREE, Inc. All Rights Reserved. 1. Self Study Source code from OSS 19/56
  21. 21. Copyright © GREE, Inc. All Rights Reserved. 2. Study Club Share what we learned by ourselves Introduction to functional programming
  22. 22. 2. Study Club • Each engineer solves the problems provided by Scala experts Copyright © GREE, Inc. All Rights Reserved. • e.g., Binary Tree, Fibonacci Sequence 21/56
  23. 23. • Example of factorial generation algorithm Copyright © GREE, Inc. All Rights Reserved. 2. Study Club Discussion with Members Use var Use recursion 22/56
  24. 24. • Example of factorial generation algorithm Use var Use recursion Copyright © GREE, Inc. All Rights Reserved. 2. Study Club Discussion with Members No side effect 23/56
  25. 25. 3. Pair Programming Practice by using the knowledge Copyright © GREE, Inc. All Rights Reserved. obtained from study club
  26. 26. Copyright © GREE, Inc. All Rights Reserved. 3. Pair Programming • Effective learning • Learn about symbols (e.g., +, @), which cannot be searched easily on the Internet • Supplement the knowledge not taught at study club • Complex syntaxes (e.g., implicit, currying) • Development tools (e.g., sbt, IntelliJ) 25/56
  27. 27. Copyright © GREE, Inc. All Rights Reserved. Summary: Adopting Scala • Reasons of selecting Scala • Compatible with concurrent programming • Maintainability because of static typing • Learning of Scala • Self Study, Study Club, Pair Programming • Big burdens on Scala experts • Needed for teaching team members • Maintain the quality of codes 26/56
  28. 28. • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary Copyright © GREE, Inc. All Rights Reserved. Outline
  29. 29. Architecture of Backend (simplified) Copyright © GREE, Inc. All Rights Reserved. • Servers separated by Queues • API Server: Process users’ requests • EventBus Server: Process events in Queue • Stream Server: Supply event to connected users 28/56
  30. 30. Role of Each Server Copyright © GREE, Inc. All Rights Reserved. and Framework
  31. 31. Copyright © GREE, Inc. All Rights Reserved. API Server Process users’ requests • Enqueue events • e.g., Send message Join/leave conversation • Techniques to process a lot of requests • Delegate heavy tasks (e.g., Disk I/O) to others • Logic is written to run asynchronously • scala.concurrent.Future 30/56
  32. 32. • RPC system for JVM based on Netty • Support us to write asynchronous logic Copyright © GREE, Inc. All Rights Reserved. • Used by large scale web services • e.g., Twitter, Tumblr, Foursquare, Pinterest • Equipped with various clients • Redis, Memcached • With retry policy, connection pools 31/56
  33. 33. Copyright © GREE, Inc. All Rights Reserved. EventBus Server Process events in Queue • Examples of tasks • Logging • Inquire GREE internal system • Store events in DB • etc. • Problems of concurrent processing • Deadlock, race condition, … 32/56
  34. 34. Framework to write concurrent, distributed logic more easily • No need to handle shared resources Copyright © GREE, Inc. All Rights Reserved. • Race condition, dead-lock • Logic separation • Parent-Child • Fault tolerance 33/56
  35. 35. Copyright © GREE, Inc. All Rights Reserved. Example of Akka 34/56
  36. 36. Copyright © GREE, Inc. All Rights Reserved. Example of Akka Match by Event type 35/56
  37. 37. Copyright © GREE, Inc. All Rights Reserved. Example of Akka Send copy to child 36/56
  38. 38. Copyright © GREE, Inc. All Rights Reserved. Example of Akka log 37/56
  39. 39. Copyright © GREE, Inc. All Rights Reserved. Stream Server Supply events to connected users • Connect by streaming • Increase connections as much as possible • Keep users data in the memory to reduce I/O • Asynchronous I/O by Finagle, Akka • Task division by Akka • Supply events to users • Update users data • Send KeepAlive to users 38/56
  40. 40. Copyright © GREE, Inc. All Rights Reserved. Summary: Architecture • Each server is separated by Queue • API, EventBus, Stream Server • Framework is selected by considering asynchronous processing • Finagle, Akka • Task is divided by Akka • Easy to write concurrent logic 39/56
  41. 41. Copyright © GREE, Inc. All Rights Reserved. Outline • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary
  42. 42. Copyright © GREE, Inc. All Rights Reserved. Obstacles 1.JVM • Full GC 2. Self-created Scala library • Sharding • ID Architecture 41/56
  43. 43. Full GC Problems Copyright © GREE, Inc. All Rights Reserved.
  44. 44. • Scala runs on JVM • We want to prevent “stop the world” • GC in young generation is faster than full GC • Try not to let it store object in old generation Copyright © GREE, Inc. All Rights Reserved. Full GC Young generation Old generation 43/56
  45. 45. Copyright © GREE, Inc. All Rights Reserved. Causes of Uncollected Reference Problem • Use method that has side effects • Reuse object everywhere • Forget to release used resources 44/56
  46. 46. Copyright © GREE, Inc. All Rights Reserved. Causes of Uncollected Reference Problem • Use method that has side effects • Reuse object everywhere • Forget to release used resources Difficult to recognize by developers 45/56
  47. 47. Copyright © GREE, Inc. All Rights Reserved. Short-Lived Object • Use val variable (final variable) • Don’t use mutable variable • Don’t leave references to objects for too long Create new objects instead of updating old ones 46/56
  48. 48. Sharding: Scalable Storage 47/56 Copyright © GREE, Inc. All Rights Reserved.
  49. 49. Copyright © GREE, Inc. All Rights Reserved. Scale up? 48/56
  50. 50. Copyright © GREE, Inc. All Rights Reserved. Scale out! 49/56
  51. 51. Copyright © GREE, Inc. All Rights Reserved. Horizontal Partitioning • Sharding is necessary! • Considering capacity and #accesses • We have Cascade, a sharding library • For PHP, not Scala • So, we created Aurora and make it OSS • Aurora and application were developed simultaneously • So difficult to integrate them during development • Code modification must be done in many parts 50/56
  52. 52. ID Architecture: UUID and InnoDB Ref:MySQL InnoDB Primary Key Choice: GUID/UUID vs Integer Insert Performance http://kccoder.com/mysql/uuid-vs-int-insert-performance/ Copyright © GREE, Inc. All Rights Reserved.
  53. 53. Ref:MySQL InnoDB Primary Key Choice: GUID/UUID vs Integer Insert Performance http://kccoder.com/mysql/uuid-vs-int-insert-performance/ Copyright © GREE, Inc. All Rights Reserved. UUID AUTO_INCREMENT #record #time
  54. 54. Drop UUID, Take Time-Based • At first, we used UUID as primary key • But this caused MySQL to insert randomly • Created a library to generate ID instead • Referred to Snowflake by Twitter • Sequential insertion based on sorted created time 53/56 Copyright © GREE, Inc. All Rights Reserved.
  55. 55. Copyright © GREE, Inc. All Rights Reserved. Summary: Obstacles & Workaround • JVM • Use short-lived objects to avoid Full GC • Self-created Scala library • Sharding • Made Scala library OSS • Integration of library to application was difficult • ID Architecture • Adopt time-based ID for sequential insertion 54/56
  56. 56. Copyright © GREE, Inc. All Rights Reserved. Outline • Reason of Selecting Scala • Learning Scala in a Team • Architecture of GREE Chat • Obstacles • Summary
  57. 57. Copyright © GREE, Inc. All Rights Reserved. Summary • Reasons for selecting Scala • Compatible with concurrent programming • Plan Scala learning within the team • Learning cost is expensive • Architecture & Frameworks • Each server is separated by Queue • Finagle, Akka • Shortage of libraries • We must have made libraries by ourselves • Needed to pull request to OSS repository 56/56
  58. 58. Copyright © GREE, Inc. All Rights Reserved. Thanks! @tomoyoshi_ogura, @kyo_ago, @takc923 @yoshie_777, @le_chang, @beketa, @j5ik2o and ScalaMatsuri staffs

×