Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ScalaMatsuri 2016 ドワンゴアカウントシステムを支えるScala技術


Published on @pab_tech

Published in: Software
  • Be the first to comment

ScalaMatsuri 2016 ドワンゴアカウントシステムを支えるScala技術

  1. 1. 1 Scala technology supporting the Dwango account system Seitaro Yuki / @pab_tech / DWANGO Co., Ltd. ドワンゴアカウントシステムを支えるScala技術 結城清太郎/ @pab_tech / 株式会社ドワンゴ
  2. 2. 2 Overview Dwango is often named as a leading company to adopt Scala among the enterprises in Japan. The account system is known to be largest Scala project in our company, responsible for registering niconico users and authenticating them. In this session we will introduce a variety of Scala programming techniques that are used in the account system ドワンゴのアカウントシステムで使われている Scalaプログラミング技術について解説します
  3. 3. 3 Contents Web applications by continuation monads Dependency injection by Minimal Cake Pattern New transaction monad using subtyping 継続モナド、Minimal Cake Pattern、トランザ クションモナドについて解説します
  4. 4. 4 Web applications by continuation monads 継続モナドによるWebアプリケーション
  5. 5. 5 Technical problems: How to construct web components Web application needs to be decomposed to request processing, response processing, filtering processing, etc. Each component needs to be easy to understand and test We want a simple and powerful component technology 課題: Webアプリケーションの部品化
  6. 6. 6 Simple and powerful component technology: Monads We will explain how to build a web application using continuation monads 継続モナドを使ったWebアプリケーションの 構成方法を説明します
  7. 7. 7 Similarity of a typical web application and continuation monads まずは一般的なWebアプリケーションの構造 と継続モナドの類似性から見ていきましょう
  8. 8. 8 Typical web application structure Typical web applications have nested structure like onions e.g. Java Servlet, Python WSGI, Ruby Rack Middleware performs the following operations It processes a request, and passes the result to the next middleware It modifies a response that came back, and returns the result to the before middleware Or, when an error occurs it returns the result without calling the next middleware 典型的なWebアプリケーションの構造
  9. 9. 9 Typical web application structure Web application Web Browser Web Server Filter A Filter B Main Processing request request request request response response response response 典型的なWebアプリケーションの構造
  10. 10. 10 Authentication middleware Let's consider authentication middleware We assume that users logined already somewhere Authentication middleware retrieves a session ID from cookies, confirms using a storage such as Redis, passes the result to the next middleware Or, if there is no session ID or the session is invalid, authentication middleware redirects to a login form 認証処理をおこなうミドルウェアについて考 えてみる
  11. 11. 11 Authentication middleware Web Browser Web Server Authentication Filter Main Processing request request alt ["Authentication is successful"] request request response response response ["Authentication is failed"] redirect response 認証処理のシークエンス図
  12. 12. 12 Code this authentication middleware using continuation monad では、この認証処理を継続モナドを使って書 いてみましょう
  13. 13. 13 What is continuation? Continuation represents the rest of the computation at a given point in the process's execution Continuation from A Continuation from B B R Caller A Caller プログラムの実行のある時点において評価さ れていない残りのプログラムを継続と呼ぶ
  14. 14. 14 Continuation-passing style (CPS) CPS is a style of programming in which control is passed a continuation explicitly Caller Caller A A B B R R pass a continuation after A and execute pass a continuation after B and execute pass a continuation after R and execute execute the continuation execute the continuation execute the continuation 継続を明示的に受け渡しするプログラミング スタイルを継続渡しスタイルと呼ぶ
  15. 15. 15 CPS and web applications are similar Web application seems to be able to be written in CPS Further if continuation is monad, it becomes a simple and powerful component CPSとWebアプリケーションは似ている 継続がモナドならさらによい
  16. 16. 16 Continuation monad Ais a type of value at a given point, Ris a final result type A=>Rrepresents a continuation (A=>R)=>Rrepresents a function that take a continuation Cont[R,A]is a constructor that creates a monad from a function that take a continuation 継続モナドは、継続を受け取る関数を受け取 って、モナドを作る case class Cont[R, A](run: (A => R) => R) { def map[B](f: A => B): Cont[R, B] = Cont(k => run(a => k(f(a)))) def flatMap[B](f: A => Cont[R, B]): Cont[R, B] = Cont(k => run(a => f(a).run(k))) }
  17. 17. 17 Let's make web application using continuation monads 継続モナドを使ってWebアプリケーションを 作ってみよう
  18. 18. 18 Authentication of continuation monad 継続モナドによる認証処理 def readSessionIdFromCookie(request: Request): Option[SessionId] = ??? def readSessionFromRedis(sessionId: SessionId): Option[Session] = ??? def redirectLoginForm: Response = ??? def auth(request: Request): Cont[Response, Session] = Cont((k: Session => Response) => readSessionIdFromCookie(request) .flatMap(readSessionFromRedis) .map(k) .getOrElse(redirectLoginForm)
  19. 19. 19 Web application using for-expression for式を使ってWebアプリケーションを組み 立てる def compress(request: Request): Cont[Response, Unit] = ??? def mainProcessing(request: Request, session: Session): Cont[Response, def webApplication(request: Request): Cont[Response, Response] = for { session <- auth(request) _ <- compress(request) response <- mainProcessing(request, session) } yield response // execution webApplication(request).run(identity) // => Response
  20. 20. 20 Let's look at Play Action composition methods for comparison 比較のためにPlay標準のAction合成方法を見 てみよう
  21. 21. 21 Play Action composition methods ActionFunction def invokeBlock ActionBuilder ActionRefiner ActionFilter ActionTransformerAction The methods of Play Action composision is performed by functional composition PlayのAction合成方法 ActionFunctionの合成によっておこなわれる
  22. 22. 22 Actually continuation also appears in Play block:P[A]=>Future[Result]is continuation But it was not a monad 実はPlayでも継続の概念が使われている しかしモナドではなかった trait ActionFunction[-R[_], +P[_]] { def invokeBlock[A](request: R[A], block: P[A] => Future[Result]): Future }
  23. 23. 23 Problems of Play Action composition Functional composition of ActionFunctionis less flexible than monadic composition For example, AuthenticatedBuilderreturns AuthenticatedRequest Therefore, AuthenticatedBuilder can be composed to only ActionFunctionreceives AuthenticatedRequest PlayのAction合成の問題点 ActionFunctionの合成は柔軟性に欠ける
  24. 24. 24 Advantages of using a continuation monad Continuation monad is simple It can be represented by 6 lines Monadic composition is more flexible rather than functional composition It is possible to combine with other kinds of monad by using monad transformers The account system uses ContT[Future,Result,A] for error handling 継続モナドをWebアプリケーションの記述に 使う利点
  25. 25. 25 Dependency Injection by Minimal Cake Pattern Minimal Cake PatternによるDependency Injection
  26. 26. 26 Technical problems: Large-scale system development The account system has 230,000 lines in only code of Scala For comparison, Play has 90,000 lines, Slick has 30,000 lines How to modularize How to test 課題: 大規模システム開発 23万行のScalaコードがある
  27. 27. 27 Let's solve these problems by the dependency injection Dependency Injectionでこれらの課題を解決 しよう
  28. 28. 28 DI technology that we have adopted DI technology that we have adopted is defined as "Minimal Cake Pattern" We will explain the following contents Comparison of various other DI technologies of Scala What is difference from the existing Cake Pattern Advantages for large-scale system development ここでは我々が採用している"Minimal Cake Pattern"というDI技術について解説をします
  29. 29. 29 Various methods of DI in Scala DI Container such as Guice Cake pattern Reader monad 様々なScalaのDependency Injection方法
  30. 30. 30 DI Container such as Guice Advantages Experience in Java Also adopted in Play Dependencies in source code are eliminated completely Shortening of compile time Disadvantages DI container often causes a run-time error of object creation We should learn about the DI container GuiceなどのDIコンテナを使う方法の利点と 欠点
  31. 31. 31 Cake pattern Advantages Checking at compile time Frameworks such as DI container are not required Disadvantages Cake patterns would complicate a system as a component technology Increase boilerplates Cakeパターンを使う方法の利点と欠点
  32. 32. 32 Reader monad Advantages Same as Cake Pattern Disadvantages It will be full of reader monads in everywhere It is difficult to manage a large number of dependencies It is sensitive to change such as IO monads in Haskell We conclude that reader monads should not be used for DI Readerモナドを使う方法の利点と欠点
  33. 33. 33 Which is good for large-scale system development? 大規模システム開発ならどの方法がいいの か?
  34. 34. 34 Rethink of Cake Pattern Cake Pattern is a proposed component technique in "Scalable Component Abstractions" by Odersky's et al. But, there are some problems as a component technique By using an abstract type, types are likely to become too complicated Cakes are likely to be tightly coupled to each other We will consider Cake Pattern not as a component technique コンポーネント技術としてのCake Patternの 問題点
  35. 35. 35 Mimimal Cake Pattern Do not use advanced features such as abstract types Provide only one field variable Do not nest class Define the Cake Pattern with such features as "Mimimal Cake Pattern" Mimimal Cake Patternとはコンポーネント技 術ではないCake Patternである
  36. 36. 36 Example of not use DI まずはDIを使わない例 // Interface trait Repository // Implementation object RepositoryImpl extends Repository // Client object Service { // ↓Here's the problem referring to the Implementation val repository: Repository = RepositoryImpl }
  37. 37. 37 Example of Mimimal Cake Pattern (Interface) Add Usestrait to declare interface Usestrait just provides one field variable Do not use abstract types and nested classes Interfaceの例 Usesトレイトを追加する trait UsesRepository { val repository: Repository } trait Repository
  38. 38. 38 Example of Mimimal Cake Pattern (Implementation) Add MixIntrait to provide the implementation 実装の例 MixInトレイトを追加する trait MixInRepository extends UsesRepository { val repository = repositoryImpl } object RepositoryImpl extends Repository
  39. 39. 39 Example of Mimimal Cake Pattern(Client) Extend UsesRepositoryfor using Repository Use MixInRepositoryin MixInService for providing Service implementation We can replace the implementation here UsesとMixInを使ったServiceの例 ServiceもMixInトレイトを提供する trait Service extends UsesRepository trait MixInService extends UsesService { val service = new Service with MixInRepository }
  40. 40. 40 Development style using Minimal Cake Pattern Our development style is based on Minimal Cake Pattern Interface, Usestrait, implementation, MixIntraits and test form a development unit We should write a comment for all methods in the interface, including all error values Tests should include all error cases in the interface Only when there are all present, the code is ready to be reviewed Mimimal Cake Patternによる開発スタイル
  41. 41. 41 Advantage of Minimal Cake Pattern The account system consists of 400 or more modules We need to simplify cake pattern by discarding extra features not related to DI Static checking is desirable in large-scale system If we use DI container, it should be verified by starting the system each time DI has established our development style We want to re-evaluate Cake Pattern by limiting only to DI Mimimal Cake Patternの利点の説明 DIだけのCake Patternを再評価したい
  42. 42. 42 New transaction monad using subtyping サブタイピングを使った新たなトランザクシ ョンモナド
  43. 43. 43 Technical problems: Abstraction of storage access We got the power to abstract components by DI We want to do even abstraction of storage access, such as MySQL and Redis How to abstract transaction objects of the infrastructure layer, such as DB session How to compose transactions We want to facilitate a programming of Master/Slave structure 技術的な課題: ストレージアクセスの抽象化
  44. 44. 44 We will introduce a new transaction monad to solve these problems これらの問題を解決する新しいトランザクシ ョンモナドを紹介します
  45. 45. 45 Transaction monad: Fujitask Fujitask enables abstraction of storage access and composition of transactions Fujitask treats transaction objects in the same way as a reader monad Fujitask determines whether the composed monad will query Master or Slave by using subtyping トランザクションモナド: Fujitask
  46. 46. 46 Common problems of the Master / Slave structure In programming for Master/Slave structure, often we should explicitly specify whether each code will query Master or Slave A run-time errors when send a query of update to Slave It is hard to notice that we send a query that should be sent to Slave to Master because no error occurs Fujitask will solve the these problems Master/Slave構成のよくある問題 Fujitaskはこのような問題を解決します
  47. 47. 47 Basic parts of Fujitask Fujitaskの基本部分 trait Task[-R, +A] { lhs => // To be implemented def execute(r: R)(implicit ec: ExecutionContext): Future[A] def flatMap[S <: R, B](f: A => Task[S, B]): Task[S, B] = ??? def map[B](f: A => B): Task[R, B] = ??? def run[S <: R]()(implicit runner: TaskRunner[S]): Future[A] = ??? } trait TaskRunner[R] { // To be implemented def run[A](task: Task[R, A]): Future[A] }
  48. 48. 48 How to use Fujitask We'll explain how to use the Fujitask through the implementation of ScalikeJDBC ScalikeJDBC版のFujitaskの実装を通じて Fujitaskの使い方を説明します
  49. 49. 49 Define types of transaction objects Define kinds of transaction These are assigned to a left type variable of Task[R,A] The important point is that ReadWriteTransaction inherits ReadTransaction トランザクションオブジェクトを定義する ReadWriteがReadを継承するのがポイント trait Transaction trait ReadTransaction extends Transaction trait ReadWriteTransaction extends ReadTransaction
  50. 50. 50 Implement transaction objects of ScalikeJDBC These classes inherit Transactionand wrap DBSession of ScalikeJDBC ScalikeJDBCのトランザクションオブジェク トの型を定義する abstract class ScalikeJDBCTransaction(val session: DBSession) class ScalikeJDBCReadTransaction(session: DBSession) extends ScalikeJDBCTransaction(session) with ReadTransaction class ScalikeJDBCReadWriteTransaction(session: DBSession) extends ScalikeJDBCTransaction(session) with ReadWriteTransaction
  51. 51. 51 Implement TaskRunner of ScalikeJDBC (1) Define askmethod to get the transaction object Derived from the ask function of Reader monad ScalikeJDBC版のTaskRunnerを実装する askメソッドを定義する package object scalikejdbc { def ask: Task[Transaction, DBSession] = new Task[Transaction, DBSession] { def execute(transaction: Transaction) (implicit ec: ExecutionContext): Future[DBSession] = Future.successful(transaction.asInstanceOf[ScalikeJDBCTransaction } }
  52. 52. 52 Implement TaskRunner of ScalikeJDBC (2) Define TaskRunnerfor ReadTransaction ReadTransaction用のTaskRunnerを定義する package object scalikejdbc { implicit def readRunner[R >: ReadTransaction] : TaskRunner[R] = new TaskRunner[R] { def run[A](task: Task[R, A]): Future[A] = { val session = DB.readOnlySession() val future = task.execute(new ScalikeJDBCReadTransaction(session)) future.onComplete(_ => session.close()) future } } }
  53. 53. 53 Implement TaskRunner of ScalikeJDBC (3) Define TaskRunnerfor ReadWriteTransaction ReadWriteTransaction用のTaskRunnerを定義 する package object scalikejdbc { implicit def readWriteRunner[R >: ReadWriteTransaction]: TaskRunner[ new TaskRunner[R] { def run[A](task: Task[R, A]): Future[A] = { DB.futureLocalTx(session => task.execute(new ScalikeJDBCReadWriteTransaction(session))) } } }
  54. 54. 54 Now, we can use Fujitask of ScalikeJDBC TaskRunnercan be implemented for another DB library and Redis client in the same way Next, we create domain code using this Fujitask of ScalikeJDBC これでScalikeJDBCのFujitaskを使うことがで きるようになりました
  55. 55. 55 Define an interface of Repository Repositoryのインターフェースを定義する trait UserRepository { def create(name: String): Task[ReadWriteTransaction, User] def read(id: Long): Task[ReadTransaction, Option[User]] }
  56. 56. 56 Implement Repository using ScalikeJDBC Fujitask ScalikeJDBC版Fujitaskを使ったRepositoryの 実装 object UserRepositoryImpl extends UserRepository { def create(name: String): Task[ReadWriteTransaction, User] = { implicit session => val sql = sql"""insert into users (name) values ($name)""" val id = sql.updateAndReturnGeneratedKey.apply() User(id, name) } def read(id: Long): Task[ReadTransaction, Option[User]] = { implicit session => val sql = sql"""select * from users where id = $id""" => User(rs.long("id"), rs.string("name"))).single.apply() } }
  57. 57. 57 Implement Service using Repository Fujitaskで作ったRepositoryを使ってみる trait MessageService extends UsesUserRepository with UsesMessageRepository { def createByUserId(message: String, userId: Long): Future[Message] = { val task = for { userOpt <- user = userOpt.getOrElse( throw new IllegalArgumentException("User Not Found")) message <- messageRepository.create(message, } yield message } }
  58. 58. 58 How Fujitask works Transactions are combined by for-expression userRepository.readhas ReadTransactionand messageRepository.createhas ReadWritetransaction, then the combined monad has ReadWritetransactionby subtyping And, when task.runis invoked, an appropriate TaskRunneris selected by implicit parameter mechanism Fujitaskはどのように動作するか
  59. 59. 59 The advantage of Fujitask Storage access can be abstracted by Fujitask We could compose transactions by Fujitask Master/Slave structure Programming is made easier Fujitaskの利点
  60. 60. 60 Sharding by Fujitask Use ShardedTaskhaving a shard key instead of the Task if we want sharding Type of shard key is checked statically by ShardedTask Also, the value of the shard key is checked at run time whether the same in flatMap Fujitaskによるシャーディング abstract class ShardedTask[Key, -R, +A](val shardKey: Key)
  61. 61. 61 Conclusion 総括
  62. 62. 62 Let's consider the commonalities among these techniques 最後に今回説明した技術の共通点を考えたい
  63. 63. 63 Simple Continuation monad can be represented by 6 lines Minimal Cake Pattern is the minimized Cake Pattern for DI Even though Fujitask has a variety of storage access functions, it can be written in about 30 lines 今回我々が紹介した技術はどれもシンプルで ある
  64. 64. 64 Use new features introduced in Scala Continuation monad can be briefly composed by Scala monads syntax Also, It can be combined with other monads in monad transformers by higher kinded types Minimal Cake Pattern uses a mixin of traits Fujitask uses variances and ad hoc polymorphism by implicit parameter どれもScalaで新しく導入された機能を使っ ている
  65. 65. 65 Without frameworks Web components are implemented by web framework in other languages We have proposed a method to use continuation monad Dependency Injection has been implemented by DI container in Java We can use cake pattern instead of DI container We would have to use AOP If we try to implement the functions of the Fujitask in other languages 我々の手法ではフレームワークは必要ではな い
  66. 66. 66 The Dwango account system is supported by the advanced features of Scala Thanks to the advanced features of Scala, our system became more simple, more understandable and more convenient ドワンゴアカウントシステムはScalaの高度 な言語機能に支えられている
  67. 67. 67 So, we want to conclude this session that ... よって、このセッションをこう締め括りたい
  68. 68. 68 We will continue to use the advanced languages like Scala 我々はScalaのような進歩した言語を使って いきたい
  69. 69. 69 And, we hope that programming languages continue to make progress そして、これからもプログラミング言語は進 歩しつづけてほしい
  70. 70. 70 Thank you for listening ご清聴ありがとうございました