Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020

1. 1 Taro L. Saito Treasure Data October 17th, 2020 Scala Matsuri 2020 Scala For Everything From Frontend to Backend Applications Scala無双: フロントエンドからバックエンドも全てScalaでなぎ倒せ 

2. Copyright 1995-2020 Treasure Data. All rights reserved. About Me: Taro L. Saito 2 ● Treasure Data ○ Principal Software Engineer ○ Living in US for 5 years ● Ph.D. of Computer Science ○ Database systems, genome science. ● OSS: ○ Airframe, snappy-java (used in Parquet, Spark, etc.), sbt-sonatype, sbt-pack, etc. ● Book: 自己紹介 

3. Copyright 1995-2020 Treasure Data. All rights reserved. Today’s Topic ● There are a lot of libraries and frameworks for backend and frontend applications ● Airframe ○ An OSS library collection that enables Scala-oriented development ● Goal: Forget everything other than Scala 3 Scala無双：Scala以外を全て駆逐する  Airframe Scala.js Web Browsers JSON

4. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe: Core Scala Modules of Treasure Data ● Airframe (GitHub: wvlet/airframe) ○ 20+ Common Utilities for Scala and Scala.js (Scala Matsuri 2019) ■ Logging, Dependency Injection (DI), JSON/MessagePack parser, etc. ○ Airframe RPC, Airframe Rx (Scala Matsuri 2020) ○ AirSpec ■ Testing framework for Scala and Scala.js (ScalaDays. Seattle, May 2021) 4 Knowledge Experiences Design Decisions Products 24/7 Services Business Values Programming OSS Outcome Airframe AirframeはScala.jsでも利用することを前提に開発されてきたOSSツール群 

5. Copyright 1995-2020 Treasure Data. All rights reserved. Treasure Data: A Ready-to-Use Cloud Data Platform 5 Logs Device Data Batch Data PlazmaDB Table Schema Data Collection Cloud Storage Distributed Data Processing Jobs Job Management SQL Editor Scheduler Workflows Machine Learning Treasure Data OSS Third Party OSS Data トレジャーデータはすぐに使えるクラウドデータプラットフォーム 

7. Copyright 1995-2020 Treasure Data. All rights reserved. Server APIs and Library for Spark Airframe DI DataFrame MPC1 airframe-codec airframe-msgpack Plazma Public API airframe-http airframe-finagle Airframe DI Airframe RPC airframe-fluentd Master Worker DesignSparkContext TDSparkContext TDSparkService MPC1 Reader/Writer IO Manager Airframe DI airframe-http airframe-config airframe-launcher airframe-jmx airframe-metrics airframe-control airframe-metrics td-spark.jarairframe-log airframe-log airframe-codec airframe-json Airframe 7 Sparkをサポートするために、ScalaとAirframeで作られたサービスが使われている  ● MPC1: MessagePack Columnar Format ● See also td-spark internals (Spark Meetup Tokyo 2020)

8. Copyright 1995-2020 Treasure Data. All rights reserved. Scala is Versatile ● Backend ○ Can leverage JVM ecosystem (e.g., netty, gRPC, etc.) ● Frontend ○ Scala.js: ■ 1.0.0 is released in 2020 ● The current version is 1.3.0 ■ Compile Scala code into Javascripts ○ Out of scope of this talk ■ iOS/iPhone native applications ● How can we bridge server (Scala JVM) and client (Scala.js) applications? Scalaはバックエンド、フロントエンド開発に使える言語に。では両者をどう繋ぐ？  Scala.js Web Browsers Serve-Client Interaction 8

9. Copyright 1995-2020 Treasure Data. All rights reserved. Bridging Scala and Scala.js with RPC Program Function Call Return Local Function Call (ideal) Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize ローカル関数呼び出しと違い、実際にはネットワーク経由でのRPCが必要になる  Remote Procedure Call (reality) Network Client Call Return Server Call Return 9 Scala.js Scala.js

10. Copyright 1995-2020 Treasure Data. All rights reserved. Existing Approaches for Implementing RPC ● REST ○ Define function interfaces with HTTP endpoints (e.g., GET/POST/PUT/DELETE, etc.) ○ REST web frameworks for Scala ■ Play framework, akka-http, Finatra, Finch, skinny-framework, etc. ● gRPC ○ Define function interfaces with Google’s ProtocolBuffers (.proto) schema language ○ Generate server and client code stubs from proto files ○ Scala wrappers: ■ ScalaPB, muScala, akka-grpc, etc. RPCを実装するには、RESTやgRPCなどが使われている  Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Return Server Call Return 10

11. Copyright 1995-2020 Treasure Data. All rights reserved. API Design War https://cloud.google.com/blog/products/api-management/ understanding-grpc-openapi-and-rest-and-when-to-use-th em 11 ● Which API design should we choose? ● REST API ○ Designing good APIs over HTTP requires experiences and signiﬁcant amount of time ● gRPC ○ Requires special software like code generators for using the ecosystem around ProtocolBuﬀers ● And also, there are too many choices for web frameworks しかし、APIデザイン、フレームワーク選びでの争いは付きない 

12. Copyright 1995-2020 Treasure Data. All rights reserved. Why Not Using Scala for RPC Interface? ● Scala is a functional and statically-typed object-oriented programming language ○ Scala functions = RPC methods ○ Scala objects (statically typed) = RPC request/response types ● Scala is a perfect ﬁt for RPC! 12 関数型で静的型付きオブジェクト指向言語のScalaはRPCインターフェースに適している 

13. Copyright 1995-2020 Treasure Data. All rights reserved. Scala-First Approach: Airframe RPC RPC InterfaceScala.js Client RPC Web Server Generates Scala.js Web Application API Documentation 13 Airframe RPCはScalaをサーバー・クライアント共通インターフェースとして使う新手法  RPC Call RPC Call Implements ● Forget about REST or ProtocolBuﬀers! ● Use Scala as a uniﬁed interface between server and clients Airframe

14. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe RPC Backend Is Pluggable RPC Interface RPC Web Server Generates API Documentation 14 Airframe RPCのServer, Clientは差し替え可能  Airframe RPC Call RPC Call Implements ● Backend server ○ Finagle (HTTP/1), gRPC (HTTP/2) ● Backend client ○ Finagle, gRPC, okhttp, Java URLConnection client etc. RPC Clients

15. Copyright 1995-2020 Treasure Data. All rights reserved. Building Blocks of RPC Frameworks ● 1. Message serializer and deserializer ● 2. Network data format ○ JSON (REST), Protobuf (gRPC), or MessagePack (Airframe RPC) ● 3. RPC interface language ○ REST API, Protobuf, or Scala ● 4. HTTP client and server implementation ○ Code generator 15 Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Return Server Call Return RPCフレームワークを作るのに必要な構成要素 

16. Copyright 1995-2020 Treasure Data. All rights reserved. MessagePack-Based Object Serialization ● MessagePack ○ A compact binary format compatible with JSON Object Object Pack Unpack PackUnpack Server SideClient Side 16 Scala.js JSON Airframe MessagePackを使ってクライアントとサーバー間通信を行う 

17. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-codec: Pack/Unpack Interface ● Pack ○ Converting Data to MessagePack ● Unpack ○ Converting MessagePack to Data Input Output Pack Unpack PackUnpack 17 Airframe MessageCodec MessageCodec MessagePackを通したデータ変換インターフェースをScalaに適用 

18. Copyright 1995-2020 Treasure Data. All rights reserved. Pre-deﬁned Codecs in airframe-codec ● Primitive Codecs ○ ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec ○ FloatCodec, DoubleCodec ○ StringCodec ○ BooleanCodec ○ TimeStampCodec ● Collection Codec ○ ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec, etc. ○ OptionCodec ● Java-speciﬁc Codec ○ UUIDCodec, FileCodec, ZonedDateTimeCodec, InstantCodec ○ JDBCResultSetCodec ● etc. 18 Scalaで必要なほぼ全てのデータ型へのマッピングをサポート  Airframe

19. Copyright 1995-2020 Treasure Data. All rights reserved. Serializing Complex Objects Pack Unpack IntCodec StringCodec DoubleCodec MessageCodec.of[A] 19 オブジェクトの型に合わせてCodecを合成できる  Serialize Deserialize JSON

20. Copyright 1995-2020 Treasure Data. All rights reserved. Deﬁning RPC Interface with Scala ● Add ＠RPC annotation to a trait (= RPC interface) ● Deﬁne model classes (= data structures) using case classes ● Use the same interface and model classes between servers and clients RPC InterfaceをScalaで記述する  20

21. Copyright 1995-2020 Treasure Data. All rights reserved. Serializing Function Call Request ● Function call arguments ○ A sequence of argument name -> value, …. ● Example: ○ Map(“person” -> Person(1, “leo”), “message” -> “Hello RPC!”) ○ Serialize this Map into MessagePack with airframe-codec ● HTTP Mapping ○ HTTP method: POST ○ Path: /(package name)/hello ■ e.g., /hello.api.v1/hello ○ Content body: serialized function call argument data 21 関数呼び出しの引数もシリアライズする 

23. Copyright 1995-2020 Treasure Data. All rights reserved. ● Finagle: Twitter’s HTTP server written in Scala ○ HTTP/1 based. Accessible from web browsers running Scala.js code airframe-http: Starting A Finagle Backend RPC Server 23 Airframe Scala.js airframe-rx-html airframe-http: FinagleバックエンドのHTTPサーバーを起動する  airframe-http-finagle

24. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-http: gRPC Backend ● Suited to internal microservices ● HTTP/2 based protocol ○ Multiplexing multiple RPC requests within a single connection ● No .proto ﬁle is required 24 Airframe airframe-http: gRPCバックエンドサーバーを起動する。ProtoBuf定義は必要ない 

25. Copyright 1995-2020 Treasure Data. All rights reserved. Advanced Topic: Extending gRPC ● gRPC is data-format agonistic framework ○ To support Scala interface and MessagePack, we have two extension points in grpc-java. ● MethodDescriptor ○ Deﬁne gRPC endpoints corresponding to Scala functions ○ Register marshallers ● Request/ResponseMarshaller ○ Deﬁne how to encode/decode RPC request/response data 25 Airframe RPCはgrpc-javaを拡張してMessagePackデータの受け渡しをしている 

26. Copyright 1995-2020 Treasure Data. All rights reserved. RPC Performance Comparison (Greeter Service) ● Airframe RPC ○ serde: MessagePack (airframe-codec) <-> Scala case classes ○ Finagle (HTTP1) or gRPC (HTTP2) ● ScalaPB ○ serde: Protobuf <-> Scala case classes ○ gRPC (HTTP2) ● grpc-java ○ serde: Protobuf <-> Java classes ○ gRPC (HTTP2) ● Notes ○ Using Finagle with HTTP2 had almost no beneﬁt over HTTP1 ■ Multiplexing RPC requests over HTTP2 is the key performance factor ○ Overhead of ScalaPB ■ Scala Future (10% overhead) ■ Mapping Protobuf to Scala case classes (10% overhead) 26 gRPCはリクエスト多重送信により高速。MessagePackによるオーバーヘッドは少ない  HTTP/1 HTTP/2 (gRPC)

27. Copyright 1995-2020 Treasure Data. All rights reserved. sbt-airframe: Generating RPC Clients ● sbt-airframe plugin ○ Read RPC interface classes ○ Generate HTTP client code for accessing the RPC server ○ OpenAPI schema generation support sbt-airframe Code Generation RPC Client Scala.js 27 Scala.js Client HTTP/gRPC Client Open API Spec Cross-Language RPC Client RPCクライアントを生成するsbt-airframeプラグイン 

28. Copyright 1995-2020 Treasure Data. All rights reserved. sbt-airframe: Generating Open API Schema ● RPC Interface -> Open API schema YAML ﬁle ● Generating RPC Clients & Swagger Documentation 28 ScalaのインターフェースからOpenAPIのYAMLを生成する  RPC Interface API Documentation Open API Spec (YAML) Cross-Language RPC Client Generate sbt-airframe API Documentation

29. Copyright 1995-2020 Treasure Data. All rights reserved. Making An RPC Call RPCを実行する。サーバー、クライアント共にScalaで動いている  Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Response Server Call Response AirframeScala.js 29

30. Copyright 1995-2020 Treasure Data. All rights reserved. Scala.js Programming Tips ● Scala.js: Compiling Scala -> JavaScript ● Use Pure Scala Code ○ [NG] Java code ■ Dependent libraries also need to be pure Scala ● Airframe is carefully designed to minimize non-Scala dependencies ○ [NG] Annotation, runtime-reﬂection ○ [OK] Scala Macros (compile-time meta-programming) ● RPC ○ Asynchronous calls using Scala Future ○ Await(Future) is not supported ■ Because JS is a single-threaded model. ■ Need to properly chain Future operators with map, ﬂatMap, rescue, etc. 30 Scala.jsでのプログラミングのコツ  Scala.js

32. Copyright 1995-2020 Treasure Data. All rights reserved. Rendering DOM with Scala.js ● DOM element: ○ <div class=”container”> …. </div> ○ scalajs-dom library has functions necessary for generating DOMs ● Previous approach: Converting XML literal into DOM elements ○ monadic-html, Binding.scala ○ But XML literal will be deprecated in Dotty (Scala 3.0) 32 Scala.jsでのDOMレンダリング手法には様々な手法がある 

33. Copyright 1995-2020 Treasure Data. All rights reserved. Rendering DOM ● Current Best Practice ○ Using Scala functions for building DOM elements ● Option 1: Separation between DOM elements and attributes ○ ScalaTags, Slinky ○ Too many parentheses! ● Option 2: No separation between DOM elements and attributes ○ scalajs-react, Airframe Rx ○ More Scala-friendly syntax 33 DOMのelement, attributeを区別しない方が関数型に適している  ScalaTags Airframe Rx

34. Copyright 1995-2020 Treasure Data. All rights reserved. Complex DOM Rendering ● Airframe Rx ● RxElement interface ○ def render(): RxElement ○ Support nesting with apply(...) ■ Embedding Scala functions and collections 34 airframe-rx-html: Scalaの関数をネストしてDOMをレンダリングする 

35. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe Rx ● Rx.variable ○ Re-render DOM if the variable is updated ● Gathering best practices from existing libraries ○ Type-safe DOM elements and attributes ■ ScalaTags, scalajs-react ○ Reactive streaming ■ Monix ■ Scala Collection ■ Scala.rx ○ Cancelable Interface ■ Cleanup DOM and event handlers when updating DOM ■ Monix, monadic-html 35 既存技術の良いところ取りをしたAirframe Rx 

36. Copyright 1995-2020 Treasure Data. All rights reserved. Reactive Stream Interface: Rx[X] ● Rx[X] ○ ReactiveX stream implementation ○ e.g., Monix, Scala.Rx ● Reactive DOM Rendering ○ Rewriting DOM interactively upon upstream data change ● Reactive Operators ○ map, ﬂatMap, ﬁlter, join, etc. ○ Almost the same with Scala collection APIs 36 Reactive stream interface Rx[X]でインタラクティブなDOMを作成 

37. Copyright 1995-2020 Treasure Data. All rights reserved. gRPC Streaming with Airframe Rx ● Interactive communication between server and clients with reactive stream interface: Rx[A] 37 gRPC StreamingもRxインターフェースでサポートできる 

38. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe Enables Scala-Oriented Development 38 RPC InterfaceScala.js Client RPC Web Server Generates Scala.js Web Application API Documentation RPC Call RPC Call Implements Airframe AirframeでScala中心のアプリケーション開発を可能に  ● Only need to learn how to write Scala for kick-starting client/server application development

39. Copyright 1995-2020 Treasure Data. All rights reserved. Summary ● Scala is a powerful language that can consolidate existing technologies ● Minimizing the learning cost of application development with Scala-First approach ○ Airframe OSS (GitHub: wvlet/airframe) is designed for this purpose 39 まとめ：既存技術の長所を兼ね備えることで、Scala無双を可能にするAirframe  Airframe Scala.js Web Browsers JSON

Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020

You are reading a preview.

Check these out next

Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020

More Related Content

Slideshows for you (20)

Similar to Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020 (20)

More from Taro L. Saito (18)

Recently uploaded (20)