Airframe HTTP enables using Scala as IDL for building web applications The live coding demo project can be found from here https://github.com/xerial/airframe-http-demo

1. Taro L. Saito, Ph.D.
Arm Treasure Data
July 9th, 2019
Arm Treasure Data Tokyo Office
Airframe HTTP
Airframe Meetup #2
1

2. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Today’s Agenda
● Airframe Updates 2019
● New modules, new features
● Airframe 19 Roadmap
● Airframe HTTP
● Scala as IDL
● Airframe HTTP internals
● Live Coding Demo
● Discussion & Hackathon
2
Airframe

3. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
About Me: Taro L. Saito (Leo)
3
● Principal Software Engineer at Arm
Treasure Data
● Building distributed query engine service
● Living in US for 4 years
● DBMS & Data Science Background
● Ph.D. of Computer Science
● OSS Projects Around Scala
● sbt-sonatype: used for releasing 3000+
Scala projects
● snappy-java: a compression library used
in Spark, Parquet, etc.

4. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
New Release from O’Reilly Japan
● Techniques and concepts around distributed
data processing systems
● Available at Amazon.co.jp and O’Reilly Japan
web sites
● Will be published on July 18, 2019
4
分散データシステム入門の決定版の翻訳が来月発売

5. 400+
Customers
Founded in
2011
Raised
$54M
Security
Acquired by Arm / Softbank
2018
Arm Treasure Data
Arm Treasure Dataの概要

6. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
The Architecture of Arm Treasure Data
6
DataLogs
Device
Data
Batch
Data
PlazmaDB
Table Schema
Data Collection Cloud Storage Distributed Data Processing
2 million records / sec. 130 trillion records 1 billion rows processed / sec.
Jobs
Job Management
SQL Editor
Scheduler
Workflows
Machine
Learning
Treasure Data OSS
Third Party OSS

7. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Module Mix-InPackaging
HTTP Requests and
Responses
Data
airframe-launcher
> _
airframe-log
production:
port: 10010
user: xxxx
...
airframe-config
airframe-codec
sbt-pack
airframe-fluentd
Scala
Objects
Table Data
(CSV, TSV)
JSON
airframe-jsonairframe-surface
airframe-tablet
airframe-jmx
Monitor Runtime States
Generate Mapping Codec
Metrics &
Log Data
JDBC
ResultSets
airframe-jdbc
airframe-http
airframe-http-finagle
Launch HTTP
Services
airframe DI
Debug Logs
Schema-On-Read
Mapping
Airframe

8. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
5 Years Ago (2014)
● No Scala engineer in the company
● Scala in 2014: Scala 2.9.x
8
Knowledge
Experiences
Design Decisions
Products
24/7 Services
Business Values
Programming Ruby, Java Outcome

9. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Small Investments to Scala
● For saving time of code development
● If you can save 1 minute in a day, your can spend 6 hours for this improvement
■ Save 1 minute / day = 365 minutes / year = 6 hour investment
■ Save 10 minutes / week = 520 minutes / year = 8.6 hour investment
■ Save 1 hour / week = 52 hours / year = 2.2 day investment
● Time is your most valuable asset
9

10. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Airframe
● Lightweight Building Blocks for Scala
● Collection of our investments to Scala
● Repackaged into wvlet.airframe in 2016
● airframe-log
● airframe-launcher
● airframe-config
● airframe-surface
● airframe-di
● airframe-codec
● ...
● As of 2019, Airframe has 20 modules
● 35+ releases in 2018
● Already had 17+ releases in 2019
● Contributing to the Scala Community Build
● To test the latest Scala versions
102016年に各種ツールをAirframeとして統合。20のモジュール、頻繁なリリースサイクル
Airframe

11. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Our OSS Strategy Around Scala
● Gather the best practices of Scala into Airframe OSS
● Get the real experiences by operating 24/7 services
11
Knowledge
Experiences
Design Decisions
Products
24/7 Services
Business Values
Programming OSS Outcome
Airframe

12. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
airframe-http
● Created a simple HTTP framework
● Based on Airframe modules:
■ airframe-surface
■ airframe-codec
■ airframe-msgpack
■ etc.
● Blog
● Building Low-Friction Web Service
Over Finagle
● Save the time for choosing a web
framework:
● Many many web frameworks….
● e.g, Finatra, Finch, akka-http, spring,
RESTeasy, open-api, swagger, etc.
12Airframe資産を活用して、Webフレームワークも手軽に作成

13. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Airframe Updates 2019
● The latest version 19.7.2 (2019-07)
● YY.MM.patch versioning scheme
● was version 0.69 on October 2018 (Airframe Meetup #1)
● 30+ releases
● What’s New Since 0.69
● New Airframe logo
● Scala 2.13 support
● New features (explained later)
● New modules
■ airframe-fluentd
■ airframe-launcher
■ airframe-http-client
■ airframe-http-recorder
■ airframe-canvas
13
Airframe

14. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
airframe-di: Scala-Friendly DI
● Dependency injection (DI) library for
Scala
● Usage
● import wvlet.airframe._
● Simple 3 Step DI
● bind
● design
● build
14

15. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
airframe-di use cases
● Configure a lot of modules
● Switch production/test modules
● Manage FILO lifecycle of services
● See also:
● Airframe Meetup #1 Report (2018)
15

16. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
airframe-di: Constructor vs In-Trait Injections
● Constructor Injection
● In-Trait Injection
16

17. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
New: More Detailed DI Failure Messages
17

18. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
New: DI Tracing
● chrome://tracing
● Tracer interface
18

19. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
New: DI Coverage Stats
19

20. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
New: airframe-canvas
● Off-heap memory buffer
● Save the CPU cost of zero-filing when initializing Java/Scala arrays.
● Managing memory outside JVM heap
■ Allocate a larger memory region than -Xmx (heap size) option.
● Quickly release allocated memory with Canvas.release.
● Even if you forget to release canvases, the allocated memory can be released
upon GC.
● A successor of LArray
● https://github.com/xerial/larray
● Development Plan
● InfiniteCanvas
■ Automatically read/write data using disks
■ Cursor rollback support for error handling at airframe-codec
20

21. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Update: airframe-log
● Added msec to log time stamps
● A Modern Logging Library for Scala (at Medium)
● ANSI color and source code location display
● Just add LogSupport trait to your class
21プログラムの開発をログメッセージで効率化する

22. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
airframe-launcher (renamed from airframe-opts)
● Handle complex command line options and nested commands
● Enabled annotation-based command line definitions
22複雑なコマンドラインプログラムを簡単に作成できるようにする

23. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
airframe-config: Application Configuration Flow
● YAML config (embedded into Docker)
● Override credentials, then bind to config objects
YAML
development:
addr: api-dev.com
production:
addr: api.com
Config Object
case class ServerConfig(
addr: String,
port: Int = 8080,
password: String
)
production:
addr: api.com
command: -e production Credentials and Local
Configurations
Merge
Immutable
Object Default Parameters
(e.g., port = 8080)
Object
Mapping
23アプリケーション設定のフローをライブラリ化
airframe-launcher
> _

24. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
New: airframe-config + DI Integration
24

25. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
sbt-pack plugin
● A sbt plugin to create standalone Scala packages
● A single folder package with bin and lib folders containing all dependent JARs
● Generates command-line launcher scripts
● My small investment in 2012 to save packaging time
25sbt-packでプログラムをパッケージングし、Dockerイメージを手軽に作成
airframe-launcher
airframe-config
YAML config file
Standalone
Scala Package
sbt-pack Dockerfile

26. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Object-Data Mapping Pattern
● Data -> Object Mapping
● How many data readers and object mappers do we need?
● How can we save our time for handling such various data types?
YAML
JDBC
ResultSet
YAML Parser +
Object Mapper
Config
Object
Table
Object
Object-Relation
Mapper
JSON
JSON Parser +
Object Mapper
Object
26入力データをScala Objectにマッピングしたいケースは多い

27. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
airframe-msgpack: MessagePack as Universal Data Format
● MessagePack (msgpack.org)
● Compact JSON-like binary format
● Describes data types and data values at the same time (self-describing)
Object
Unpack
Pack
JDBC
ResultSet
Pack/Unpack
YAML
JSON
27MessagePackを中間フォーマットとして使うと、オブジェクトマッパーの実装は１つに
MessagePack

28. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
● Many JSON libraries
● json4s, jackson, jawn, circe, uPickle, etc.
● Implemented in 2 days by using Jawn parser as a reference implementation
● JSON to MessagePack conversion
● Comparable performance with Jawn and Circe
● Saves time to work with other projects
● e.g., Adding a fast pass tailored to MessagePack
airframe-json
28
JDBC
ResultSet
Object
Unpack
Pack
YAML
JSON
MessagePack

29. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
airframe-codec: Schema-On-Read Pack/Unpack Interface
● Apply schema-on-read for Scala objects
Input MessagePack Output
Pack Unpack
PackUnpack
29MessagePackを通したSchema-on-readデータ変換インターフェースをScalaに適用

30. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Pre-defined Codecs in airframe-codec
● Primitive Codecs
● ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec
● FloatCodec, DoubleCodec
● StringCodec
● BooleanCodec
● TimeStampCodec
● Collection Codec
● ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec, etc.
● OptionCodec
● JsonCodec (airframe-json)
● Java-specific Codec
● FileCodec, ZonedDateTimeCodec, JDBCResultSetCodec, etc.
● Adding Custom Codecs
● Implement MessageCodec[X] interface
30Scalaで必要なほぼ全てのデータ型へのマッピングをサポート

31. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
PlazmaDB: MessagePack DBMS
● Fluentd -> MessagePack -> Arm Treasure Data
● Automatically generating table schema from MessagePack data
● Apply schema–on-read for providing table data for Presto/Hive/Spark, etc.
Table Schema
Int Column Reader
String Column Reader
Update
Schema
Generate
Reader Set
Table Reader
Schema-free Data
31
Data Collection Distributed Data Processing
Arm Treasure DataはMessagePackベースのSchema-on-readシステム

32. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Schema-On-Read Data Processing with MessagePack
● Users can store arbitrary typed data (No table design is required)
● Data can be read in a target type required by the application (e.g., SQL query)
Int
Float
Boolean
String
Array
Map
Binary
SQL BigInt
parseInt
toInt
0 or 1
IntCodec
Pack Unpack
Error or null
“100”
(string)
100
(int)
100
(int)
32
Logs
データ読み込み時に、アプリケーションの要求する型に合わせる (Schema-on-Read)
CSV
command-line
arguments

33. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
airframe-surface: Object Shape Inspector
● Surface.of[X]
33

34. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
airframe-surface
● Reading Type Signatures From ScalaSig
● Scala compiler embeds Scala Type Signatures (ScalaSig) to class files
● Surface.of[A]
■ returns A’s parameter names and types
class A (data:List[B])
class A
data: List[java.lang.Object]
class A
data: List[java.lang.Object]
ScalaSig: data:List[B]
javac
scalac
Surface.of[A]
data: List[B]
scala.reflect.runtime.universe.TypeTag
Type erasure removes
generic type information
34オブジェクトの型情報をScalaSigから取得する

35. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
MessageCodec.of[A]: Combination of Codecs
Unpack
Pack
IntCodec
StringCodec
DoubleCodec
MessagePack
MessageCodec.of[A]
35オブジェクトの型に合わせてCodecを合成
Surface.of[A]

36. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
New: airframe-fluentd
● Collect Scala Application Logs To Fluentd
● Scala Objects -> MessagePack -> Fluentd
36MessagePackを受け取るFluentdには、airframe-codeの出力を渡せる
Collect Query Logs
Analyze Query Logs
Machine
Learning
Query
Optimization
Optimize System
airframe-fluentd
Scala
Objects
airframe-codec

37. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
New: airframe-control: Retry
● Everything can fail …
● Network disconnection
● Servere crash
● ...
● Retry
● Exponential backoff
■ 2x, 4x, ...
● Jittering
■ 1 sec., 2 * rand, 4 * rand, …
● Customize error type classifiers
● retryable failures
● non-retryable failures
37リトライ処理をパターン化

38. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
New: airframe-control: Parallel
● Running tasks in parallel
● Customize parallelism
● Fixes slow tail issue of parallel collection
38

39. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Airframe 19 Milestones
● Airframe 19 Milestones
● https://github.com/wvlet/airframe/issues/528
● airframe-http
● Under active development for production use
■ Request filters
■ Request logging
■ JMX metrics
■ OpenAPI spec generator (no code
generation is planned)
39

40. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
airframe-http
● Created a simple HTTP framework
● Based on Airframe modules:
■ airframe-surface
■ airframe-codec
■ airframe-msgpack
■ etc.
● Blog
● Building Low-Friction Web Service
Over Finagle
● Save the time for choosing a web
framework:
● Many frameworks exist:
● e.g, Finatra, Finch, akka-http, spring,
RESTeasy, open-api, swagger, etc.
40Airframe資産を活用して、Webフレームワークも手軽に作成

41. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
airframe-http: Scala as IDL
● IDL: Interface Definition Language
● e.g. ProtocolBuffers, gRPC
● Use Scala’s Standard Functionalities
● case classes
■ Define HTTP request/response
data structure
● functions
■ Define HTTP endpoints
■ Use @Endpoint annotation for
for URL to method mapping
● Router
● Router.add[X]
● filter andThen router (available in
the next release)
41

42. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Path Mapping
● :param
● Mapping to a function argument
● /user/:id
■ /user/1
■ def getUser(id:Int)
○ getUser(1)
● *param
● Mapping url path tail to a function
argument
● /resource/*path
■ /resource/my/key/path
■ def getPath(path:String)
○ getPath(my/key/path)
● case classes
● request/response body
● JSON <-> MessagePack <-> Object
42

43. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
airframe-http-client
● Read server responses as target object types
● Accept: application/json (default)
● Accept: application/x-msgpack (Receive data in MessagePack format)
43

44. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Retry Patterns for HTTP requests
● Error handling of HTTP requests is
difficult
● 4xx, 5xx status code
● Should we retry the request?
■ IOException, EOFException
■ TimeoutException
■ InterruptedException
■ SSLException
■ InvocationTargetException
● HTTP client
● request retries
● response mapping
■ JSON, MessagePack format
● airframe-codec
44間違いやすいHTTPリクエストのエラーハンドリングをライブラリ化

45. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
RouteMatcher
● Router
● List of @Endpoint definitions
● Graph
● Node
■ Path fragment
● Action
■ Put variable mappings for :param or *param
● Edge
■ Node -> Token (path fragment) -> Node
● Input:
● URL request path fragments
● Algorithm
● Build NFA (Non-deterministic Finite Automaton)
● Convert NFA into DFA (deterministic finite automaton)
● Traverse the automaton
● Call the matched function (MethodSurface)
■ buildControllerMethodArgs (Route.scala)
45

46. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Http Request/Response Adapters
● Finagle Adapters
● HttpRequest[finagle.Request]
● Upcoming Adapters
● jetty, okhttp
46

47. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
airframe-http-recorder
● Testing against actual web services is time consuming
● Record & Replay HTTP responses
● Reproducible results
● Runnable on small machines (e.g., Travis CI)
47HTTPリクエストをレコーディングして、Webサービスのテストを効率化する
HTTP
Request
HTTP
Recorder
Request
Real Web
Service
Recording Mode:
Response
HTTP
Request
HTTP
Recorder
Replay Mode:
Request
Response Recording
Responses
Request
Recorded
Responses

48. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
[WIP] Scala.js RPC
● Scala.js
● Compiling Scala code into JavaScript for Web Browsers
● airframe-codec: Passing model class data between Scala and Scala.js
UserInfo MessagePack UserInfo
Pack Unpack
PackUnpack
Scala
Server Side
Scala.js
Client Side
XML RPC
48airframe-codecはScala.js（ブラウザ側)とのデータ受け渡しにも使える

49. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved.
Demo: Live Coding
● Build a simple web server
● JSON messaging support
● build.sbt
● project/plugins.sbt
■ sbt-pack, sbt-revolver
● Command line program launcher
● airframe-launcher
● packaging
● sbt-pack
● Docker
● Add web APIs
49

50. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Scala At Arm Treasure Data
● Scala is now an official language at Arm Treasure Data
● 0 -> 10+ engineers who can write Scala
● Use cases are growing:
● Query optimization, API, Spark, data analysis,
storage systems, service operation, etc.
● We are happy to share our Scala assets through Airframe!
50
Airframe
Scalaエンジニアが充実してきたArm Treasure Data。Scalaの適用範囲も広がっている

51. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Presto Conference Tokyo 2019
● July 11 (Thu), 2019, 13:30 ~ (Free)
● https://techplay.jp/event/733772
● Inviting Presto Creators (Martin, Dain, David)
● Presto Software Foundation
● Talks from big Presto users in Japan
● Yahoo! JAPAN, LINE, Arm Treasure Data
● Presto Source Code Navigation
51
Presto Conference Tokyo 2019を7/11(木) 13:30~より開催 (参加無料)

52. Confidential © Arm 2017Confidential © Arm 2017Confidential © Arm 2017
Thank You!
Danke!
Merci!
谢谢!
ありがとう!
Gracias!
Kiitos!
52