Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Airframe HTTP - Airframe Meetup #2 Tokyo 2019-07-09

358 views

Published on

Airframe HTTP enables using Scala as IDL for building web applications

The live coding demo project can be found from here https://github.com/xerial/airframe-http-demo

Published in: Technology
  • Be the first to comment

Airframe HTTP - Airframe Meetup #2 Tokyo 2019-07-09

  1. 1. Taro L. Saito, Ph.D. Arm Treasure Data July 9th, 2019 Arm Treasure Data Tokyo Office Airframe HTTP Airframe Meetup #2 1
  2. 2. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Today’s Agenda ● Airframe Updates 2019 ● New modules, new features ● Airframe 19 Roadmap ● Airframe HTTP ● Scala as IDL ● Airframe HTTP internals ● Live Coding Demo ● Discussion & Hackathon 2 Airframe
  3. 3. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. About Me: Taro L. Saito (Leo) 3 ● Principal Software Engineer at Arm Treasure Data ● Building distributed query engine service ● Living in US for 4 years ● DBMS & Data Science Background ● Ph.D. of Computer Science ● OSS Projects Around Scala ● sbt-sonatype: used for releasing 3000+ Scala projects ● snappy-java: a compression library used in Spark, Parquet, etc.
  4. 4. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. New Release from O’Reilly Japan ● Techniques and concepts around distributed data processing systems ● Available at Amazon.co.jp and O’Reilly Japan web sites ● Will be published on July 18, 2019 4 分散データシステム入門の決定版の翻訳が来月発売

  5. 5. 400+ Customers Founded in 2011 Raised $54M Security Acquired by Arm / Softbank 2018 Arm Treasure Data Arm Treasure Dataの概要

  6. 6. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. The Architecture of Arm Treasure Data 6 DataLogs Device Data Batch Data PlazmaDB Table Schema Data Collection Cloud Storage Distributed Data Processing 2 million records / sec. 130 trillion records 1 billion rows processed / sec. Jobs Job Management SQL Editor Scheduler Workflows Machine Learning Treasure Data OSS Third Party OSS
  7. 7. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Module Mix-InPackaging HTTP Requests and Responses Data airframe-launcher > _ airframe-log production: port: 10010 user: xxxx ... airframe-config airframe-codec sbt-pack airframe-fluentd Scala Objects Table Data (CSV, TSV) JSON airframe-jsonairframe-surface airframe-tablet airframe-jmx Monitor Runtime States Generate Mapping Codec Metrics & Log Data JDBC ResultSets airframe-jdbc airframe-http airframe-http-finagle Launch HTTP Services airframe DI Debug Logs Schema-On-Read Mapping Airframe
  8. 8. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. 5 Years Ago (2014) ● No Scala engineer in the company ● Scala in 2014: Scala 2.9.x 8 Knowledge Experiences Design Decisions Products 24/7 Services Business Values Programming Ruby, Java Outcome
  9. 9. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Small Investments to Scala ● For saving time of code development ● If you can save 1 minute in a day, your can spend 6 hours for this improvement ■ Save 1 minute / day = 365 minutes / year = 6 hour investment ■ Save 10 minutes / week = 520 minutes / year = 8.6 hour investment ■ Save 1 hour / week = 52 hours / year = 2.2 day investment ● Time is your most valuable asset 9
  10. 10. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Airframe ● Lightweight Building Blocks for Scala ● Collection of our investments to Scala ● Repackaged into wvlet.airframe in 2016 ● airframe-log ● airframe-launcher ● airframe-config ● airframe-surface ● airframe-di ● airframe-codec ● ... ● As of 2019, Airframe has 20 modules ● 35+ releases in 2018 ● Already had 17+ releases in 2019 ● Contributing to the Scala Community Build ● To test the latest Scala versions 102016年に各種ツールをAirframeとして統合。20のモジュール、頻繁なリリースサイクル
 Airframe
  11. 11. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Our OSS Strategy Around Scala ● Gather the best practices of Scala into Airframe OSS ● Get the real experiences by operating 24/7 services 11 Knowledge Experiences Design Decisions Products 24/7 Services Business Values Programming OSS Outcome Airframe
  12. 12. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. airframe-http ● Created a simple HTTP framework ● Based on Airframe modules: ■ airframe-surface ■ airframe-codec ■ airframe-msgpack ■ etc. ● Blog ● Building Low-Friction Web Service Over Finagle ● Save the time for choosing a web framework: ● Many many web frameworks…. ● e.g, Finatra, Finch, akka-http, spring, RESTeasy, open-api, swagger, etc. 12Airframe資産を活用して、Webフレームワークも手軽に作成

  13. 13. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Airframe Updates 2019 ● The latest version 19.7.2 (2019-07) ● YY.MM.patch versioning scheme ● was version 0.69 on October 2018 (Airframe Meetup #1) ● 30+ releases ● What’s New Since 0.69 ● New Airframe logo ● Scala 2.13 support ● New features (explained later) ● New modules ■ airframe-fluentd ■ airframe-launcher ■ airframe-http-client ■ airframe-http-recorder ■ airframe-canvas 13 Airframe
  14. 14. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. airframe-di: Scala-Friendly DI ● Dependency injection (DI) library for Scala ● Usage ● import wvlet.airframe._ ● Simple 3 Step DI ● bind ● design ● build 14
  15. 15. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. airframe-di use cases ● Configure a lot of modules ● Switch production/test modules ● Manage FILO lifecycle of services ● See also: ● Airframe Meetup #1 Report (2018) 15
  16. 16. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. airframe-di: Constructor vs In-Trait Injections ● Constructor Injection ● In-Trait Injection 16
  17. 17. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. New: More Detailed DI Failure Messages 17
  18. 18. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. New: DI Tracing ● chrome://tracing ● Tracer interface 18
  19. 19. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. New: DI Coverage Stats 19
  20. 20. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. New: airframe-canvas ● Off-heap memory buffer ● Save the CPU cost of zero-filing when initializing Java/Scala arrays. ● Managing memory outside JVM heap ■ Allocate a larger memory region than -Xmx (heap size) option. ● Quickly release allocated memory with Canvas.release. ● Even if you forget to release canvases, the allocated memory can be released upon GC. ● A successor of LArray ● https://github.com/xerial/larray ● Development Plan ● InfiniteCanvas ■ Automatically read/write data using disks ■ Cursor rollback support for error handling at airframe-codec 20
  21. 21. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Update: airframe-log ● Added msec to log time stamps ● A Modern Logging Library for Scala (at Medium) ● ANSI color and source code location display ● Just add LogSupport trait to your class 21プログラムの開発をログメッセージで効率化する

  22. 22. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. airframe-launcher (renamed from airframe-opts) ● Handle complex command line options and nested commands ● Enabled annotation-based command line definitions 22複雑なコマンドラインプログラムを簡単に作成できるようにする

  23. 23. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. airframe-config: Application Configuration Flow ● YAML config (embedded into Docker) ● Override credentials, then bind to config objects YAML development: addr: api-dev.com production: addr: api.com Config Object case class ServerConfig( addr: String, port: Int = 8080, password: String ) production: addr: api.com command: -e production Credentials and Local Configurations Merge Immutable Object Default Parameters (e.g., port = 8080) Object Mapping 23アプリケーション設定のフローをライブラリ化
 airframe-launcher > _
  24. 24. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. New: airframe-config + DI Integration 24
  25. 25. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. sbt-pack plugin ● A sbt plugin to create standalone Scala packages ● A single folder package with bin and lib folders containing all dependent JARs ● Generates command-line launcher scripts ● My small investment in 2012 to save packaging time 25sbt-packでプログラムをパッケージングし、Dockerイメージを手軽に作成
 airframe-launcher airframe-config YAML config file Standalone Scala Package sbt-pack Dockerfile
  26. 26. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Object-Data Mapping Pattern ● Data -> Object Mapping ● How many data readers and object mappers do we need? ● How can we save our time for handling such various data types? YAML JDBC ResultSet YAML Parser + Object Mapper Config Object Table Object Object-Relation Mapper JSON JSON Parser + Object Mapper Object 26入力データをScala Objectにマッピングしたいケースは多い

  27. 27. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. airframe-msgpack: MessagePack as Universal Data Format ● MessagePack (msgpack.org) ● Compact JSON-like binary format ● Describes data types and data values at the same time (self-describing) Object Unpack Pack JDBC ResultSet Pack/Unpack YAML JSON 27MessagePackを中間フォーマットとして使うと、オブジェクトマッパーの実装は1つに
 MessagePack
  28. 28. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. ● Many JSON libraries ● json4s, jackson, jawn, circe, uPickle, etc. ● Implemented in 2 days by using Jawn parser as a reference implementation ● JSON to MessagePack conversion ● Comparable performance with Jawn and Circe ● Saves time to work with other projects ● e.g., Adding a fast pass tailored to MessagePack airframe-json 28 JDBC ResultSet Object Unpack Pack YAML JSON MessagePack
  29. 29. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. airframe-codec: Schema-On-Read Pack/Unpack Interface ● Apply schema-on-read for Scala objects Input MessagePack Output Pack Unpack PackUnpack 29MessagePackを通したSchema-on-readデータ変換インターフェースをScalaに適用

  30. 30. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Pre-defined Codecs in airframe-codec ● Primitive Codecs ● ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec ● FloatCodec, DoubleCodec ● StringCodec ● BooleanCodec ● TimeStampCodec ● Collection Codec ● ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec, etc. ● OptionCodec ● JsonCodec (airframe-json) ● Java-specific Codec ● FileCodec, ZonedDateTimeCodec, JDBCResultSetCodec, etc. ● Adding Custom Codecs ● Implement MessageCodec[X] interface 30Scalaで必要なほぼ全てのデータ型へのマッピングをサポート

  31. 31. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. PlazmaDB: MessagePack DBMS ● Fluentd -> MessagePack -> Arm Treasure Data ● Automatically generating table schema from MessagePack data ● Apply schema–on-read for providing table data for Presto/Hive/Spark, etc. Table Schema Int Column Reader String Column Reader Update Schema Generate Reader Set Table Reader Schema-free Data 31 Data Collection Distributed Data Processing Arm Treasure DataはMessagePackベースのSchema-on-readシステム

  32. 32. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Schema-On-Read Data Processing with MessagePack ● Users can store arbitrary typed data (No table design is required) ● Data can be read in a target type required by the application (e.g., SQL query) Int Float Boolean String Array Map Binary SQL BigInt parseInt toInt 0 or 1 IntCodec Pack Unpack Error or null “100” (string) 100 (int) 100 (int) 32 Logs データ読み込み時に、アプリケーションの要求する型に合わせる (Schema-on-Read)
 CSV command-line arguments
  33. 33. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. airframe-surface: Object Shape Inspector ● Surface.of[X] 33
  34. 34. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. airframe-surface ● Reading Type Signatures From ScalaSig ● Scala compiler embeds Scala Type Signatures (ScalaSig) to class files ● Surface.of[A] ■ returns A’s parameter names and types class A (data:List[B]) class A data: List[java.lang.Object] class A data: List[java.lang.Object] ScalaSig: data:List[B] javac scalac Surface.of[A] data: List[B] scala.reflect.runtime.universe.TypeTag Type erasure removes generic type information 34オブジェクトの型情報をScalaSigから取得する

  35. 35. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. MessageCodec.of[A]: Combination of Codecs Unpack Pack IntCodec StringCodec DoubleCodec MessagePack MessageCodec.of[A] 35オブジェクトの型に合わせてCodecを合成
 Surface.of[A]
  36. 36. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. New: airframe-fluentd ● Collect Scala Application Logs To Fluentd ● Scala Objects -> MessagePack -> Fluentd 36MessagePackを受け取るFluentdには、airframe-codeの出力を渡せる
 Collect Query Logs Analyze Query Logs Machine Learning Query Optimization Optimize System airframe-fluentd Scala Objects airframe-codec
  37. 37. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. New: airframe-control: Retry ● Everything can fail … ● Network disconnection ● Servere crash ● ... ● Retry ● Exponential backoff ■ 2x, 4x, ... ● Jittering ■ 1 sec., 2 * rand, 4 * rand, … ● Customize error type classifiers ● retryable failures ● non-retryable failures 37リトライ処理をパターン化

  38. 38. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. New: airframe-control: Parallel ● Running tasks in parallel ● Customize parallelism ● Fixes slow tail issue of parallel collection 38
  39. 39. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Airframe 19 Milestones ● Airframe 19 Milestones ● https://github.com/wvlet/airframe/issues/528 ● airframe-http ● Under active development for production use ■ Request filters ■ Request logging ■ JMX metrics ■ OpenAPI spec generator (no code generation is planned) 39
  40. 40. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. airframe-http ● Created a simple HTTP framework ● Based on Airframe modules: ■ airframe-surface ■ airframe-codec ■ airframe-msgpack ■ etc. ● Blog ● Building Low-Friction Web Service Over Finagle ● Save the time for choosing a web framework: ● Many frameworks exist: ● e.g, Finatra, Finch, akka-http, spring, RESTeasy, open-api, swagger, etc. 40Airframe資産を活用して、Webフレームワークも手軽に作成

  41. 41. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. airframe-http: Scala as IDL ● IDL: Interface Definition Language ● e.g. ProtocolBuffers, gRPC ● Use Scala’s Standard Functionalities ● case classes ■ Define HTTP request/response data structure ● functions ■ Define HTTP endpoints ■ Use @Endpoint annotation for for URL to method mapping ● Router ● Router.add[X] ● filter andThen router (available in the next release) 41
  42. 42. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Path Mapping ● :param ● Mapping to a function argument ● /user/:id ■ /user/1 ■ def getUser(id:Int) ○ getUser(1) ● *param ● Mapping url path tail to a function argument ● /resource/*path ■ /resource/my/key/path ■ def getPath(path:String) ○ getPath(my/key/path) ● case classes ● request/response body ● JSON <-> MessagePack <-> Object 42
  43. 43. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. airframe-http-client ● Read server responses as target object types ● Accept: application/json (default) ● Accept: application/x-msgpack (Receive data in MessagePack format) 43
  44. 44. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Retry Patterns for HTTP requests ● Error handling of HTTP requests is difficult ● 4xx, 5xx status code ● Should we retry the request? ■ IOException, EOFException ■ TimeoutException ■ InterruptedException ■ SSLException ■ InvocationTargetException ● HTTP client ● request retries ● response mapping ■ JSON, MessagePack format ● airframe-codec 44間違いやすいHTTPリクエストのエラーハンドリングをライブラリ化

  45. 45. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. RouteMatcher ● Router ● List of @Endpoint definitions ● Graph ● Node ■ Path fragment ● Action ■ Put variable mappings for :param or *param ● Edge ■ Node -> Token (path fragment) -> Node ● Input: ● URL request path fragments ● Algorithm ● Build NFA (Non-deterministic Finite Automaton) ● Convert NFA into DFA (deterministic finite automaton) ● Traverse the automaton ● Call the matched function (MethodSurface) ■ buildControllerMethodArgs (Route.scala) 45
  46. 46. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Http Request/Response Adapters ● Finagle Adapters ● HttpRequest[finagle.Request] ● Upcoming Adapters ● jetty, okhttp 46
  47. 47. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. airframe-http-recorder ● Testing against actual web services is time consuming ● Record & Replay HTTP responses ● Reproducible results ● Runnable on small machines (e.g., Travis CI) 47HTTPリクエストをレコーディングして、Webサービスのテストを効率化する
 HTTP Request HTTP Recorder Request Real Web Service Recording Mode: Response HTTP Request HTTP Recorder Replay Mode: Request Response Recording Responses Request Recorded Responses
  48. 48. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. [WIP] Scala.js RPC ● Scala.js ● Compiling Scala code into JavaScript for Web Browsers ● airframe-codec: Passing model class data between Scala and Scala.js UserInfo MessagePack UserInfo Pack Unpack PackUnpack Scala Server Side Scala.js Client Side XML RPC 48airframe-codecはScala.js(ブラウザ側)とのデータ受け渡しにも使える

  49. 49. Copyright 1995-2018 Arm Limited (or its affiliates). All rights reserved. Demo: Live Coding ● Build a simple web server ● JSON messaging support ● build.sbt ● project/plugins.sbt ■ sbt-pack, sbt-revolver ● Command line program launcher ● airframe-launcher ● packaging ● sbt-pack ● Docker ● Add web APIs 49
  50. 50. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Scala At Arm Treasure Data ● Scala is now an official language at Arm Treasure Data ● 0 -> 10+ engineers who can write Scala ● Use cases are growing: ● Query optimization, API, Spark, data analysis, storage systems, service operation, etc. ● We are happy to share our Scala assets through Airframe! 50 Airframe Scalaエンジニアが充実してきたArm Treasure Data。Scalaの適用範囲も広がっている

  51. 51. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Presto Conference Tokyo 2019 ● July 11 (Thu), 2019, 13:30 ~ (Free) ● https://techplay.jp/event/733772 ● Inviting Presto Creators (Martin, Dain, David) ● Presto Software Foundation ● Talks from big Presto users in Japan ● Yahoo! JAPAN, LINE, Arm Treasure Data ● Presto Source Code Navigation 51 Presto Conference Tokyo 2019を7/11(木) 13:30~より開催 (参加無料)

  52. 52. Confidential © Arm 2017Confidential © Arm 2017Confidential © Arm 2017 Thank You! Danke! Merci! 谢谢! ありがとう! Gracias! Kiitos! 52

×