SlideShare a Scribd company logo

Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020

Scala is a powerful language; You can build front-end applications with Scala.js, and efficient backend application servers for JVM. In this session, we will learn how to build everything with Scala by using Airframe OSS framework. Airframe is a library designed for maximizing the advantages of Scala as a hybrid of object-oriented and functional programming language. In this session, we will learn how to use Airframe to build REST APIs and RPC (with Finagle or gRPC) services, and how to create frontend applications in Scala.js that interact with the servers using functional interfaces for dynamically updating web pages.

1 of 39
Download to read offline
1
Taro L. Saito
Treasure Data
October 17th, 2020
Scala Matsuri 2020
Scala For Everything
From Frontend to Backend Applications
Scala無双: フロントエンドからバックエンドも全てScalaでなぎ倒せ

Copyright 1995-2020 Treasure Data. All rights reserved.
About Me: Taro L. Saito
2
● Treasure Data
○ Principal Software Engineer
○ Living in US for 5 years
● Ph.D. of Computer Science
○ Database systems, genome science.
● OSS:
○ Airframe, snappy-java (used in
Parquet, Spark, etc.), sbt-sonatype,
sbt-pack, etc.
● Book:
自己紹介

Copyright 1995-2020 Treasure Data. All rights reserved.
Today’s Topic
● There are a lot of libraries and frameworks for backend and frontend applications
● Airframe
○ An OSS library collection that enables Scala-oriented development
● Goal: Forget everything other than Scala
3
Scala無双:Scala以外を全て駆逐する

Airframe
Scala.js Web Browsers
JSON
Copyright 1995-2020 Treasure Data. All rights reserved.
Airframe: Core Scala Modules of Treasure Data
● Airframe (GitHub: wvlet/airframe)
○ 20+ Common Utilities for Scala and Scala.js (Scala Matsuri 2019)
■ Logging, Dependency Injection (DI), JSON/MessagePack parser, etc.
○ Airframe RPC, Airframe Rx (Scala Matsuri 2020)
○ AirSpec
■ Testing framework for Scala and Scala.js (ScalaDays. Seattle, May 2021)
4
Knowledge
Experiences
Design Decisions
Products
24/7 Services
Business Values
Programming OSS Outcome
Airframe
AirframeはScala.jsでも利用することを前提に開発されてきたOSSツール群

Copyright 1995-2020 Treasure Data. All rights reserved.
Treasure Data: A Ready-to-Use Cloud Data Platform
5
Logs
Device
Data
Batch
Data
PlazmaDB
Table Schema
Data Collection Cloud Storage Distributed Data Processing
Jobs
Job Management
SQL Editor
Scheduler
Workflows
Machine
Learning
Treasure Data OSS
Third Party OSS
Data
トレジャーデータはすぐに使えるクラウドデータプラットフォーム

Copyright 1995-2020 Treasure Data. All rights reserved.
Scala.js Application: Presto Service Monitor
● Presto Usage: More than 1,000,000 queries / day (2020), 4 regions
6
Scala.jsで開発されたアプリケーションの例


Recommended

Airframe Meetup #3: 2019 Updates & AirSpec
Airframe Meetup #3: 2019 Updates & AirSpecAirframe Meetup #3: 2019 Updates & AirSpec
Airframe Meetup #3: 2019 Updates & AirSpecTaro L. Saito
 
How To Use Scala At Work - Airframe In Action at Arm Treasure Data
How To Use Scala At Work - Airframe In Action at Arm Treasure DataHow To Use Scala At Work - Airframe In Action at Arm Treasure Data
How To Use Scala At Work - Airframe In Action at Arm Treasure DataTaro L. Saito
 
td-spark internals: Extending Spark with Airframe - Spark Meetup Tokyo #3 2020
td-spark internals: Extending Spark with Airframe - Spark Meetup Tokyo #3 2020td-spark internals: Extending Spark with Airframe - Spark Meetup Tokyo #3 2020
td-spark internals: Extending Spark with Airframe - Spark Meetup Tokyo #3 2020Taro L. Saito
 
Unifying Frontend and Backend Development with Scala - ScalaCon 2021
Unifying Frontend and Backend Development with Scala - ScalaCon 2021Unifying Frontend and Backend Development with Scala - ScalaCon 2021
Unifying Frontend and Backend Development with Scala - ScalaCon 2021Taro L. Saito
 
Reading The Source Code of Presto
Reading The Source Code of PrestoReading The Source Code of Presto
Reading The Source Code of PrestoTaro L. Saito
 
Presto At Arm Treasure Data - 2019 Updates
Presto At Arm Treasure Data - 2019 UpdatesPresto At Arm Treasure Data - 2019 Updates
Presto At Arm Treasure Data - 2019 UpdatesTaro L. Saito
 
Journey of Migrating 1 Million Presto Queries - Presto Webinar 2020
Journey of Migrating 1 Million Presto Queries - Presto Webinar 2020Journey of Migrating 1 Million Presto Queries - Presto Webinar 2020
Journey of Migrating 1 Million Presto Queries - Presto Webinar 2020Taro L. Saito
 

More Related Content

What's hot

Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17Taro L. Saito
 
Docker for mac & local developer environment optimization
Docker for mac & local developer environment optimizationDocker for mac & local developer environment optimization
Docker for mac & local developer environment optimizationRadek Baczynski
 
Search engine based on Elasticsearch
Search engine based on ElasticsearchSearch engine based on Elasticsearch
Search engine based on ElasticsearchRadek Baczynski
 
Rust in TiKV
Rust in TiKVRust in TiKV
Rust in TiKVPingCAP
 
Scale Relational Database with NewSQL
Scale Relational Database with NewSQLScale Relational Database with NewSQL
Scale Relational Database with NewSQLPingCAP
 
How to build TiDB
How to build TiDBHow to build TiDB
How to build TiDBPingCAP
 
A Brief Introduction of TiDB (Percona Live)
A Brief Introduction of TiDB (Percona Live)A Brief Introduction of TiDB (Percona Live)
A Brief Introduction of TiDB (Percona Live)PingCAP
 
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia GuptaIntro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia GuptaInfluxData
 
The Dark Side Of Go -- Go runtime related problems in TiDB in production
The Dark Side Of Go -- Go runtime related problems in TiDB  in productionThe Dark Side Of Go -- Go runtime related problems in TiDB  in production
The Dark Side Of Go -- Go runtime related problems in TiDB in productionPingCAP
 
TiDB for Big Data
TiDB for Big DataTiDB for Big Data
TiDB for Big DataPingCAP
 
TiDB as an HTAP Database
TiDB as an HTAP DatabaseTiDB as an HTAP Database
TiDB as an HTAP DatabasePingCAP
 
Building a transactional key-value store that scales to 100+ nodes (percona l...
Building a transactional key-value store that scales to 100+ nodes (percona l...Building a transactional key-value store that scales to 100+ nodes (percona l...
Building a transactional key-value store that scales to 100+ nodes (percona l...PingCAP
 
TiDB Introduction - San Francisco MySQL Meetup
TiDB Introduction - San Francisco MySQL MeetupTiDB Introduction - San Francisco MySQL Meetup
TiDB Introduction - San Francisco MySQL MeetupMorgan Tocker
 
Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...
Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...
Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...Flink Forward
 
Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...
Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...
Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...InfluxData
 
InfluxDB 2.0: Dashboarding 101 by David G. Simmons
InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxDB 2.0: Dashboarding 101 by David G. Simmons
InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxData
 
p4srv6 (P4-16) design document rev1.0
p4srv6 (P4-16) design document rev1.0p4srv6 (P4-16) design document rev1.0
p4srv6 (P4-16) design document rev1.0Kentaro Ebisawa
 
InfluxDB 2.0 Client Libraries by Noah Crowley
InfluxDB 2.0 Client Libraries by Noah CrowleyInfluxDB 2.0 Client Libraries by Noah Crowley
InfluxDB 2.0 Client Libraries by Noah CrowleyInfluxData
 
[Webinar Slides] Programming the Network Dataplane in P4
[Webinar Slides] Programming the Network Dataplane in P4[Webinar Slides] Programming the Network Dataplane in P4
[Webinar Slides] Programming the Network Dataplane in P4Open Networking Summits
 

What's hot (20)

Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
 
Docker for mac & local developer environment optimization
Docker for mac & local developer environment optimizationDocker for mac & local developer environment optimization
Docker for mac & local developer environment optimization
 
Search engine based on Elasticsearch
Search engine based on ElasticsearchSearch engine based on Elasticsearch
Search engine based on Elasticsearch
 
Rust in TiKV
Rust in TiKVRust in TiKV
Rust in TiKV
 
Scale Relational Database with NewSQL
Scale Relational Database with NewSQLScale Relational Database with NewSQL
Scale Relational Database with NewSQL
 
How to build TiDB
How to build TiDBHow to build TiDB
How to build TiDB
 
A Brief Introduction of TiDB (Percona Live)
A Brief Introduction of TiDB (Percona Live)A Brief Introduction of TiDB (Percona Live)
A Brief Introduction of TiDB (Percona Live)
 
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia GuptaIntro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
 
The Dark Side Of Go -- Go runtime related problems in TiDB in production
The Dark Side Of Go -- Go runtime related problems in TiDB  in productionThe Dark Side Of Go -- Go runtime related problems in TiDB  in production
The Dark Side Of Go -- Go runtime related problems in TiDB in production
 
TiDB for Big Data
TiDB for Big DataTiDB for Big Data
TiDB for Big Data
 
TiDB Introduction
TiDB IntroductionTiDB Introduction
TiDB Introduction
 
TiDB as an HTAP Database
TiDB as an HTAP DatabaseTiDB as an HTAP Database
TiDB as an HTAP Database
 
Building a transactional key-value store that scales to 100+ nodes (percona l...
Building a transactional key-value store that scales to 100+ nodes (percona l...Building a transactional key-value store that scales to 100+ nodes (percona l...
Building a transactional key-value store that scales to 100+ nodes (percona l...
 
TiDB Introduction - San Francisco MySQL Meetup
TiDB Introduction - San Francisco MySQL MeetupTiDB Introduction - San Francisco MySQL Meetup
TiDB Introduction - San Francisco MySQL Meetup
 
Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...
Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...
Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...
 
Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...
Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...
Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...
 
InfluxDB 2.0: Dashboarding 101 by David G. Simmons
InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxDB 2.0: Dashboarding 101 by David G. Simmons
InfluxDB 2.0: Dashboarding 101 by David G. Simmons
 
p4srv6 (P4-16) design document rev1.0
p4srv6 (P4-16) design document rev1.0p4srv6 (P4-16) design document rev1.0
p4srv6 (P4-16) design document rev1.0
 
InfluxDB 2.0 Client Libraries by Noah Crowley
InfluxDB 2.0 Client Libraries by Noah CrowleyInfluxDB 2.0 Client Libraries by Noah Crowley
InfluxDB 2.0 Client Libraries by Noah Crowley
 
[Webinar Slides] Programming the Network Dataplane in P4
[Webinar Slides] Programming the Network Dataplane in P4[Webinar Slides] Programming the Network Dataplane in P4
[Webinar Slides] Programming the Network Dataplane in P4
 

Similar to Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020

Spark Summit EU 2015: Reynold Xin Keynote
Spark Summit EU 2015: Reynold Xin KeynoteSpark Summit EU 2015: Reynold Xin Keynote
Spark Summit EU 2015: Reynold Xin KeynoteDatabricks
 
Scaling Apache Spark on Kubernetes at Lyft
Scaling Apache Spark on Kubernetes at LyftScaling Apache Spark on Kubernetes at Lyft
Scaling Apache Spark on Kubernetes at LyftDatabricks
 
REST in Peace. Long live gRPC!
REST in Peace. Long live gRPC!REST in Peace. Long live gRPC!
REST in Peace. Long live gRPC!QAware GmbH
 
Pulsar summit asia 2021 apache pulsar with mqtt for edge computing
Pulsar summit asia 2021   apache pulsar with mqtt for edge computingPulsar summit asia 2021   apache pulsar with mqtt for edge computing
Pulsar summit asia 2021 apache pulsar with mqtt for edge computingTimothy Spann
 
Detecting Spoofing at IXPs
Detecting Spoofing at IXPsDetecting Spoofing at IXPs
Detecting Spoofing at IXPsAPNIC
 
Detecting spoofing at IxP's
Detecting spoofing at IxP'sDetecting spoofing at IxP's
Detecting spoofing at IxP'sTom Paseka
 
Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021
Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021
Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021StreamNative
 
apidays LIVE Helsinki - Implementing OpenAPI and GraphQL Services with gRPC b...
apidays LIVE Helsinki - Implementing OpenAPI and GraphQL Services with gRPC b...apidays LIVE Helsinki - Implementing OpenAPI and GraphQL Services with gRPC b...
apidays LIVE Helsinki - Implementing OpenAPI and GraphQL Services with gRPC b...apidays
 
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019Thomas Weise
 
Stratio Sparta 2.0
Stratio Sparta 2.0Stratio Sparta 2.0
Stratio Sparta 2.0Stratio
 
Up and Running with gRPC & Cloud Career [GDG-Cloud-Dhaka-IO/2022}
Up and Running with gRPC & Cloud Career [GDG-Cloud-Dhaka-IO/2022}Up and Running with gRPC & Cloud Career [GDG-Cloud-Dhaka-IO/2022}
Up and Running with gRPC & Cloud Career [GDG-Cloud-Dhaka-IO/2022}Md. Sadhan Sarker
 
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2aspyker
 
Seattle Spark Meetup Mobius CSharp API
Seattle Spark Meetup Mobius CSharp APISeattle Spark Meetup Mobius CSharp API
Seattle Spark Meetup Mobius CSharp APIshareddatamsft
 
GraphQL is actually rest
GraphQL is actually restGraphQL is actually rest
GraphQL is actually restJakub Riedl
 
Hands on with CoAP and Californium
Hands on with CoAP and CaliforniumHands on with CoAP and Californium
Hands on with CoAP and CaliforniumJulien Vermillard
 
Introduction to Backend Engineering
Introduction to Backend EngineeringIntroduction to Backend Engineering
Introduction to Backend EngineeringUdayYadav90
 
Network Test Automation - Net Ops Coding 2015
Network Test Automation - Net Ops Coding 2015Network Test Automation - Net Ops Coding 2015
Network Test Automation - Net Ops Coding 2015Hiroshi Ota
 
Apache Pulsar Development 101 with Python
Apache Pulsar Development 101 with PythonApache Pulsar Development 101 with Python
Apache Pulsar Development 101 with PythonTimothy Spann
 
Cloud native IPC for Microservices Workshop @ Containerdays 2022
Cloud native IPC for Microservices Workshop @ Containerdays 2022Cloud native IPC for Microservices Workshop @ Containerdays 2022
Cloud native IPC for Microservices Workshop @ Containerdays 2022QAware GmbH
 

Similar to Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020 (20)

Spark Summit EU 2015: Reynold Xin Keynote
Spark Summit EU 2015: Reynold Xin KeynoteSpark Summit EU 2015: Reynold Xin Keynote
Spark Summit EU 2015: Reynold Xin Keynote
 
Cloud Native API Design and Management
Cloud Native API Design and ManagementCloud Native API Design and Management
Cloud Native API Design and Management
 
Scaling Apache Spark on Kubernetes at Lyft
Scaling Apache Spark on Kubernetes at LyftScaling Apache Spark on Kubernetes at Lyft
Scaling Apache Spark on Kubernetes at Lyft
 
REST in Peace. Long live gRPC!
REST in Peace. Long live gRPC!REST in Peace. Long live gRPC!
REST in Peace. Long live gRPC!
 
Pulsar summit asia 2021 apache pulsar with mqtt for edge computing
Pulsar summit asia 2021   apache pulsar with mqtt for edge computingPulsar summit asia 2021   apache pulsar with mqtt for edge computing
Pulsar summit asia 2021 apache pulsar with mqtt for edge computing
 
Detecting Spoofing at IXPs
Detecting Spoofing at IXPsDetecting Spoofing at IXPs
Detecting Spoofing at IXPs
 
Detecting spoofing at IxP's
Detecting spoofing at IxP'sDetecting spoofing at IxP's
Detecting spoofing at IxP's
 
Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021
Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021
Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021
 
apidays LIVE Helsinki - Implementing OpenAPI and GraphQL Services with gRPC b...
apidays LIVE Helsinki - Implementing OpenAPI and GraphQL Services with gRPC b...apidays LIVE Helsinki - Implementing OpenAPI and GraphQL Services with gRPC b...
apidays LIVE Helsinki - Implementing OpenAPI and GraphQL Services with gRPC b...
 
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
 
Stratio Sparta 2.0
Stratio Sparta 2.0Stratio Sparta 2.0
Stratio Sparta 2.0
 
Up and Running with gRPC & Cloud Career [GDG-Cloud-Dhaka-IO/2022}
Up and Running with gRPC & Cloud Career [GDG-Cloud-Dhaka-IO/2022}Up and Running with gRPC & Cloud Career [GDG-Cloud-Dhaka-IO/2022}
Up and Running with gRPC & Cloud Career [GDG-Cloud-Dhaka-IO/2022}
 
Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2Netflix Open Source Meetup Season 4 Episode 2
Netflix Open Source Meetup Season 4 Episode 2
 
Seattle Spark Meetup Mobius CSharp API
Seattle Spark Meetup Mobius CSharp APISeattle Spark Meetup Mobius CSharp API
Seattle Spark Meetup Mobius CSharp API
 
GraphQL is actually rest
GraphQL is actually restGraphQL is actually rest
GraphQL is actually rest
 
Hands on with CoAP and Californium
Hands on with CoAP and CaliforniumHands on with CoAP and Californium
Hands on with CoAP and Californium
 
Introduction to Backend Engineering
Introduction to Backend EngineeringIntroduction to Backend Engineering
Introduction to Backend Engineering
 
Network Test Automation - Net Ops Coding 2015
Network Test Automation - Net Ops Coding 2015Network Test Automation - Net Ops Coding 2015
Network Test Automation - Net Ops Coding 2015
 
Apache Pulsar Development 101 with Python
Apache Pulsar Development 101 with PythonApache Pulsar Development 101 with Python
Apache Pulsar Development 101 with Python
 
Cloud native IPC for Microservices Workshop @ Containerdays 2022
Cloud native IPC for Microservices Workshop @ Containerdays 2022Cloud native IPC for Microservices Workshop @ Containerdays 2022
Cloud native IPC for Microservices Workshop @ Containerdays 2022
 

More from Taro L. Saito

Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018Taro L. Saito
 
Tips For Maintaining OSS Projects
Tips For Maintaining OSS ProjectsTips For Maintaining OSS Projects
Tips For Maintaining OSS ProjectsTaro L. Saito
 
Learning Silicon Valley Culture
Learning Silicon Valley CultureLearning Silicon Valley Culture
Learning Silicon Valley CultureTaro L. Saito
 
Presto At Treasure Data
Presto At Treasure DataPresto At Treasure Data
Presto At Treasure DataTaro L. Saito
 
Scala at Treasure Data
Scala at Treasure DataScala at Treasure Data
Scala at Treasure DataTaro L. Saito
 
Introduction to Presto at Treasure Data
Introduction to Presto at Treasure DataIntroduction to Presto at Treasure Data
Introduction to Presto at Treasure DataTaro L. Saito
 
Workflow Hacks #1 - dots. Tokyo
Workflow Hacks #1 - dots. TokyoWorkflow Hacks #1 - dots. Tokyo
Workflow Hacks #1 - dots. TokyoTaro L. Saito
 
Presto @ Treasure Data - Presto Meetup Boston 2015
Presto @ Treasure Data - Presto Meetup Boston 2015Presto @ Treasure Data - Presto Meetup Boston 2015
Presto @ Treasure Data - Presto Meetup Boston 2015Taro L. Saito
 
Presto As A Service - Treasure DataでのPresto運用事例
Presto As A Service - Treasure DataでのPresto運用事例Presto As A Service - Treasure DataでのPresto運用事例
Presto As A Service - Treasure DataでのPresto運用事例Taro L. Saito
 
Presto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoringPresto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoringTaro L. Saito
 
Treasure Dataを支える技術 - MessagePack編
Treasure Dataを支える技術 - MessagePack編Treasure Dataを支える技術 - MessagePack編
Treasure Dataを支える技術 - MessagePack編Taro L. Saito
 
Weaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
Weaving Dataflows with Silk - ScalaMatsuri 2014, TokyoWeaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
Weaving Dataflows with Silk - ScalaMatsuri 2014, TokyoTaro L. Saito
 
Spark Internals - Hadoop Source Code Reading #16 in Japan
Spark Internals - Hadoop Source Code Reading #16 in JapanSpark Internals - Hadoop Source Code Reading #16 in Japan
Spark Internals - Hadoop Source Code Reading #16 in JapanTaro L. Saito
 
Streaming Distributed Data Processing with Silk #deim2014
Streaming Distributed Data Processing with Silk #deim2014Streaming Distributed Data Processing with Silk #deim2014
Streaming Distributed Data Processing with Silk #deim2014Taro L. Saito
 
Silkによる並列分散ワークフロープログラミング
Silkによる並列分散ワークフロープログラミングSilkによる並列分散ワークフロープログラミング
Silkによる並列分散ワークフロープログラミングTaro L. Saito
 
2011年度 生物データベース論 2日目 木構造データ
2011年度 生物データベース論 2日目 木構造データ2011年度 生物データベース論 2日目 木構造データ
2011年度 生物データベース論 2日目 木構造データTaro L. Saito
 
Relational-Style XML Query @ SIGMOD-J 2008 Dec.
Relational-Style XML Query @ SIGMOD-J 2008 Dec.Relational-Style XML Query @ SIGMOD-J 2008 Dec.
Relational-Style XML Query @ SIGMOD-J 2008 Dec.Taro L. Saito
 

More from Taro L. Saito (18)

Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
 
Tips For Maintaining OSS Projects
Tips For Maintaining OSS ProjectsTips For Maintaining OSS Projects
Tips For Maintaining OSS Projects
 
Learning Silicon Valley Culture
Learning Silicon Valley CultureLearning Silicon Valley Culture
Learning Silicon Valley Culture
 
Presto At Treasure Data
Presto At Treasure DataPresto At Treasure Data
Presto At Treasure Data
 
Scala at Treasure Data
Scala at Treasure DataScala at Treasure Data
Scala at Treasure Data
 
Introduction to Presto at Treasure Data
Introduction to Presto at Treasure DataIntroduction to Presto at Treasure Data
Introduction to Presto at Treasure Data
 
Workflow Hacks #1 - dots. Tokyo
Workflow Hacks #1 - dots. TokyoWorkflow Hacks #1 - dots. Tokyo
Workflow Hacks #1 - dots. Tokyo
 
Presto @ Treasure Data - Presto Meetup Boston 2015
Presto @ Treasure Data - Presto Meetup Boston 2015Presto @ Treasure Data - Presto Meetup Boston 2015
Presto @ Treasure Data - Presto Meetup Boston 2015
 
Presto As A Service - Treasure DataでのPresto運用事例
Presto As A Service - Treasure DataでのPresto運用事例Presto As A Service - Treasure DataでのPresto運用事例
Presto As A Service - Treasure DataでのPresto運用事例
 
JNuma Library
JNuma LibraryJNuma Library
JNuma Library
 
Presto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoringPresto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoring
 
Treasure Dataを支える技術 - MessagePack編
Treasure Dataを支える技術 - MessagePack編Treasure Dataを支える技術 - MessagePack編
Treasure Dataを支える技術 - MessagePack編
 
Weaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
Weaving Dataflows with Silk - ScalaMatsuri 2014, TokyoWeaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
Weaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
 
Spark Internals - Hadoop Source Code Reading #16 in Japan
Spark Internals - Hadoop Source Code Reading #16 in JapanSpark Internals - Hadoop Source Code Reading #16 in Japan
Spark Internals - Hadoop Source Code Reading #16 in Japan
 
Streaming Distributed Data Processing with Silk #deim2014
Streaming Distributed Data Processing with Silk #deim2014Streaming Distributed Data Processing with Silk #deim2014
Streaming Distributed Data Processing with Silk #deim2014
 
Silkによる並列分散ワークフロープログラミング
Silkによる並列分散ワークフロープログラミングSilkによる並列分散ワークフロープログラミング
Silkによる並列分散ワークフロープログラミング
 
2011年度 生物データベース論 2日目 木構造データ
2011年度 生物データベース論 2日目 木構造データ2011年度 生物データベース論 2日目 木構造データ
2011年度 生物データベース論 2日目 木構造データ
 
Relational-Style XML Query @ SIGMOD-J 2008 Dec.
Relational-Style XML Query @ SIGMOD-J 2008 Dec.Relational-Style XML Query @ SIGMOD-J 2008 Dec.
Relational-Style XML Query @ SIGMOD-J 2008 Dec.
 

Recently uploaded

Roundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfRoundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfMostafa Higazy
 
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...ShapeBlue
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueCloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueShapeBlue
 
iOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingeriOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingerssuser9354ce
 
Battle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsBattle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsEvangelia Mitsopoulou
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriSafe Software
 
Enterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewEnterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewAshraf Fouad
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeJosh Gellers
 
Centralized TLS Certificates Management Using Vault PKI + Cert-Manager
Centralized TLS Certificates Management Using Vault PKI + Cert-ManagerCentralized TLS Certificates Management Using Vault PKI + Cert-Manager
Centralized TLS Certificates Management Using Vault PKI + Cert-ManagerSaiLinnThu2
 
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...Product School
 
New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024ThousandEyes
 
Q4 2023 Quarterly Investor Presentation - FINAL.pdf
Q4 2023 Quarterly Investor Presentation - FINAL.pdfQ4 2023 Quarterly Investor Presentation - FINAL.pdf
Q4 2023 Quarterly Investor Presentation - FINAL.pdfTejal81
 
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Umar Saif
 
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...SearchNorwich
 
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...UiPathCommunity
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxNeo4j
 
My Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceMy Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceVijayananda Mohire
 
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlueVM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlueShapeBlue
 
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...ShapeBlue
 
AI improves software testing to be more fault tolerant, focused and efficient
AI improves software testing to be more fault tolerant, focused and efficientAI improves software testing to be more fault tolerant, focused and efficient
AI improves software testing to be more fault tolerant, focused and efficientKari Kakkonen
 

Recently uploaded (20)

Roundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfRoundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdf
 
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueCloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
 
iOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingeriOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostinger
 
Battle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsBattle of React State Managers in frontend applications
Battle of React State Managers in frontend applications
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Enterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewEnterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book Review
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human Justice
 
Centralized TLS Certificates Management Using Vault PKI + Cert-Manager
Centralized TLS Certificates Management Using Vault PKI + Cert-ManagerCentralized TLS Certificates Management Using Vault PKI + Cert-Manager
Centralized TLS Certificates Management Using Vault PKI + Cert-Manager
 
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
 
New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024
 
Q4 2023 Quarterly Investor Presentation - FINAL.pdf
Q4 2023 Quarterly Investor Presentation - FINAL.pdfQ4 2023 Quarterly Investor Presentation - FINAL.pdf
Q4 2023 Quarterly Investor Presentation - FINAL.pdf
 
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
 
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
ChatGPT's Code Interpreter: Your secret weapon for SEO automation success - S...
 
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
 
My Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceMy Journey towards Artificial Intelligence
My Journey towards Artificial Intelligence
 
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlueVM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
 
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
 
AI improves software testing to be more fault tolerant, focused and efficient
AI improves software testing to be more fault tolerant, focused and efficientAI improves software testing to be more fault tolerant, focused and efficient
AI improves software testing to be more fault tolerant, focused and efficient
 

Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020

  • 1. 1 Taro L. Saito Treasure Data October 17th, 2020 Scala Matsuri 2020 Scala For Everything From Frontend to Backend Applications Scala無双: フロントエンドからバックエンドも全てScalaでなぎ倒せ

  • 2. Copyright 1995-2020 Treasure Data. All rights reserved. About Me: Taro L. Saito 2 ● Treasure Data ○ Principal Software Engineer ○ Living in US for 5 years ● Ph.D. of Computer Science ○ Database systems, genome science. ● OSS: ○ Airframe, snappy-java (used in Parquet, Spark, etc.), sbt-sonatype, sbt-pack, etc. ● Book: 自己紹介

  • 3. Copyright 1995-2020 Treasure Data. All rights reserved. Today’s Topic ● There are a lot of libraries and frameworks for backend and frontend applications ● Airframe ○ An OSS library collection that enables Scala-oriented development ● Goal: Forget everything other than Scala 3 Scala無双:Scala以外を全て駆逐する
 Airframe Scala.js Web Browsers JSON
  • 4. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe: Core Scala Modules of Treasure Data ● Airframe (GitHub: wvlet/airframe) ○ 20+ Common Utilities for Scala and Scala.js (Scala Matsuri 2019) ■ Logging, Dependency Injection (DI), JSON/MessagePack parser, etc. ○ Airframe RPC, Airframe Rx (Scala Matsuri 2020) ○ AirSpec ■ Testing framework for Scala and Scala.js (ScalaDays. Seattle, May 2021) 4 Knowledge Experiences Design Decisions Products 24/7 Services Business Values Programming OSS Outcome Airframe AirframeはScala.jsでも利用することを前提に開発されてきたOSSツール群

  • 5. Copyright 1995-2020 Treasure Data. All rights reserved. Treasure Data: A Ready-to-Use Cloud Data Platform 5 Logs Device Data Batch Data PlazmaDB Table Schema Data Collection Cloud Storage Distributed Data Processing Jobs Job Management SQL Editor Scheduler Workflows Machine Learning Treasure Data OSS Third Party OSS Data トレジャーデータはすぐに使えるクラウドデータプラットフォーム

  • 6. Copyright 1995-2020 Treasure Data. All rights reserved. Scala.js Application: Presto Service Monitor ● Presto Usage: More than 1,000,000 queries / day (2020), 4 regions 6 Scala.jsで開発されたアプリケーションの例

  • 7. Copyright 1995-2020 Treasure Data. All rights reserved. Server APIs and Library for Spark Airframe DI DataFrame MPC1 airframe-codec airframe-msgpack Plazma Public API airframe-http airframe-finagle Airframe DI Airframe RPC airframe-fluentd Master Worker DesignSparkContext TDSparkContext TDSparkService MPC1 Reader/Writer IO Manager Airframe DI airframe-http airframe-config airframe-launcher airframe-jmx airframe-metrics airframe-control airframe-metrics td-spark.jarairframe-log airframe-log airframe-codec airframe-json Airframe 7 Sparkをサポートするために、ScalaとAirframeで作られたサービスが使われている
 ● MPC1: MessagePack Columnar Format ● See also td-spark internals (Spark Meetup Tokyo 2020)
  • 8. Copyright 1995-2020 Treasure Data. All rights reserved. Scala is Versatile ● Backend ○ Can leverage JVM ecosystem (e.g., netty, gRPC, etc.) ● Frontend ○ Scala.js: ■ 1.0.0 is released in 2020 ● The current version is 1.3.0 ■ Compile Scala code into Javascripts ○ Out of scope of this talk ■ iOS/iPhone native applications ● How can we bridge server (Scala JVM) and client (Scala.js) applications? Scalaはバックエンド、フロントエンド開発に使える言語に。では両者をどう繋ぐ?
 Scala.js Web Browsers Serve-Client Interaction 8
  • 9. Copyright 1995-2020 Treasure Data. All rights reserved. Bridging Scala and Scala.js with RPC Program Function Call Return Local Function Call (ideal) Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize ローカル関数呼び出しと違い、実際にはネットワーク経由でのRPCが必要になる
 Remote Procedure Call (reality) Network Client Call Return Server Call Return 9 Scala.js Scala.js
  • 10. Copyright 1995-2020 Treasure Data. All rights reserved. Existing Approaches for Implementing RPC ● REST ○ Define function interfaces with HTTP endpoints (e.g., GET/POST/PUT/DELETE, etc.) ○ REST web frameworks for Scala ■ Play framework, akka-http, Finatra, Finch, skinny-framework, etc. ● gRPC ○ Define function interfaces with Google’s ProtocolBuffers (.proto) schema language ○ Generate server and client code stubs from proto files ○ Scala wrappers: ■ ScalaPB, muScala, akka-grpc, etc. RPCを実装するには、RESTやgRPCなどが使われている
 Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Return Server Call Return 10
  • 11. Copyright 1995-2020 Treasure Data. All rights reserved. API Design War https://cloud.google.com/blog/products/api-management/ understanding-grpc-openapi-and-rest-and-when-to-use-th em 11 ● Which API design should we choose? ● REST API ○ Designing good APIs over HTTP requires experiences and significant amount of time ● gRPC ○ Requires special software like code generators for using the ecosystem around ProtocolBuffers ● And also, there are too many choices for web frameworks しかし、APIデザイン、フレームワーク選びでの争いは付きない

  • 12. Copyright 1995-2020 Treasure Data. All rights reserved. Why Not Using Scala for RPC Interface? ● Scala is a functional and statically-typed object-oriented programming language ○ Scala functions = RPC methods ○ Scala objects (statically typed) = RPC request/response types ● Scala is a perfect fit for RPC! 12 関数型で静的型付きオブジェクト指向言語のScalaはRPCインターフェースに適している

  • 13. Copyright 1995-2020 Treasure Data. All rights reserved. Scala-First Approach: Airframe RPC RPC InterfaceScala.js Client RPC Web Server Generates Scala.js Web Application API Documentation 13 Airframe RPCはScalaをサーバー・クライアント共通インターフェースとして使う新手法
 RPC Call RPC Call Implements ● Forget about REST or ProtocolBuffers! ● Use Scala as a unified interface between server and clients Airframe
  • 14. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe RPC Backend Is Pluggable RPC Interface RPC Web Server Generates API Documentation 14 Airframe RPCのServer, Clientは差し替え可能
 Airframe RPC Call RPC Call Implements ● Backend server ○ Finagle (HTTP/1), gRPC (HTTP/2) ● Backend client ○ Finagle, gRPC, okhttp, Java URLConnection client etc. RPC Clients
  • 15. Copyright 1995-2020 Treasure Data. All rights reserved. Building Blocks of RPC Frameworks ● 1. Message serializer and deserializer ● 2. Network data format ○ JSON (REST), Protobuf (gRPC), or MessagePack (Airframe RPC) ● 3. RPC interface language ○ REST API, Protobuf, or Scala ● 4. HTTP client and server implementation ○ Code generator 15 Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Return Server Call Return RPCフレームワークを作るのに必要な構成要素

  • 16. Copyright 1995-2020 Treasure Data. All rights reserved. MessagePack-Based Object Serialization ● MessagePack ○ A compact binary format compatible with JSON Object Object Pack Unpack PackUnpack Server SideClient Side 16 Scala.js JSON Airframe MessagePackを使ってクライアントとサーバー間通信を行う

  • 17. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-codec: Pack/Unpack Interface ● Pack ○ Converting Data to MessagePack ● Unpack ○ Converting MessagePack to Data Input Output Pack Unpack PackUnpack 17 Airframe MessageCodec MessageCodec MessagePackを通したデータ変換インターフェースをScalaに適用

  • 18. Copyright 1995-2020 Treasure Data. All rights reserved. Pre-defined Codecs in airframe-codec ● Primitive Codecs ○ ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec ○ FloatCodec, DoubleCodec ○ StringCodec ○ BooleanCodec ○ TimeStampCodec ● Collection Codec ○ ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec, etc. ○ OptionCodec ● Java-specific Codec ○ UUIDCodec, FileCodec, ZonedDateTimeCodec, InstantCodec ○ JDBCResultSetCodec ● etc. 18 Scalaで必要なほぼ全てのデータ型へのマッピングをサポート
 Airframe
  • 19. Copyright 1995-2020 Treasure Data. All rights reserved. Serializing Complex Objects Pack Unpack IntCodec StringCodec DoubleCodec MessageCodec.of[A] 19 オブジェクトの型に合わせてCodecを合成できる
 Serialize Deserialize JSON
  • 20. Copyright 1995-2020 Treasure Data. All rights reserved. Defining RPC Interface with Scala ● Add @RPC annotation to a trait (= RPC interface) ● Define model classes (= data structures) using case classes ● Use the same interface and model classes between servers and clients RPC InterfaceをScalaで記述する
 20
  • 21. Copyright 1995-2020 Treasure Data. All rights reserved. Serializing Function Call Request ● Function call arguments ○ A sequence of argument name -> value, …. ● Example: ○ Map(“person” -> Person(1, “leo”), “message” -> “Hello RPC!”) ○ Serialize this Map into MessagePack with airframe-codec ● HTTP Mapping ○ HTTP method: POST ○ Path: /(package name)/hello ■ e.g., /hello.api.v1/hello ○ Content body: serialized function call argument data 21 関数呼び出しの引数もシリアライズする

  • 22. Copyright 1995-2020 Treasure Data. All rights reserved. Implementing RPC Servers ● Extending the RPC interface trait Serverの実装はRPC traitの継承
 22
  • 23. Copyright 1995-2020 Treasure Data. All rights reserved. ● Finagle: Twitter’s HTTP server written in Scala ○ HTTP/1 based. Accessible from web browsers running Scala.js code airframe-http: Starting A Finagle Backend RPC Server 23 Airframe Scala.js airframe-rx-html airframe-http: FinagleバックエンドのHTTPサーバーを起動する
 airframe-http-finagle
  • 24. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-http: gRPC Backend ● Suited to internal microservices ● HTTP/2 based protocol ○ Multiplexing multiple RPC requests within a single connection ● No .proto file is required 24 Airframe airframe-http: gRPCバックエンドサーバーを起動する。ProtoBuf定義は必要ない

  • 25. Copyright 1995-2020 Treasure Data. All rights reserved. Advanced Topic: Extending gRPC ● gRPC is data-format agonistic framework ○ To support Scala interface and MessagePack, we have two extension points in grpc-java. ● MethodDescriptor ○ Define gRPC endpoints corresponding to Scala functions ○ Register marshallers ● Request/ResponseMarshaller ○ Define how to encode/decode RPC request/response data 25 Airframe RPCはgrpc-javaを拡張してMessagePackデータの受け渡しをしている

  • 26. Copyright 1995-2020 Treasure Data. All rights reserved. RPC Performance Comparison (Greeter Service) ● Airframe RPC ○ serde: MessagePack (airframe-codec) <-> Scala case classes ○ Finagle (HTTP1) or gRPC (HTTP2) ● ScalaPB ○ serde: Protobuf <-> Scala case classes ○ gRPC (HTTP2) ● grpc-java ○ serde: Protobuf <-> Java classes ○ gRPC (HTTP2) ● Notes ○ Using Finagle with HTTP2 had almost no benefit over HTTP1 ■ Multiplexing RPC requests over HTTP2 is the key performance factor ○ Overhead of ScalaPB ■ Scala Future (10% overhead) ■ Mapping Protobuf to Scala case classes (10% overhead) 26 gRPCはリクエスト多重送信により高速。MessagePackによるオーバーヘッドは少ない
 HTTP/1 HTTP/2 (gRPC)
  • 27. Copyright 1995-2020 Treasure Data. All rights reserved. sbt-airframe: Generating RPC Clients ● sbt-airframe plugin ○ Read RPC interface classes ○ Generate HTTP client code for accessing the RPC server ○ OpenAPI schema generation support sbt-airframe Code Generation RPC Client Scala.js 27 Scala.js Client HTTP/gRPC Client Open API Spec Cross-Language RPC Client RPCクライアントを生成するsbt-airframeプラグイン

  • 28. Copyright 1995-2020 Treasure Data. All rights reserved. sbt-airframe: Generating Open API Schema ● RPC Interface -> Open API schema YAML file ● Generating RPC Clients & Swagger Documentation 28 ScalaのインターフェースからOpenAPIのYAMLを生成する
 RPC Interface API Documentation Open API Spec (YAML) Cross-Language RPC Client Generate sbt-airframe API Documentation
  • 29. Copyright 1995-2020 Treasure Data. All rights reserved. Making An RPC Call RPCを実行する。サーバー、クライアント共にScalaで動いている
 Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Response Server Call Response AirframeScala.js 29
  • 30. Copyright 1995-2020 Treasure Data. All rights reserved. Scala.js Programming Tips ● Scala.js: Compiling Scala -> JavaScript ● Use Pure Scala Code ○ [NG] Java code ■ Dependent libraries also need to be pure Scala ● Airframe is carefully designed to minimize non-Scala dependencies ○ [NG] Annotation, runtime-reflection ○ [OK] Scala Macros (compile-time meta-programming) ● RPC ○ Asynchronous calls using Scala Future ○ Await(Future) is not supported ■ Because JS is a single-threaded model. ■ Need to properly chain Future operators with map, flatMap, rescue, etc. 30 Scala.jsでのプログラミングのコツ
 Scala.js
  • 31. Copyright 1995-2020 Treasure Data. All rights reserved. Debugging Scala.js Code with airframe-log ● Showing logs at the developer console of the browser 31 airframe-logとブラウザのコンソールでScala.jsコードのデバッグ

  • 32. Copyright 1995-2020 Treasure Data. All rights reserved. Rendering DOM with Scala.js ● DOM element: ○ <div class=”container”> …. </div> ○ scalajs-dom library has functions necessary for generating DOMs ● Previous approach: Converting XML literal into DOM elements ○ monadic-html, Binding.scala ○ But XML literal will be deprecated in Dotty (Scala 3.0) 32 Scala.jsでのDOMレンダリング手法には様々な手法がある

  • 33. Copyright 1995-2020 Treasure Data. All rights reserved. Rendering DOM ● Current Best Practice ○ Using Scala functions for building DOM elements ● Option 1: Separation between DOM elements and attributes ○ ScalaTags, Slinky ○ Too many parentheses! ● Option 2: No separation between DOM elements and attributes ○ scalajs-react, Airframe Rx ○ More Scala-friendly syntax 33 DOMのelement, attributeを区別しない方が関数型に適している
 ScalaTags Airframe Rx
  • 34. Copyright 1995-2020 Treasure Data. All rights reserved. Complex DOM Rendering ● Airframe Rx ● RxElement interface ○ def render(): RxElement ○ Support nesting with apply(...) ■ Embedding Scala functions and collections 34 airframe-rx-html: Scalaの関数をネストしてDOMをレンダリングする

  • 35. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe Rx ● Rx.variable ○ Re-render DOM if the variable is updated ● Gathering best practices from existing libraries ○ Type-safe DOM elements and attributes ■ ScalaTags, scalajs-react ○ Reactive streaming ■ Monix ■ Scala Collection ■ Scala.rx ○ Cancelable Interface ■ Cleanup DOM and event handlers when updating DOM ■ Monix, monadic-html 35 既存技術の良いところ取りをしたAirframe Rx

  • 36. Copyright 1995-2020 Treasure Data. All rights reserved. Reactive Stream Interface: Rx[X] ● Rx[X] ○ ReactiveX stream implementation ○ e.g., Monix, Scala.Rx ● Reactive DOM Rendering ○ Rewriting DOM interactively upon upstream data change ● Reactive Operators ○ map, flatMap, filter, join, etc. ○ Almost the same with Scala collection APIs 36 Reactive stream interface Rx[X]でインタラクティブなDOMを作成

  • 37. Copyright 1995-2020 Treasure Data. All rights reserved. gRPC Streaming with Airframe Rx ● Interactive communication between server and clients with reactive stream interface: Rx[A] 37 gRPC StreamingもRxインターフェースでサポートできる

  • 38. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe Enables Scala-Oriented Development 38 RPC InterfaceScala.js Client RPC Web Server Generates Scala.js Web Application API Documentation RPC Call RPC Call Implements Airframe AirframeでScala中心のアプリケーション開発を可能に
 ● Only need to learn how to write Scala for kick-starting client/server application development
  • 39. Copyright 1995-2020 Treasure Data. All rights reserved. Summary ● Scala is a powerful language that can consolidate existing technologies ● Minimizing the learning cost of application development with Scala-First approach ○ Airframe OSS (GitHub: wvlet/airframe) is designed for this purpose 39 まとめ:既存技術の長所を兼ね備えることで、Scala無双を可能にするAirframe
 Airframe Scala.js Web Browsers JSON