SlideShare a Scribd company logo

Airframe Meetup #3: 2019 Updates & AirSpec

Presentation slides of Airframe Meetup #3 https://airframe.connpass.com/event/148169/ - Airframe 19 Milestone - AirSpec: A new testing library for Scala -

1 of 40
Download to read offline
Taro L. Saito, Ph.D.
Arm Treasure Data
October 23nd, 2019
Arm Treasure Data Tokyo Office
Airframe Meetup #3
1
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
About Me: Taro L. Saito (Leo)
2
● Principal Software Engineer at Arm
Treasure Data
● Building distributed query engine services
● Living in US for 4 years
● DBMS & Data Science Background
● Ph.D. of Computer Science
● OSS Projects around Scala
● sbt-sonatype: used for releasing 3000+
Scala projects
● snappy-java: a compression library used
in Spark, Parquet, etc.
● etc.
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Recent Work: Designing Data-Intensive Applications
● Techniques and concepts around distributed
data processing systems
3
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Today’s Agenda
● Airframe Updates
● The current version: 19.10.1
■ 25+ releases since July 2019~
● AirSpec: a new testing framework for Scala
● Lifecycle design
● airframe-http updates
■ Route filters
● Releasing OSS with GitHub Actions
● sbt-sonatype: Blazing Fast Release to Sonatype
● 3 Lightning Talks
4
Airframe
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Airframe: Collection of Our Best Practices
● Gather the best practices of Scala into Airframe OSS
● Get the real experiences by operating 24/7 services
5
Knowledge
Experiences
Design Decisions
Products
24/7 Services
Business Values
Programming OSS Outcome
Airframe
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
AirSpec: A New Testing Framework for Scala and Scala.js
● Simple Usage
● import wvlet.airspec._
● Extend AirSpec
● Define test cases as public functions
● Low Learning Cost
● No annotation (like JUnit5) is
necessary.
● Simple assertions: assert(cond), x
shouldBe y, etc.
■ No need to learn other complex
DSLs.
6

Recommended

How To Use Scala At Work - Airframe In Action at Arm Treasure Data
How To Use Scala At Work - Airframe In Action at Arm Treasure DataHow To Use Scala At Work - Airframe In Action at Arm Treasure Data
How To Use Scala At Work - Airframe In Action at Arm Treasure DataTaro L. Saito
 
Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020
Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020
Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020Taro L. Saito
 
Presto At Arm Treasure Data - 2019 Updates
Presto At Arm Treasure Data - 2019 UpdatesPresto At Arm Treasure Data - 2019 Updates
Presto At Arm Treasure Data - 2019 UpdatesTaro L. Saito
 
td-spark internals: Extending Spark with Airframe - Spark Meetup Tokyo #3 2020
td-spark internals: Extending Spark with Airframe - Spark Meetup Tokyo #3 2020td-spark internals: Extending Spark with Airframe - Spark Meetup Tokyo #3 2020
td-spark internals: Extending Spark with Airframe - Spark Meetup Tokyo #3 2020Taro L. Saito
 
Reading The Source Code of Presto
Reading The Source Code of PrestoReading The Source Code of Presto
Reading The Source Code of PrestoTaro L. Saito
 
Unifying Frontend and Backend Development with Scala - ScalaCon 2021
Unifying Frontend and Backend Development with Scala - ScalaCon 2021Unifying Frontend and Backend Development with Scala - ScalaCon 2021
Unifying Frontend and Backend Development with Scala - ScalaCon 2021Taro L. Saito
 
Journey of Migrating 1 Million Presto Queries - Presto Webinar 2020
Journey of Migrating 1 Million Presto Queries - Presto Webinar 2020Journey of Migrating 1 Million Presto Queries - Presto Webinar 2020
Journey of Migrating 1 Million Presto Queries - Presto Webinar 2020Taro L. Saito
 

More Related Content

What's hot

Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17Taro L. Saito
 
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018Taro L. Saito
 
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia GuptaIntro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia GuptaInfluxData
 
Docker for mac & local developer environment optimization
Docker for mac & local developer environment optimizationDocker for mac & local developer environment optimization
Docker for mac & local developer environment optimizationRadek Baczynski
 
Search engine based on Elasticsearch
Search engine based on ElasticsearchSearch engine based on Elasticsearch
Search engine based on ElasticsearchRadek Baczynski
 
Rust in TiKV
Rust in TiKVRust in TiKV
Rust in TiKVPingCAP
 
Managing Machine Learning workflows on Treasure Data
Managing Machine Learning workflows on Treasure DataManaging Machine Learning workflows on Treasure Data
Managing Machine Learning workflows on Treasure DataAki Ariga
 
Scale Relational Database with NewSQL
Scale Relational Database with NewSQLScale Relational Database with NewSQL
Scale Relational Database with NewSQLPingCAP
 
Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...
Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...
Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...InfluxData
 
How to build TiDB
How to build TiDBHow to build TiDB
How to build TiDBPingCAP
 
A Brief Introduction of TiDB (Percona Live)
A Brief Introduction of TiDB (Percona Live)A Brief Introduction of TiDB (Percona Live)
A Brief Introduction of TiDB (Percona Live)PingCAP
 
Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...
Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...
Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...Flink Forward
 
InfluxDB 2.0: Dashboarding 101 by David G. Simmons
InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxDB 2.0: Dashboarding 101 by David G. Simmons
InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxData
 
TiDB for Big Data
TiDB for Big DataTiDB for Big Data
TiDB for Big DataPingCAP
 
InfluxDB 2.0 Client Libraries by Noah Crowley
InfluxDB 2.0 Client Libraries by Noah CrowleyInfluxDB 2.0 Client Libraries by Noah Crowley
InfluxDB 2.0 Client Libraries by Noah CrowleyInfluxData
 
Golang in TiDB (GopherChina 2017)
Golang in TiDB  (GopherChina 2017)Golang in TiDB  (GopherChina 2017)
Golang in TiDB (GopherChina 2017)PingCAP
 
Best Practices for Scaling an InfluxEnterprise Cluster
Best Practices for Scaling an InfluxEnterprise ClusterBest Practices for Scaling an InfluxEnterprise Cluster
Best Practices for Scaling an InfluxEnterprise ClusterInfluxData
 
TiDB as an HTAP Database
TiDB as an HTAP DatabaseTiDB as an HTAP Database
TiDB as an HTAP DatabasePingCAP
 
Introduction to Flink Streaming
Introduction to Flink StreamingIntroduction to Flink Streaming
Introduction to Flink Streamingdatamantra
 

What's hot (20)

Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
 
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
 
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia GuptaIntro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
 
Docker for mac & local developer environment optimization
Docker for mac & local developer environment optimizationDocker for mac & local developer environment optimization
Docker for mac & local developer environment optimization
 
Search engine based on Elasticsearch
Search engine based on ElasticsearchSearch engine based on Elasticsearch
Search engine based on Elasticsearch
 
Rust in TiKV
Rust in TiKVRust in TiKV
Rust in TiKV
 
Managing Machine Learning workflows on Treasure Data
Managing Machine Learning workflows on Treasure DataManaging Machine Learning workflows on Treasure Data
Managing Machine Learning workflows on Treasure Data
 
Scale Relational Database with NewSQL
Scale Relational Database with NewSQLScale Relational Database with NewSQL
Scale Relational Database with NewSQL
 
Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...
Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...
Wayfair Storefront Performance Monitoring with InfluxEnterprise by Richard La...
 
How to build TiDB
How to build TiDBHow to build TiDB
How to build TiDB
 
A Brief Introduction of TiDB (Percona Live)
A Brief Introduction of TiDB (Percona Live)A Brief Introduction of TiDB (Percona Live)
A Brief Introduction of TiDB (Percona Live)
 
Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...
Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...
Flink Forward Berlin 2017: Roberto Bentivoglio, Saverio Veltri - NSDB (Natura...
 
InfluxDB 2.0: Dashboarding 101 by David G. Simmons
InfluxDB 2.0: Dashboarding 101 by David G. SimmonsInfluxDB 2.0: Dashboarding 101 by David G. Simmons
InfluxDB 2.0: Dashboarding 101 by David G. Simmons
 
TiDB for Big Data
TiDB for Big DataTiDB for Big Data
TiDB for Big Data
 
InfluxDB 2.0 Client Libraries by Noah Crowley
InfluxDB 2.0 Client Libraries by Noah CrowleyInfluxDB 2.0 Client Libraries by Noah Crowley
InfluxDB 2.0 Client Libraries by Noah Crowley
 
TiDB Introduction
TiDB IntroductionTiDB Introduction
TiDB Introduction
 
Golang in TiDB (GopherChina 2017)
Golang in TiDB  (GopherChina 2017)Golang in TiDB  (GopherChina 2017)
Golang in TiDB (GopherChina 2017)
 
Best Practices for Scaling an InfluxEnterprise Cluster
Best Practices for Scaling an InfluxEnterprise ClusterBest Practices for Scaling an InfluxEnterprise Cluster
Best Practices for Scaling an InfluxEnterprise Cluster
 
TiDB as an HTAP Database
TiDB as an HTAP DatabaseTiDB as an HTAP Database
TiDB as an HTAP Database
 
Introduction to Flink Streaming
Introduction to Flink StreamingIntroduction to Flink Streaming
Introduction to Flink Streaming
 

Similar to Airframe Meetup #3: 2019 Updates & AirSpec

Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17
Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17
Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17Muga Nishizawa
 
Revisit Dependency Injection in scala
Revisit Dependency Injection in scalaRevisit Dependency Injection in scala
Revisit Dependency Injection in scalatakezoe
 
Apache Spark Performance Observations
Apache Spark Performance ObservationsApache Spark Performance Observations
Apache Spark Performance ObservationsAdam Roberts
 
IBM Runtimes Performance Observations with Apache Spark
IBM Runtimes Performance Observations with Apache SparkIBM Runtimes Performance Observations with Apache Spark
IBM Runtimes Performance Observations with Apache SparkAdamRobertsIBM
 
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
DevOpsDays Taipei 2019 - Mastering IaC the DevOps WayDevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Waysmalltown
 
Kubernetes is hard! Lessons learned taking our apps to Kubernetes - Eldad Ass...
Kubernetes is hard! Lessons learned taking our apps to Kubernetes - Eldad Ass...Kubernetes is hard! Lessons learned taking our apps to Kubernetes - Eldad Ass...
Kubernetes is hard! Lessons learned taking our apps to Kubernetes - Eldad Ass...Cloud Native Day Tel Aviv
 
Cisco connect montreal 2018 saalvare md-program-xr-v2
Cisco connect montreal 2018 saalvare md-program-xr-v2Cisco connect montreal 2018 saalvare md-program-xr-v2
Cisco connect montreal 2018 saalvare md-program-xr-v2Cisco Canada
 
Integrating ChatGPT with Apache Airflow
Integrating ChatGPT with Apache AirflowIntegrating ChatGPT with Apache Airflow
Integrating ChatGPT with Apache AirflowTatiana Al-Chueyr
 
Developer insight into why applications run amazingly Fast in CF 2018
Developer insight into why applications run amazingly Fast in CF 2018Developer insight into why applications run amazingly Fast in CF 2018
Developer insight into why applications run amazingly Fast in CF 2018Pavan Kumar
 
Integrating Existing C++ Libraries into PySpark with Esther Kundin
Integrating Existing C++ Libraries into PySpark with Esther KundinIntegrating Existing C++ Libraries into PySpark with Esther Kundin
Integrating Existing C++ Libraries into PySpark with Esther KundinDatabricks
 
Exposing GraphQLs as Managed APIs
Exposing GraphQLs as Managed APIsExposing GraphQLs as Managed APIs
Exposing GraphQLs as Managed APIsWSO2
 
Five cool ways the JVM can run Apache Spark faster
Five cool ways the JVM can run Apache Spark fasterFive cool ways the JVM can run Apache Spark faster
Five cool ways the JVM can run Apache Spark fasterTim Ellison
 
High-speed Database Throughput Using Apache Arrow Flight SQL
High-speed Database Throughput Using Apache Arrow Flight SQLHigh-speed Database Throughput Using Apache Arrow Flight SQL
High-speed Database Throughput Using Apache Arrow Flight SQLScyllaDB
 
Terraform 101: What's infrastructure as code?
Terraform 101: What's infrastructure as code?Terraform 101: What's infrastructure as code?
Terraform 101: What's infrastructure as code?GDX Wu
 
Sprint 134
Sprint 134Sprint 134
Sprint 134ManageIQ
 
Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...
Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...
Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...Nelson Calero
 
TSC BoF: OSS Toolchain Discussion - SFO17-409
TSC BoF: OSS Toolchain Discussion - SFO17-409TSC BoF: OSS Toolchain Discussion - SFO17-409
TSC BoF: OSS Toolchain Discussion - SFO17-409Linaro
 
OSDC 2019 | Introducing Kudo – Kubernetes Operators the easy way by Matt Jarvis
OSDC 2019 | Introducing Kudo – Kubernetes Operators the easy way by Matt JarvisOSDC 2019 | Introducing Kudo – Kubernetes Operators the easy way by Matt Jarvis
OSDC 2019 | Introducing Kudo – Kubernetes Operators the easy way by Matt JarvisNETWAYS
 
Accelerate Your C/C++ Applications with Amazon EC2 F1 Instances (CMP405) - AW...
Accelerate Your C/C++ Applications with Amazon EC2 F1 Instances (CMP405) - AW...Accelerate Your C/C++ Applications with Amazon EC2 F1 Instances (CMP405) - AW...
Accelerate Your C/C++ Applications with Amazon EC2 F1 Instances (CMP405) - AW...Amazon Web Services
 
Apache Big Data Europe 2016
Apache Big Data Europe 2016Apache Big Data Europe 2016
Apache Big Data Europe 2016Tim Ellison
 

Similar to Airframe Meetup #3: 2019 Updates & AirSpec (20)

Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17
Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17
Custom Script Execution Environment on TD Workflow @ TD Tech Talk 2018-10-17
 
Revisit Dependency Injection in scala
Revisit Dependency Injection in scalaRevisit Dependency Injection in scala
Revisit Dependency Injection in scala
 
Apache Spark Performance Observations
Apache Spark Performance ObservationsApache Spark Performance Observations
Apache Spark Performance Observations
 
IBM Runtimes Performance Observations with Apache Spark
IBM Runtimes Performance Observations with Apache SparkIBM Runtimes Performance Observations with Apache Spark
IBM Runtimes Performance Observations with Apache Spark
 
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
DevOpsDays Taipei 2019 - Mastering IaC the DevOps WayDevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
 
Kubernetes is hard! Lessons learned taking our apps to Kubernetes - Eldad Ass...
Kubernetes is hard! Lessons learned taking our apps to Kubernetes - Eldad Ass...Kubernetes is hard! Lessons learned taking our apps to Kubernetes - Eldad Ass...
Kubernetes is hard! Lessons learned taking our apps to Kubernetes - Eldad Ass...
 
Cisco connect montreal 2018 saalvare md-program-xr-v2
Cisco connect montreal 2018 saalvare md-program-xr-v2Cisco connect montreal 2018 saalvare md-program-xr-v2
Cisco connect montreal 2018 saalvare md-program-xr-v2
 
Integrating ChatGPT with Apache Airflow
Integrating ChatGPT with Apache AirflowIntegrating ChatGPT with Apache Airflow
Integrating ChatGPT with Apache Airflow
 
Developer insight into why applications run amazingly Fast in CF 2018
Developer insight into why applications run amazingly Fast in CF 2018Developer insight into why applications run amazingly Fast in CF 2018
Developer insight into why applications run amazingly Fast in CF 2018
 
Integrating Existing C++ Libraries into PySpark with Esther Kundin
Integrating Existing C++ Libraries into PySpark with Esther KundinIntegrating Existing C++ Libraries into PySpark with Esther Kundin
Integrating Existing C++ Libraries into PySpark with Esther Kundin
 
Exposing GraphQLs as Managed APIs
Exposing GraphQLs as Managed APIsExposing GraphQLs as Managed APIs
Exposing GraphQLs as Managed APIs
 
Five cool ways the JVM can run Apache Spark faster
Five cool ways the JVM can run Apache Spark fasterFive cool ways the JVM can run Apache Spark faster
Five cool ways the JVM can run Apache Spark faster
 
High-speed Database Throughput Using Apache Arrow Flight SQL
High-speed Database Throughput Using Apache Arrow Flight SQLHigh-speed Database Throughput Using Apache Arrow Flight SQL
High-speed Database Throughput Using Apache Arrow Flight SQL
 
Terraform 101: What's infrastructure as code?
Terraform 101: What's infrastructure as code?Terraform 101: What's infrastructure as code?
Terraform 101: What's infrastructure as code?
 
Sprint 134
Sprint 134Sprint 134
Sprint 134
 
Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...
Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...
Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...
 
TSC BoF: OSS Toolchain Discussion - SFO17-409
TSC BoF: OSS Toolchain Discussion - SFO17-409TSC BoF: OSS Toolchain Discussion - SFO17-409
TSC BoF: OSS Toolchain Discussion - SFO17-409
 
OSDC 2019 | Introducing Kudo – Kubernetes Operators the easy way by Matt Jarvis
OSDC 2019 | Introducing Kudo – Kubernetes Operators the easy way by Matt JarvisOSDC 2019 | Introducing Kudo – Kubernetes Operators the easy way by Matt Jarvis
OSDC 2019 | Introducing Kudo – Kubernetes Operators the easy way by Matt Jarvis
 
Accelerate Your C/C++ Applications with Amazon EC2 F1 Instances (CMP405) - AW...
Accelerate Your C/C++ Applications with Amazon EC2 F1 Instances (CMP405) - AW...Accelerate Your C/C++ Applications with Amazon EC2 F1 Instances (CMP405) - AW...
Accelerate Your C/C++ Applications with Amazon EC2 F1 Instances (CMP405) - AW...
 
Apache Big Data Europe 2016
Apache Big Data Europe 2016Apache Big Data Europe 2016
Apache Big Data Europe 2016
 

More from Taro L. Saito

Tips For Maintaining OSS Projects
Tips For Maintaining OSS ProjectsTips For Maintaining OSS Projects
Tips For Maintaining OSS ProjectsTaro L. Saito
 
Learning Silicon Valley Culture
Learning Silicon Valley CultureLearning Silicon Valley Culture
Learning Silicon Valley CultureTaro L. Saito
 
Presto At Treasure Data
Presto At Treasure DataPresto At Treasure Data
Presto At Treasure DataTaro L. Saito
 
Scala at Treasure Data
Scala at Treasure DataScala at Treasure Data
Scala at Treasure DataTaro L. Saito
 
Introduction to Presto at Treasure Data
Introduction to Presto at Treasure DataIntroduction to Presto at Treasure Data
Introduction to Presto at Treasure DataTaro L. Saito
 
Workflow Hacks #1 - dots. Tokyo
Workflow Hacks #1 - dots. TokyoWorkflow Hacks #1 - dots. Tokyo
Workflow Hacks #1 - dots. TokyoTaro L. Saito
 
Presto @ Treasure Data - Presto Meetup Boston 2015
Presto @ Treasure Data - Presto Meetup Boston 2015Presto @ Treasure Data - Presto Meetup Boston 2015
Presto @ Treasure Data - Presto Meetup Boston 2015Taro L. Saito
 
Presto As A Service - Treasure DataでのPresto運用事例
Presto As A Service - Treasure DataでのPresto運用事例Presto As A Service - Treasure DataでのPresto運用事例
Presto As A Service - Treasure DataでのPresto運用事例Taro L. Saito
 
Presto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoringPresto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoringTaro L. Saito
 
Treasure Dataを支える技術 - MessagePack編
Treasure Dataを支える技術 - MessagePack編Treasure Dataを支える技術 - MessagePack編
Treasure Dataを支える技術 - MessagePack編Taro L. Saito
 
Weaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
Weaving Dataflows with Silk - ScalaMatsuri 2014, TokyoWeaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
Weaving Dataflows with Silk - ScalaMatsuri 2014, TokyoTaro L. Saito
 
Spark Internals - Hadoop Source Code Reading #16 in Japan
Spark Internals - Hadoop Source Code Reading #16 in JapanSpark Internals - Hadoop Source Code Reading #16 in Japan
Spark Internals - Hadoop Source Code Reading #16 in JapanTaro L. Saito
 
Streaming Distributed Data Processing with Silk #deim2014
Streaming Distributed Data Processing with Silk #deim2014Streaming Distributed Data Processing with Silk #deim2014
Streaming Distributed Data Processing with Silk #deim2014Taro L. Saito
 
Silkによる並列分散ワークフロープログラミング
Silkによる並列分散ワークフロープログラミングSilkによる並列分散ワークフロープログラミング
Silkによる並列分散ワークフロープログラミングTaro L. Saito
 
2011年度 生物データベース論 2日目 木構造データ
2011年度 生物データベース論 2日目 木構造データ2011年度 生物データベース論 2日目 木構造データ
2011年度 生物データベース論 2日目 木構造データTaro L. Saito
 
Relational-Style XML Query @ SIGMOD-J 2008 Dec.
Relational-Style XML Query @ SIGMOD-J 2008 Dec.Relational-Style XML Query @ SIGMOD-J 2008 Dec.
Relational-Style XML Query @ SIGMOD-J 2008 Dec.Taro L. Saito
 

More from Taro L. Saito (17)

Tips For Maintaining OSS Projects
Tips For Maintaining OSS ProjectsTips For Maintaining OSS Projects
Tips For Maintaining OSS Projects
 
Learning Silicon Valley Culture
Learning Silicon Valley CultureLearning Silicon Valley Culture
Learning Silicon Valley Culture
 
Presto At Treasure Data
Presto At Treasure DataPresto At Treasure Data
Presto At Treasure Data
 
Scala at Treasure Data
Scala at Treasure DataScala at Treasure Data
Scala at Treasure Data
 
Introduction to Presto at Treasure Data
Introduction to Presto at Treasure DataIntroduction to Presto at Treasure Data
Introduction to Presto at Treasure Data
 
Workflow Hacks #1 - dots. Tokyo
Workflow Hacks #1 - dots. TokyoWorkflow Hacks #1 - dots. Tokyo
Workflow Hacks #1 - dots. Tokyo
 
Presto @ Treasure Data - Presto Meetup Boston 2015
Presto @ Treasure Data - Presto Meetup Boston 2015Presto @ Treasure Data - Presto Meetup Boston 2015
Presto @ Treasure Data - Presto Meetup Boston 2015
 
Presto As A Service - Treasure DataでのPresto運用事例
Presto As A Service - Treasure DataでのPresto運用事例Presto As A Service - Treasure DataでのPresto運用事例
Presto As A Service - Treasure DataでのPresto運用事例
 
JNuma Library
JNuma LibraryJNuma Library
JNuma Library
 
Presto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoringPresto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoring
 
Treasure Dataを支える技術 - MessagePack編
Treasure Dataを支える技術 - MessagePack編Treasure Dataを支える技術 - MessagePack編
Treasure Dataを支える技術 - MessagePack編
 
Weaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
Weaving Dataflows with Silk - ScalaMatsuri 2014, TokyoWeaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
Weaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
 
Spark Internals - Hadoop Source Code Reading #16 in Japan
Spark Internals - Hadoop Source Code Reading #16 in JapanSpark Internals - Hadoop Source Code Reading #16 in Japan
Spark Internals - Hadoop Source Code Reading #16 in Japan
 
Streaming Distributed Data Processing with Silk #deim2014
Streaming Distributed Data Processing with Silk #deim2014Streaming Distributed Data Processing with Silk #deim2014
Streaming Distributed Data Processing with Silk #deim2014
 
Silkによる並列分散ワークフロープログラミング
Silkによる並列分散ワークフロープログラミングSilkによる並列分散ワークフロープログラミング
Silkによる並列分散ワークフロープログラミング
 
2011年度 生物データベース論 2日目 木構造データ
2011年度 生物データベース論 2日目 木構造データ2011年度 生物データベース論 2日目 木構造データ
2011年度 生物データベース論 2日目 木構造データ
 
Relational-Style XML Query @ SIGMOD-J 2008 Dec.
Relational-Style XML Query @ SIGMOD-J 2008 Dec.Relational-Style XML Query @ SIGMOD-J 2008 Dec.
Relational-Style XML Query @ SIGMOD-J 2008 Dec.
 

Recently uploaded

Roundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfRoundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfMostafa Higazy
 
IT Nation Evolve event 2024 - Quarter 1
IT Nation Evolve event 2024  - Quarter 1IT Nation Evolve event 2024  - Quarter 1
IT Nation Evolve event 2024 - Quarter 1Inbay UK
 
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Product School
 
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...DianaGray10
 
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17Ana-Maria Mihalceanu
 
How AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxHow AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxInfosec
 
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docxLeveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docxVotarikari Shravan
 
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...Neo4j
 
Enterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewEnterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewAshraf Fouad
 
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...ISPMAIndia
 
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31shyamraj55
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoProduct School
 
"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor Fesenko"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor FesenkoFwdays
 
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Umar Saif
 
Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?MENGSAYLOEM1
 
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsFrom Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsInflectra
 
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...UiPathCommunity
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVARobert McDermott
 
Dynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringDynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringMassimo Talia
 
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Product School
 

Recently uploaded (20)

Roundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfRoundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdf
 
IT Nation Evolve event 2024 - Quarter 1
IT Nation Evolve event 2024  - Quarter 1IT Nation Evolve event 2024  - Quarter 1
IT Nation Evolve event 2024 - Quarter 1
 
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
 
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
 
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
 
How AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxHow AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptx
 
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docxLeveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
 
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
 
Enterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewEnterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book Review
 
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
 
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
Unleash the Solace Pub Sub connector | Banaglore MuleSoft Meetup #31
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
 
"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor Fesenko"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor Fesenko
 
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
 
Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?
 
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsFrom Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
 
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVA
 
Dynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringDynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineering
 
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
 

Airframe Meetup #3: 2019 Updates & AirSpec

  • 1. Taro L. Saito, Ph.D. Arm Treasure Data October 23nd, 2019 Arm Treasure Data Tokyo Office Airframe Meetup #3 1
  • 2. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. About Me: Taro L. Saito (Leo) 2 ● Principal Software Engineer at Arm Treasure Data ● Building distributed query engine services ● Living in US for 4 years ● DBMS & Data Science Background ● Ph.D. of Computer Science ● OSS Projects around Scala ● sbt-sonatype: used for releasing 3000+ Scala projects ● snappy-java: a compression library used in Spark, Parquet, etc. ● etc.
  • 3. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Recent Work: Designing Data-Intensive Applications ● Techniques and concepts around distributed data processing systems 3
  • 4. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Today’s Agenda ● Airframe Updates ● The current version: 19.10.1 ■ 25+ releases since July 2019~ ● AirSpec: a new testing framework for Scala ● Lifecycle design ● airframe-http updates ■ Route filters ● Releasing OSS with GitHub Actions ● sbt-sonatype: Blazing Fast Release to Sonatype ● 3 Lightning Talks 4 Airframe
  • 5. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Airframe: Collection of Our Best Practices ● Gather the best practices of Scala into Airframe OSS ● Get the real experiences by operating 24/7 services 5 Knowledge Experiences Design Decisions Products 24/7 Services Business Values Programming OSS Outcome Airframe
  • 6. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec: A New Testing Framework for Scala and Scala.js ● Simple Usage ● import wvlet.airspec._ ● Extend AirSpec ● Define test cases as public functions ● Low Learning Cost ● No annotation (like JUnit5) is necessary. ● Simple assertions: assert(cond), x shouldBe y, etc. ■ No need to learn other complex DSLs. 6
  • 7. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec: Assertions 7
  • 8. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Example: AirSpec Assertions 8
  • 9. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec: Scala.js ● ScalaTest RefSpec ● Has similar syntax with AirSpec, but it doesn’t support Scala.js because of lack of runtime reflection. ● AirSpec uses airframe-surface to collect test methods ■ Scala.js: Collect test methods (MethodSurface) at compile-time ■ Scala JVM: Uses reflection at runtime 9
  • 10. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Writing Specs in Natural Languages 10
  • 11. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec: Motivation ● Q: Why do you need a new testing framework? ● A: For maintaining Airframe ● Because of (relatively) slow ScalaTest releases, we couldn’t catch up with the latest versions of Scala and Scala.js. ■ We usually needed to wait for several weeks or years (!) to test new Scala versions. ■ ScalaTest still doesn’t support Scala.js 1.0.x ● A: Taking a balance between minimalist approaches and too complex DSLs ● minimalist: minitest, scala-verify ■ Simple, but using them is not so fun (e.g., no support for test filtering by name) ● complex DSLs: ScalaTest, Specs2, uTest ■ High-learning cost ■ Need an agreement on the assertion style within the team 11
  • 12. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec: Motivation ● Another answer: It was surprisingly simple to implement a testing framework 12
  • 13. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec Features ● Production Ready ● All Airframe test cases are now using AirSpec ● sbt integration ● Handy keyword search for test methods ■ > testOnly -- (a pattern for class or method names) ● Property-based testing integrated with ScalaCheck ● Scala 2.11, 2.12, 2.13, and Scala.js (0.6.x and 1.0.x) support ● Integration with Airframe ● Automatic DI session management ● handy logging with airframe-log 13
  • 14. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec: Logging with airframe-log 14
  • 15. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. AirSpec + DI ● Injecting dependency as test function arguments ● Override AirSpec.design to customize your designs ● Managing lifecycle of test dependencies ● The lifecycle (e.g., start and shutdown) of the injected services can be managed by Airframe DI ● Internally nested Airframe sessions will be created for individual test cases ● Nest-free resource management ● no loan pattern ● no monad 15 Lifecycle Design (New Feature!)
  • 16. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Pro Tips: AirSpec + Airframe DI ● Tips ● Use objects to define component designs ● Add objects for test modules and override existing bindings (if necessary) ● Use the designs inside AirSpec ● Why? ● Your design becomes reusable for both production/test code ● Airframe DI uses lazy binding by default, so having unnecessary binding is OK ■ If necessary, use coverage report option to minimize the design 16
  • 17. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. DI Coverage Stats 17
  • 18. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Property-Based Testing with ScalaCheck 18
  • 19. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Airframe DI Update: Lifecycle Design ● AutoCloseable (parent of Closeable interface) support ● If an injected instance implements AutoCloseable interface (def close()), it will be automatically closed upon Session.shutdown ● Support defining lifecycle hooks within a design ● Useful for reusing services with a proper lifecycle management ● e.g., http client/server, database connection pool, etc. ● Testing environments can be prepared by adding necessary designs: ■ design + service1Design + service2Design + ... 19
  • 20. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Example: Test Design 20
  • 21. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. airframe-http: Using Scala As An IDL ● IDL: Interface Definition Language ● e.g. ProtocolBuffers, gRPC ● Use Scala’s Standard Functionalities ● case classes ■ Define HTTP request/response data structure ● functions ■ Define HTTP endpoints ■ Use @Endpoint annotation for for URL to method mapping ● Router ● Router.add[X] ● filter andThen router (Added in 19.8.x) 21
  • 22. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Path Mapping ● :param ● Mapping to a function argument ● /user/:id ■ /user/1 ■ def getUser(id:Int) ○ getUser(1) ● *param ● Mapping url path tail to a function argument ● /resource/*path ■ /resource/my/key/path ■ def getPath(path:String) ○ getPath(my/key/path) ● case classes ● request/response body ● JSON <-> MessagePack <-> Object 22
  • 23. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. New: airframe-http: Request Filter ● Router ● .add[X] ● .andThen[X] ● Filter ● apply(request, context) ■ returns Future[Response] 23
  • 24. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. airframe-http-client ● Read server responses as target object types ● Supports both JSON and MessagePack ● Added requestFilter argument to manipulate HTTP headers 24
  • 25. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Distributed Tracing with DataDog APM ● Setup ● Just run JVM applications with -javaagent:dd-agent.jar ● Implicit logging ● No explicit instrumentation is necessary ■ ByteBuddy is used to inject instrumentation code ● Major libraries are already supported ■ https://github.com/DataDog/dd-trace-java/tree/master/dd-java-agent/instru mentation ■ e.g., jetty, netty4, akka, okhttp, aws-sdk, servlet, spring, JDBC connections, etc. ● Cost: $31/instance/month, $1.2 / 1M tracing events 25
  • 26. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. DataDog APM: Request Summary 26
  • 27. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Airframe/AirSpec 19 Milestones ● Airframe 19 Milestone ● https://github.com/wvlet/airframe/issues/528 ■ airframe-json: ○ Stream JSON -> MessagePack converter (done) ■ airframe-codec: Scala.js support ■ airframe-control: Adding circuit breaker ■ airframe-http: production ready ○ We have migrated several internal APIs into airframe-http. Advanced features like request logging, CORS support, admin interface, JMX metrics are internally available. Need to extract them. ■ airframe-http-client: Jetty, akka-http support ● AirSpec 19 Milestone ● https://github.com/wvlet/airframe/issues/606 ■ Standalone launcher and async testing ■ Using the return values of test functions for reporting ■ IntelliJ support (?) 27
  • 28. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. [WIP] Scala.js RPC ● Scala.js ● Compiling Scala code into JavaScript for Web Browsers ● airframe-codec: Passing model class data between Scala and Scala.js UserInfo MessagePack UserInfo Pack Unpack PackUnpack Scala Server Side Scala.js Client Side XML RPC 28 airframe-codec airframe-codec
  • 29. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Releasing Airframe ● Migrated from Travis CI to GitHub Actions ● Uses .github/workflows folder ● Examples: https://github.com/wvlet/airframe/tree/master/.github/workflows ● Good integration with the pull request UI 29
  • 30. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. GitHub Actions ● Keywords ● workflow -> job -> steps ● workflow ■ YAML file ■ define conditions to trigger jobs (PR, push, etc.) ● job ■ A unit of execution ○ = a single docker container ■ Multiple jobs run in parallel ■ No local folder sharing between jobs ● steps ■ Can import third-party step definitions ■ Define a sequence of command lines 30
  • 31. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. GitHub Actions: Secrets ● Secrets (e.g., password, token) can be defined in GitHub settings ● Secrets cannot be used in PRs nor forked repositories ● exception: GITHUB_TOKEN ■ e.g., for updating PR tags, comments, etc. ● Uploading test artifacts (e.g., test coverage report) in PR to other than GitHub is not straightforward 31
  • 32. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. GitHub Actions: Releasing to Sonatype ● Steps ● Import GPG private key ● publishSigned ■ sbt-pgp ● Upload to Sonatype ● Tips ● Needed to Install GPG1 ■ TTY env of GitHub action is broken, so gpg2-agent cannot pass PGP_PASSPHRASE to GPG2 ■ This might be a bug of sbt-pgp plugin ● Encode GPG private key with base64 and set it as secret.PGP_SECRET ■ decrypt with secret.PGP_PASSPHRASE 32
  • 33. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. ● Blog post: https://medium.com/@taroleo/sbt-sonatype-f02bdafd78f1 ● Airframe needs to upload 2000+ files for each release ● When Sonatype API is slow, it could take 20 minutes ~ 1 hour ● With bundle upload, it takes less than 1 minute ● Bundle Upload ● publishTo := target/sonatype-bundle/(version) ■ Collecting artifacts to the local folder ● Upload artifacts as a single jar sbt-sonatype: Blazing Fast Upload to Sonatype 33
  • 34. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Happy reports from sbt-sonatype users 34
  • 35. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. scala-steward ● A time saver for OSS maintainers ● Create PRs for library updates ● Became the top committer for October 2019 35
  • 36. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Summary ● AirSpec ● A new testing framework with DI support ● Define reusable designs ■ lifecycle of servicers can be designed now ● Airframe ● airframe-http is production-ready with filters ● Development ● Using GitHub Actions for CI and releasing Airframe ● Bundle upload with a new sbt-sonatype plugin 36
  • 37. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Join Arm Treasure Data Team! ● Solving Challenges In The Real World Data Processing ● Building a scalable data processing platform on the cloud ● Advanced data analytics with Presto, Hive, Spark, and workflows ■ Processing 1 million queries / day ● Providing distributed computing as a service for 450+ companies 37 Airframe
  • 38. Confidential © Arm 2017Confidential © Arm 2017Confidential © Arm 2017 Thank You! Danke! Merci! 谢谢! ありがとう! Gracias! Kiitos! 38
  • 39. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Module Mix-InPackaging HTTP Requests and Responses Data airframe-launcher > _ airframe-log production: port: 10010 user: xxxx ... airframe-config airframe-codec sbt-pack airframe-fluentd Scala Objects Table Data (CSV, TSV) JSON airframe-jsonairframe-surface airframe-tablet airframe-jmx Monitor Runtime States Generate Mapping Codec Metrics & Log Data JDBC ResultSets airframe-jdbc airframe-http airframe-http-finagle Launch HTTP Services airframe DI Debug Logs Schema-On-Read Mapping Airframe
  • 40. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. The Architecture of Arm Treasure Data 40 DataLogs Device Data Batch Data PlazmaDB Table Schema Data Collection Cloud Storage Distributed Data Processing 2 million records / sec. 130 trillion records 1 billion rows processed / sec. Jobs Job Management SQL Editor Scheduler Workflows Machine Learning Treasure Data OSS Third Party OSS