SlideShare a Scribd company logo

Journey of Migrating 1 Million Presto Queries - Presto Webinar 2020

Arm Treasure Data utilizes Presto as the query engine processing over 1 million queries per day to support the data business of 500+ companies in three regions; US, EU, and Asia. Arm Treasure Data had been using Presto 0.205 and in 2019 started a big migration project to Presto 317. Although we performed extensive query simulations to check any incompatibilities, we faced many unexpected challenges during the migration at production.

1 of 35
Download to read offline
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Taro L. Saito
Arm Treasure Data
July 16th, 2020
Starburst Presto Webinar
Presto At Arm Treasure Data:
Journey of Migrating
1 Million Presto Queries
1
Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
About Me: Taro L. Saito (Leo)
2
● Principal Software Engineer of
Arm Treasure Data
● Ph.D. of Computer Science
● Active OSS developer
● Created Presto As A Service
● 6 years since 2014
Confidential © Arm 2017Confidential © Arm 2017Confidential © Arm 2017
Thank You!
Danke!
Merci!
谢谢!
ありがとう!
Gracias!
Kiitos!400+
Customers
Founded in
2011
Raised
$54M
Security
Acquired by Arm / Softbank
2018
Arm Treasure Data
Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Customer Data Platform (CDP) Solution
4
Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Arm Treasure Data: Ready-to-Use Cloud Data Platform From Day 1
5
Logs
Device
Data
Batch
Data
PlazmaDB
Table Schema
Data Collection Cloud Storage Distributed Data Processing
Jobs
Job Management
SQL Editor
Scheduler
Workflows
Machine
Learning
Treasure Data OSS
Third Party OSS
Data
Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved.
Data Connectors And Integrations
6

Recommended

How To Use Scala At Work - Airframe In Action at Arm Treasure Data
How To Use Scala At Work - Airframe In Action at Arm Treasure DataHow To Use Scala At Work - Airframe In Action at Arm Treasure Data
How To Use Scala At Work - Airframe In Action at Arm Treasure DataTaro L. Saito
 
Presto At Arm Treasure Data - 2019 Updates
Presto At Arm Treasure Data - 2019 UpdatesPresto At Arm Treasure Data - 2019 Updates
Presto At Arm Treasure Data - 2019 UpdatesTaro L. Saito
 
Airframe Meetup #3: 2019 Updates & AirSpec
Airframe Meetup #3: 2019 Updates & AirSpecAirframe Meetup #3: 2019 Updates & AirSpec
Airframe Meetup #3: 2019 Updates & AirSpecTaro L. Saito
 
Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020
Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020
Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020Taro L. Saito
 
Unifying Frontend and Backend Development with Scala - ScalaCon 2021
Unifying Frontend and Backend Development with Scala - ScalaCon 2021Unifying Frontend and Backend Development with Scala - ScalaCon 2021
Unifying Frontend and Backend Development with Scala - ScalaCon 2021Taro L. Saito
 
Reading The Source Code of Presto
Reading The Source Code of PrestoReading The Source Code of Presto
Reading The Source Code of PrestoTaro L. Saito
 
td-spark internals: Extending Spark with Airframe - Spark Meetup Tokyo #3 2020
td-spark internals: Extending Spark with Airframe - Spark Meetup Tokyo #3 2020td-spark internals: Extending Spark with Airframe - Spark Meetup Tokyo #3 2020
td-spark internals: Extending Spark with Airframe - Spark Meetup Tokyo #3 2020Taro L. Saito
 

More Related Content

What's hot

Presto conferencetokyo2019
Presto conferencetokyo2019Presto conferencetokyo2019
Presto conferencetokyo2019wyukawa
 
PGConf.ASIA 2019 Bali - Patroni on GitLab.com - Jose Cores Finnoto
PGConf.ASIA 2019 Bali - Patroni on GitLab.com - Jose Cores FinnotoPGConf.ASIA 2019 Bali - Patroni on GitLab.com - Jose Cores Finnoto
PGConf.ASIA 2019 Bali - Patroni on GitLab.com - Jose Cores FinnotoEqunix Business Solutions
 
PGConf.ASIA 2019 Bali - Partitioning in PostgreSQL - Amit Langote
PGConf.ASIA 2019 Bali -  Partitioning in PostgreSQL - Amit LangotePGConf.ASIA 2019 Bali -  Partitioning in PostgreSQL - Amit Langote
PGConf.ASIA 2019 Bali - Partitioning in PostgreSQL - Amit LangoteEqunix Business Solutions
 
PGConf.ASIA 2019 Bali - Patroni in 2019 - Alexander Kukushkin
PGConf.ASIA 2019 Bali - Patroni in 2019 - Alexander KukushkinPGConf.ASIA 2019 Bali - Patroni in 2019 - Alexander Kukushkin
PGConf.ASIA 2019 Bali - Patroni in 2019 - Alexander KukushkinEqunix Business Solutions
 
PGConf.ASIA 2019 Bali - Keynote Speech 3 - Kohei KaiGai
PGConf.ASIA 2019 Bali - Keynote Speech 3 - Kohei KaiGaiPGConf.ASIA 2019 Bali - Keynote Speech 3 - Kohei KaiGai
PGConf.ASIA 2019 Bali - Keynote Speech 3 - Kohei KaiGaiEqunix Business Solutions
 
PGConf.ASIA 2019 Bali - How did PostgreSQL Write Load Balancing of Queries Us...
PGConf.ASIA 2019 Bali - How did PostgreSQL Write Load Balancing of Queries Us...PGConf.ASIA 2019 Bali - How did PostgreSQL Write Load Balancing of Queries Us...
PGConf.ASIA 2019 Bali - How did PostgreSQL Write Load Balancing of Queries Us...Equnix Business Solutions
 
Pivotal Greenplum Cloud Marketplaces - Greenplum Summit 2019
Pivotal Greenplum Cloud Marketplaces - Greenplum Summit 2019Pivotal Greenplum Cloud Marketplaces - Greenplum Summit 2019
Pivotal Greenplum Cloud Marketplaces - Greenplum Summit 2019VMware Tanzu
 
PGConf.ASIA 2019 - PGSpider High Performance Cluster Engine - Shigeo Hirose
PGConf.ASIA 2019 - PGSpider High Performance Cluster Engine - Shigeo HirosePGConf.ASIA 2019 - PGSpider High Performance Cluster Engine - Shigeo Hirose
PGConf.ASIA 2019 - PGSpider High Performance Cluster Engine - Shigeo HiroseEqunix Business Solutions
 
Best Practices for Scaling an InfluxEnterprise Cluster
Best Practices for Scaling an InfluxEnterprise ClusterBest Practices for Scaling an InfluxEnterprise Cluster
Best Practices for Scaling an InfluxEnterprise ClusterInfluxData
 
Postgres Vision 2018: Will Postgres Live Forever?
Postgres Vision 2018: Will Postgres Live Forever?Postgres Vision 2018: Will Postgres Live Forever?
Postgres Vision 2018: Will Postgres Live Forever?EDB
 
Data Engineer’s Lunch #41: PygramETL
Data Engineer’s Lunch #41: PygramETLData Engineer’s Lunch #41: PygramETL
Data Engineer’s Lunch #41: PygramETLAnant Corporation
 
Sponsored Talk @ PGConf APAC 2018 - Choosing the right partner in your Postgr...
Sponsored Talk @ PGConf APAC 2018 - Choosing the right partner in your Postgr...Sponsored Talk @ PGConf APAC 2018 - Choosing the right partner in your Postgr...
Sponsored Talk @ PGConf APAC 2018 - Choosing the right partner in your Postgr...PGConf APAC
 
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia GuptaIntro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia GuptaInfluxData
 
PGConf.ASIA 2019 Bali - Keynote Speech 2 - Ivan Pachenko
PGConf.ASIA 2019 Bali - Keynote Speech 2 - Ivan PachenkoPGConf.ASIA 2019 Bali - Keynote Speech 2 - Ivan Pachenko
PGConf.ASIA 2019 Bali - Keynote Speech 2 - Ivan PachenkoEqunix Business Solutions
 
Iceberg: a fast table format for S3
Iceberg: a fast table format for S3Iceberg: a fast table format for S3
Iceberg: a fast table format for S3DataWorks Summit
 
TiDB Introduction - San Francisco MySQL Meetup
TiDB Introduction - San Francisco MySQL MeetupTiDB Introduction - San Francisco MySQL Meetup
TiDB Introduction - San Francisco MySQL MeetupMorgan Tocker
 
Introducing TiDB - Percona Live Frankfurt
Introducing TiDB - Percona Live FrankfurtIntroducing TiDB - Percona Live Frankfurt
Introducing TiDB - Percona Live FrankfurtMorgan Tocker
 
Geospatial querying in Apache Marmotta - ApacheCon Big Data Europe 2015
Geospatial querying in Apache Marmotta - ApacheCon Big Data Europe 2015Geospatial querying in Apache Marmotta - ApacheCon Big Data Europe 2015
Geospatial querying in Apache Marmotta - ApacheCon Big Data Europe 2015Sergio Fernández
 
PGConf APAC 2018 - PostgreSQL performance comparison in various clouds
PGConf APAC 2018 - PostgreSQL performance comparison in various cloudsPGConf APAC 2018 - PostgreSQL performance comparison in various clouds
PGConf APAC 2018 - PostgreSQL performance comparison in various cloudsPGConf APAC
 

What's hot (20)

Presto conferencetokyo2019
Presto conferencetokyo2019Presto conferencetokyo2019
Presto conferencetokyo2019
 
PGConf.ASIA 2019 Bali - Patroni on GitLab.com - Jose Cores Finnoto
PGConf.ASIA 2019 Bali - Patroni on GitLab.com - Jose Cores FinnotoPGConf.ASIA 2019 Bali - Patroni on GitLab.com - Jose Cores Finnoto
PGConf.ASIA 2019 Bali - Patroni on GitLab.com - Jose Cores Finnoto
 
PGConf.ASIA 2019 Bali - Partitioning in PostgreSQL - Amit Langote
PGConf.ASIA 2019 Bali -  Partitioning in PostgreSQL - Amit LangotePGConf.ASIA 2019 Bali -  Partitioning in PostgreSQL - Amit Langote
PGConf.ASIA 2019 Bali - Partitioning in PostgreSQL - Amit Langote
 
PGConf.ASIA 2019 Bali - Patroni in 2019 - Alexander Kukushkin
PGConf.ASIA 2019 Bali - Patroni in 2019 - Alexander KukushkinPGConf.ASIA 2019 Bali - Patroni in 2019 - Alexander Kukushkin
PGConf.ASIA 2019 Bali - Patroni in 2019 - Alexander Kukushkin
 
PGConf.ASIA 2019 Bali - Keynote Speech 3 - Kohei KaiGai
PGConf.ASIA 2019 Bali - Keynote Speech 3 - Kohei KaiGaiPGConf.ASIA 2019 Bali - Keynote Speech 3 - Kohei KaiGai
PGConf.ASIA 2019 Bali - Keynote Speech 3 - Kohei KaiGai
 
PGConf.ASIA 2019 Bali - How did PostgreSQL Write Load Balancing of Queries Us...
PGConf.ASIA 2019 Bali - How did PostgreSQL Write Load Balancing of Queries Us...PGConf.ASIA 2019 Bali - How did PostgreSQL Write Load Balancing of Queries Us...
PGConf.ASIA 2019 Bali - How did PostgreSQL Write Load Balancing of Queries Us...
 
Pivotal Greenplum Cloud Marketplaces - Greenplum Summit 2019
Pivotal Greenplum Cloud Marketplaces - Greenplum Summit 2019Pivotal Greenplum Cloud Marketplaces - Greenplum Summit 2019
Pivotal Greenplum Cloud Marketplaces - Greenplum Summit 2019
 
PGConf.ASIA 2019 - PGSpider High Performance Cluster Engine - Shigeo Hirose
PGConf.ASIA 2019 - PGSpider High Performance Cluster Engine - Shigeo HirosePGConf.ASIA 2019 - PGSpider High Performance Cluster Engine - Shigeo Hirose
PGConf.ASIA 2019 - PGSpider High Performance Cluster Engine - Shigeo Hirose
 
Best Practices for Scaling an InfluxEnterprise Cluster
Best Practices for Scaling an InfluxEnterprise ClusterBest Practices for Scaling an InfluxEnterprise Cluster
Best Practices for Scaling an InfluxEnterprise Cluster
 
TiDB Introduction
TiDB IntroductionTiDB Introduction
TiDB Introduction
 
Postgres Vision 2018: Will Postgres Live Forever?
Postgres Vision 2018: Will Postgres Live Forever?Postgres Vision 2018: Will Postgres Live Forever?
Postgres Vision 2018: Will Postgres Live Forever?
 
Data Engineer’s Lunch #41: PygramETL
Data Engineer’s Lunch #41: PygramETLData Engineer’s Lunch #41: PygramETL
Data Engineer’s Lunch #41: PygramETL
 
Sponsored Talk @ PGConf APAC 2018 - Choosing the right partner in your Postgr...
Sponsored Talk @ PGConf APAC 2018 - Choosing the right partner in your Postgr...Sponsored Talk @ PGConf APAC 2018 - Choosing the right partner in your Postgr...
Sponsored Talk @ PGConf APAC 2018 - Choosing the right partner in your Postgr...
 
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia GuptaIntro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
Intro to InfluxDB 2.0 and Your First Flux Query by Sonia Gupta
 
PGConf.ASIA 2019 Bali - Keynote Speech 2 - Ivan Pachenko
PGConf.ASIA 2019 Bali - Keynote Speech 2 - Ivan PachenkoPGConf.ASIA 2019 Bali - Keynote Speech 2 - Ivan Pachenko
PGConf.ASIA 2019 Bali - Keynote Speech 2 - Ivan Pachenko
 
Iceberg: a fast table format for S3
Iceberg: a fast table format for S3Iceberg: a fast table format for S3
Iceberg: a fast table format for S3
 
TiDB Introduction - San Francisco MySQL Meetup
TiDB Introduction - San Francisco MySQL MeetupTiDB Introduction - San Francisco MySQL Meetup
TiDB Introduction - San Francisco MySQL Meetup
 
Introducing TiDB - Percona Live Frankfurt
Introducing TiDB - Percona Live FrankfurtIntroducing TiDB - Percona Live Frankfurt
Introducing TiDB - Percona Live Frankfurt
 
Geospatial querying in Apache Marmotta - ApacheCon Big Data Europe 2015
Geospatial querying in Apache Marmotta - ApacheCon Big Data Europe 2015Geospatial querying in Apache Marmotta - ApacheCon Big Data Europe 2015
Geospatial querying in Apache Marmotta - ApacheCon Big Data Europe 2015
 
PGConf APAC 2018 - PostgreSQL performance comparison in various clouds
PGConf APAC 2018 - PostgreSQL performance comparison in various cloudsPGConf APAC 2018 - PostgreSQL performance comparison in various clouds
PGConf APAC 2018 - PostgreSQL performance comparison in various clouds
 

Similar to Journey of Migrating 1 Million Presto Queries - Presto Webinar 2020

Make your data fly - Building data platform in AWS
Make your data fly - Building data platform in AWSMake your data fly - Building data platform in AWS
Make your data fly - Building data platform in AWSKimmo Kantojärvi
 
The Lyft data platform: Now and in the future
The Lyft data platform: Now and in the futureThe Lyft data platform: Now and in the future
The Lyft data platform: Now and in the futuremarkgrover
 
Lyft data Platform - 2019 slides
Lyft data Platform - 2019 slidesLyft data Platform - 2019 slides
Lyft data Platform - 2019 slidesKarthik Murugesan
 
Scaling up uber's real time data analytics
Scaling up uber's real time data analyticsScaling up uber's real time data analytics
Scaling up uber's real time data analyticsXiang Fu
 
Managing Machine Learning workflows on Treasure Data
Managing Machine Learning workflows on Treasure DataManaging Machine Learning workflows on Treasure Data
Managing Machine Learning workflows on Treasure DataAki Ariga
 
Advanced technologies and techniques for debugging HPC applications
Advanced technologies and techniques for debugging HPC applicationsAdvanced technologies and techniques for debugging HPC applications
Advanced technologies and techniques for debugging HPC applicationsRogue Wave Software
 
Танки в облаках
Танки в облакахТанки в облаках
Танки в облакахOpenStackRussia
 
Conquering Data Migration from Oracle to Postgres
Conquering Data Migration from Oracle to PostgresConquering Data Migration from Oracle to Postgres
Conquering Data Migration from Oracle to PostgresEDB
 
Interconnect session 3498: Deployment Topologies for Jazz Reporting Service
Interconnect session 3498: Deployment Topologies for Jazz Reporting ServiceInterconnect session 3498: Deployment Topologies for Jazz Reporting Service
Interconnect session 3498: Deployment Topologies for Jazz Reporting ServiceRosa Naranjo
 
Parallel Processing in TM1 - QueBIT Consulting
Parallel Processing in TM1 - QueBIT ConsultingParallel Processing in TM1 - QueBIT Consulting
Parallel Processing in TM1 - QueBIT ConsultingQueBIT Consulting
 
Migration From Oracle to PostgreSQL
Migration From Oracle to PostgreSQLMigration From Oracle to PostgreSQL
Migration From Oracle to PostgreSQLPGConf APAC
 
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...MongoDB
 
Greenplum Database Open Source December 2015
Greenplum Database Open Source December 2015Greenplum Database Open Source December 2015
Greenplum Database Open Source December 2015PivotalOpenSourceHub
 
Beginners Guide to High Availability for Postgres
Beginners Guide to High Availability for PostgresBeginners Guide to High Availability for Postgres
Beginners Guide to High Availability for PostgresEDB
 
Module 9: CDB Technical Intro
 Module 9: CDB Technical Intro Module 9: CDB Technical Intro
Module 9: CDB Technical IntroTail-f Systems
 
OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...
OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...
OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...NETWAYS
 
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018Taro L. Saito
 
How to Upgrade Major Version of Your Production PostgreSQL
How to Upgrade Major Version of Your Production PostgreSQLHow to Upgrade Major Version of Your Production PostgreSQL
How to Upgrade Major Version of Your Production PostgreSQLKeisuke Suzuki
 
Peteris Arajs - Where is my data
Peteris Arajs - Where is my dataPeteris Arajs - Where is my data
Peteris Arajs - Where is my dataAndrejs Vorobjovs
 

Similar to Journey of Migrating 1 Million Presto Queries - Presto Webinar 2020 (20)

Make your data fly - Building data platform in AWS
Make your data fly - Building data platform in AWSMake your data fly - Building data platform in AWS
Make your data fly - Building data platform in AWS
 
The Lyft data platform: Now and in the future
The Lyft data platform: Now and in the futureThe Lyft data platform: Now and in the future
The Lyft data platform: Now and in the future
 
Lyft data Platform - 2019 slides
Lyft data Platform - 2019 slidesLyft data Platform - 2019 slides
Lyft data Platform - 2019 slides
 
Scaling up uber's real time data analytics
Scaling up uber's real time data analyticsScaling up uber's real time data analytics
Scaling up uber's real time data analytics
 
Managing Machine Learning workflows on Treasure Data
Managing Machine Learning workflows on Treasure DataManaging Machine Learning workflows on Treasure Data
Managing Machine Learning workflows on Treasure Data
 
Advanced technologies and techniques for debugging HPC applications
Advanced technologies and techniques for debugging HPC applicationsAdvanced technologies and techniques for debugging HPC applications
Advanced technologies and techniques for debugging HPC applications
 
Танки в облаках
Танки в облакахТанки в облаках
Танки в облаках
 
Conquering Data Migration from Oracle to Postgres
Conquering Data Migration from Oracle to PostgresConquering Data Migration from Oracle to Postgres
Conquering Data Migration from Oracle to Postgres
 
Interconnect session 3498: Deployment Topologies for Jazz Reporting Service
Interconnect session 3498: Deployment Topologies for Jazz Reporting ServiceInterconnect session 3498: Deployment Topologies for Jazz Reporting Service
Interconnect session 3498: Deployment Topologies for Jazz Reporting Service
 
Parallel Processing in TM1 - QueBIT Consulting
Parallel Processing in TM1 - QueBIT ConsultingParallel Processing in TM1 - QueBIT Consulting
Parallel Processing in TM1 - QueBIT Consulting
 
Migration From Oracle to PostgreSQL
Migration From Oracle to PostgreSQLMigration From Oracle to PostgreSQL
Migration From Oracle to PostgreSQL
 
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
MongoDB World 2018: Managing a Mission Critical eCommerce Application on Mong...
 
Greenplum Database Open Source December 2015
Greenplum Database Open Source December 2015Greenplum Database Open Source December 2015
Greenplum Database Open Source December 2015
 
Beginners Guide to High Availability for Postgres
Beginners Guide to High Availability for PostgresBeginners Guide to High Availability for Postgres
Beginners Guide to High Availability for Postgres
 
Module 9: CDB Technical Intro
 Module 9: CDB Technical Intro Module 9: CDB Technical Intro
Module 9: CDB Technical Intro
 
Vision2015-CBS-1148-Final
Vision2015-CBS-1148-FinalVision2015-CBS-1148-Final
Vision2015-CBS-1148-Final
 
OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...
OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...
OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...
 
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
Airframe: Lightweight Building Blocks for Scala - Scale By The Bay 2018
 
How to Upgrade Major Version of Your Production PostgreSQL
How to Upgrade Major Version of Your Production PostgreSQLHow to Upgrade Major Version of Your Production PostgreSQL
How to Upgrade Major Version of Your Production PostgreSQL
 
Peteris Arajs - Where is my data
Peteris Arajs - Where is my dataPeteris Arajs - Where is my data
Peteris Arajs - Where is my data
 

More from Taro L. Saito

Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17Taro L. Saito
 
Tips For Maintaining OSS Projects
Tips For Maintaining OSS ProjectsTips For Maintaining OSS Projects
Tips For Maintaining OSS ProjectsTaro L. Saito
 
Learning Silicon Valley Culture
Learning Silicon Valley CultureLearning Silicon Valley Culture
Learning Silicon Valley CultureTaro L. Saito
 
Presto At Treasure Data
Presto At Treasure DataPresto At Treasure Data
Presto At Treasure DataTaro L. Saito
 
Scala at Treasure Data
Scala at Treasure DataScala at Treasure Data
Scala at Treasure DataTaro L. Saito
 
Introduction to Presto at Treasure Data
Introduction to Presto at Treasure DataIntroduction to Presto at Treasure Data
Introduction to Presto at Treasure DataTaro L. Saito
 
Workflow Hacks #1 - dots. Tokyo
Workflow Hacks #1 - dots. TokyoWorkflow Hacks #1 - dots. Tokyo
Workflow Hacks #1 - dots. TokyoTaro L. Saito
 
Presto @ Treasure Data - Presto Meetup Boston 2015
Presto @ Treasure Data - Presto Meetup Boston 2015Presto @ Treasure Data - Presto Meetup Boston 2015
Presto @ Treasure Data - Presto Meetup Boston 2015Taro L. Saito
 
Presto As A Service - Treasure DataでのPresto運用事例
Presto As A Service - Treasure DataでのPresto運用事例Presto As A Service - Treasure DataでのPresto運用事例
Presto As A Service - Treasure DataでのPresto運用事例Taro L. Saito
 
Presto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoringPresto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoringTaro L. Saito
 
Treasure Dataを支える技術 - MessagePack編
Treasure Dataを支える技術 - MessagePack編Treasure Dataを支える技術 - MessagePack編
Treasure Dataを支える技術 - MessagePack編Taro L. Saito
 
Weaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
Weaving Dataflows with Silk - ScalaMatsuri 2014, TokyoWeaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
Weaving Dataflows with Silk - ScalaMatsuri 2014, TokyoTaro L. Saito
 
Spark Internals - Hadoop Source Code Reading #16 in Japan
Spark Internals - Hadoop Source Code Reading #16 in JapanSpark Internals - Hadoop Source Code Reading #16 in Japan
Spark Internals - Hadoop Source Code Reading #16 in JapanTaro L. Saito
 
Streaming Distributed Data Processing with Silk #deim2014
Streaming Distributed Data Processing with Silk #deim2014Streaming Distributed Data Processing with Silk #deim2014
Streaming Distributed Data Processing with Silk #deim2014Taro L. Saito
 
Silkによる並列分散ワークフロープログラミング
Silkによる並列分散ワークフロープログラミングSilkによる並列分散ワークフロープログラミング
Silkによる並列分散ワークフロープログラミングTaro L. Saito
 
2011年度 生物データベース論 2日目 木構造データ
2011年度 生物データベース論 2日目 木構造データ2011年度 生物データベース論 2日目 木構造データ
2011年度 生物データベース論 2日目 木構造データTaro L. Saito
 
Relational-Style XML Query @ SIGMOD-J 2008 Dec.
Relational-Style XML Query @ SIGMOD-J 2008 Dec.Relational-Style XML Query @ SIGMOD-J 2008 Dec.
Relational-Style XML Query @ SIGMOD-J 2008 Dec.Taro L. Saito
 

More from Taro L. Saito (18)

Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
Airframe: Lightweight Building Blocks for Scala @ TD Tech Talk 2018-10-17
 
Tips For Maintaining OSS Projects
Tips For Maintaining OSS ProjectsTips For Maintaining OSS Projects
Tips For Maintaining OSS Projects
 
Learning Silicon Valley Culture
Learning Silicon Valley CultureLearning Silicon Valley Culture
Learning Silicon Valley Culture
 
Presto At Treasure Data
Presto At Treasure DataPresto At Treasure Data
Presto At Treasure Data
 
Scala at Treasure Data
Scala at Treasure DataScala at Treasure Data
Scala at Treasure Data
 
Introduction to Presto at Treasure Data
Introduction to Presto at Treasure DataIntroduction to Presto at Treasure Data
Introduction to Presto at Treasure Data
 
Workflow Hacks #1 - dots. Tokyo
Workflow Hacks #1 - dots. TokyoWorkflow Hacks #1 - dots. Tokyo
Workflow Hacks #1 - dots. Tokyo
 
Presto @ Treasure Data - Presto Meetup Boston 2015
Presto @ Treasure Data - Presto Meetup Boston 2015Presto @ Treasure Data - Presto Meetup Boston 2015
Presto @ Treasure Data - Presto Meetup Boston 2015
 
Presto As A Service - Treasure DataでのPresto運用事例
Presto As A Service - Treasure DataでのPresto運用事例Presto As A Service - Treasure DataでのPresto運用事例
Presto As A Service - Treasure DataでのPresto運用事例
 
JNuma Library
JNuma LibraryJNuma Library
JNuma Library
 
Presto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoringPresto as a Service - Tips for operation and monitoring
Presto as a Service - Tips for operation and monitoring
 
Treasure Dataを支える技術 - MessagePack編
Treasure Dataを支える技術 - MessagePack編Treasure Dataを支える技術 - MessagePack編
Treasure Dataを支える技術 - MessagePack編
 
Weaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
Weaving Dataflows with Silk - ScalaMatsuri 2014, TokyoWeaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
Weaving Dataflows with Silk - ScalaMatsuri 2014, Tokyo
 
Spark Internals - Hadoop Source Code Reading #16 in Japan
Spark Internals - Hadoop Source Code Reading #16 in JapanSpark Internals - Hadoop Source Code Reading #16 in Japan
Spark Internals - Hadoop Source Code Reading #16 in Japan
 
Streaming Distributed Data Processing with Silk #deim2014
Streaming Distributed Data Processing with Silk #deim2014Streaming Distributed Data Processing with Silk #deim2014
Streaming Distributed Data Processing with Silk #deim2014
 
Silkによる並列分散ワークフロープログラミング
Silkによる並列分散ワークフロープログラミングSilkによる並列分散ワークフロープログラミング
Silkによる並列分散ワークフロープログラミング
 
2011年度 生物データベース論 2日目 木構造データ
2011年度 生物データベース論 2日目 木構造データ2011年度 生物データベース論 2日目 木構造データ
2011年度 生物データベース論 2日目 木構造データ
 
Relational-Style XML Query @ SIGMOD-J 2008 Dec.
Relational-Style XML Query @ SIGMOD-J 2008 Dec.Relational-Style XML Query @ SIGMOD-J 2008 Dec.
Relational-Style XML Query @ SIGMOD-J 2008 Dec.
 

Recently uploaded

Roundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfRoundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfMostafa Higazy
 
Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...BookNet Canada
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeJosh Gellers
 
iOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingeriOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingerssuser9354ce
 
Q4 2023 Quarterly Investor Presentation - FINAL.pdf
Q4 2023 Quarterly Investor Presentation - FINAL.pdfQ4 2023 Quarterly Investor Presentation - FINAL.pdf
Q4 2023 Quarterly Investor Presentation - FINAL.pdfTejal81
 
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...Neo4j
 
Large Language Models and Applications in Healthcare
Large Language Models and Applications in HealthcareLarge Language Models and Applications in Healthcare
Large Language Models and Applications in HealthcareAsma Ben Abacha
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriSafe Software
 
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)Jay Zhao
 
Enterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewEnterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewAshraf Fouad
 
Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...
Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...
Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...2toLead Limited
 
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Umar Saif
 
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...ShapeBlue
 
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Product School
 
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlueCloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlueShapeBlue
 
New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024ThousandEyes
 
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...Product School
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxNeo4j
 
Battle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsBattle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsEvangelia Mitsopoulou
 

Recently uploaded (20)

Roundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdfRoundtable_-_API_Research__Testing_Tools.pdf
Roundtable_-_API_Research__Testing_Tools.pdf
 
Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...Transcript: Trending now: Book subjects on the move in the Canadian market - ...
Transcript: Trending now: Book subjects on the move in the Canadian market - ...
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human Justice
 
iOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingeriOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostinger
 
In sharing we trust. Taking advantage of a diverse consortium to build a tran...
In sharing we trust. Taking advantage of a diverse consortium to build a tran...In sharing we trust. Taking advantage of a diverse consortium to build a tran...
In sharing we trust. Taking advantage of a diverse consortium to build a tran...
 
Q4 2023 Quarterly Investor Presentation - FINAL.pdf
Q4 2023 Quarterly Investor Presentation - FINAL.pdfQ4 2023 Quarterly Investor Presentation - FINAL.pdf
Q4 2023 Quarterly Investor Presentation - FINAL.pdf
 
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
 
Large Language Models and Applications in Healthcare
Large Language Models and Applications in HealthcareLarge Language Models and Applications in Healthcare
Large Language Models and Applications in Healthcare
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
Leonis Insights: The State of AI (7 trends for 2023 and 7 predictions for 2024)
 
Enterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book ReviewEnterprise Architecture As Strategy - Book Review
Enterprise Architecture As Strategy - Book Review
 
Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...
Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...
Microsoft x 2toLead Webinar Session 1 - How Employee Communication and Connec...
 
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
 
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
 
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
 
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlueCloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
 
New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024New ThousandEyes Product Features and Release Highlights: February 2024
New ThousandEyes Product Features and Release Highlights: February 2024
 
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
Harnessing the Power of GenAI for Exceptional Product Outcomes by Booking.com...
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
 
Battle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsBattle of React State Managers in frontend applications
Battle of React State Managers in frontend applications
 

Journey of Migrating 1 Million Presto Queries - Presto Webinar 2020

  • 1. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Taro L. Saito Arm Treasure Data July 16th, 2020 Starburst Presto Webinar Presto At Arm Treasure Data: Journey of Migrating 1 Million Presto Queries 1
  • 2. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. About Me: Taro L. Saito (Leo) 2 ● Principal Software Engineer of Arm Treasure Data ● Ph.D. of Computer Science ● Active OSS developer ● Created Presto As A Service ● 6 years since 2014
  • 3. Confidential © Arm 2017Confidential © Arm 2017Confidential © Arm 2017 Thank You! Danke! Merci! 谢谢! ありがとう! Gracias! Kiitos!400+ Customers Founded in 2011 Raised $54M Security Acquired by Arm / Softbank 2018 Arm Treasure Data
  • 4. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Customer Data Platform (CDP) Solution 4
  • 5. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Arm Treasure Data: Ready-to-Use Cloud Data Platform From Day 1 5 Logs Device Data Batch Data PlazmaDB Table Schema Data Collection Cloud Storage Distributed Data Processing Jobs Job Management SQL Editor Scheduler Workflows Machine Learning Treasure Data OSS Third Party OSS Data
  • 6. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Data Connectors And Integrations 6
  • 7. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Our Contributions to Presto 7
  • 8. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Presto Usage Stats (2020): 1 Million Queries / Day ● November 2013 ● Presto was open-sourced ● July 2014 ● Presto service launch ● 2015 ● 20,000 queries / day ● 2019 ● 3,500 users ● 600,000 queries / day ● 2020 ● 5,000+ users ● 1,000,000 queries / day ● 100 trillion rows processed / day ■ 1.2 billion rows / sec. 8
  • 9. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. A Challenge: Every Presto Upgrade Is A Big Migration ● 2014 ● Presto 0.64 ~ ● 2015 ● Presto 0.113 ~ ● 2016 ● Presto 0.152 ~ ● 2017 ● Presto 0.178 ~ ● 2018 ● Presto 0.205 ■ Used only this version ● 2019-2020 ● Presto 0.205 to 317 ● Only 1 customer is remaining at Presto 0.205 (July 2020) ● Migration took almost 1 year 9 Very Frequent Every 1-2 weeks Less Frequent Presto Upgrade Storage Upgrade (e.g., format change) Customer Queries Customer Queries Migration requires human actions (e.g., Fixing incompatible queries, scheduling migrations, performance tuning, etc.) Limiting Factor
  • 10. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Migration: Why Presto 317? ● 2018 ● We were using Presto 0.205 ● 2019 ● Presto Software Foundation (prestosql) was founded => Presto 300 ● Tested Presto 311, 316, and 317 (The latest version on Aug 2019) ● Presto 318 had an incorrect query result bug, so we chose 317 ■ The bug was fixed in 319 ● Criteria to choose a Presto version ● Less query correctness issues ● Stability ■ Less major GCs and memory related bugs ● Less number of incompatible changes ■ SQL syntax changes, query behavior changes. 10
  • 11. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Presto Conference Tokyo 2019 (July 11, 2019) ● Held At Arm Treasure Data Tokyo Office (6F) ● Invited 3 Presto Creators (Martin, Dain, and David) ● Conference Report: https://prestosql.io/blog/2019/07/11/report-for-presto-conference-tokyo.html ● Presto: Current And Future ● How to Read the Source Code of Presto, etc. ● After this conference, we started Presto migration ● Presto 317 was released on Aug 2019 11
  • 12. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Presto Queries Form Complex Dependency Graphs ● A real customer example ● 2000 query patterns, 5000 queries in a day ● Boxes: input tables, edges: queries, red circles: external targets ● 12
  • 13. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Collecting Query Logs ● Collecting all Presto query logs since 2015 ● Query statements, performance statistics, logs, etc. ■ QueryCreation, QueryCompletion, SplitCompletion events ● TD is one of the biggest users of TD itself ● Using Presto for optimizing Presto 13 Logs User Query Collect Query Logs Analyze Query Logs Machine Learning Query Optimization Optimize System
  • 14. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Usages of Query Logs ● Estimate Query Service-Level Objectives (SLOs) ● Query Running Time ● Query Resource Usage ● Replay Queries for Simulation ● Creating Customer-Specific Benchmark ● Problem investigation ● Finding heavy queries, performance issues, job failures, etc. 14
  • 15. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Monitoring Service Level Objectives (SLOs) ● Estimating query finishing time using the historical query stats of last 7 days ● Using p50 and MAD (median absolute deviation) ● SLO Violation: Query running time > p50 + 3MAD 15
  • 16. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Query Clustering ● Query Signature 16
  • 17. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Queries With The Same Query Signature 17
  • 18. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Query Clusters ● Using query signatures as keys for clustering queries ● For creating better clusters: ● Include not only query structures, but also input and output table names ● Replace (table_name)_(date strings) patterns into (table_name)_N 18
  • 19. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Query Simulator: Testing Customer-Specific Queries ● Testing all 1M query / day = 30M queries / month is impractical ● Standard benchmarks like TPC-H/TPC-DS don’t have sufficient query coverage. ● Clustering queries ● Reduced the number of queries into 100,000 query patterns ● Reducing query size by narrowing input table data ranges ● Running production-scale queries can be expensive 19 User QueryUser QueryUser QueryUser QueryUser QueryUser Query 30,000,000 queries clustering Query SigQuery SigQuery SigQuery Sig minimize Small QuerySmall QuerySmall QuerySmall Query 100,000 query patterns 100,000 small queries simulate queries simulation results and stats
  • 20. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Query Coverage ● Query Pattern Coverage ● SQL syntax compatibility test coverage: Almost 100% ● Reporting necessary SQL fixes in advance to customers 20
  • 21. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Testing SELECT Queries ● Problems: ● Slow query result transfer in JSON from Presto Coordinator to Client (e.g., presto-jdbc) ● Employees can’t see customer data ● Our workaround ● Run SELECT queries in two steps: ■ Check the result schema with: ○ SELECT * from (original query) limit 0 ■ Rewrite the original SQL to compute checksum (order-insensitive) ○ SELECT checksum(col_1), checksum(col_2), … from (original query) ● Report only the final checksum ■ Presto consumes all result rows (no row transfer) ● Compare only the checksum between Presto versions 21
  • 22. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Scheduling Migration ● Reporting All Breaking Changes In Advance ● Incompatible SQL queries ● Customers needed to modify their queries and workflows ● Providing Grace Period for Migration ● Supported early testing of the next Presto version ● Using a magic comment within SQL for specifying a Presto version to use ● Not only our customers, but also their partner companies are involved ● Partner companies might be writing SQL queries on behalf of customers ● Consider scheduling migrations of 400+ customers and their partners! 22
  • 23. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. No Migration At Holiday and Event Seasons ● US ● July 4th ● November ■ Thanksgiving, Black Friday, Cyber Monday ● Japan ● May 3 - May 5 ■ Golden Week ● Week of August 15th ■ Obon: Japanese tradition to visit home town ● Long winter holiday ■ Weeks around the year-end and new-year start ● We also need to avoid changes at the end of month ● Most of the companies have important monthly jobs ■ KPI, monthly revenue reporting, etc. 23
  • 24. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Queries Difficult To Simulate ● Our query simulation coverage is still around 80% ● Queries involving temporary tables ● The data might be missing ● Queries updating production tables ● DELETE (TD supports record-level deletion) ■ Need to create a copy of the data in a secure manner ● CREATE TABLE AS (CTAS), INSERT INTO ■ Using a hidden database inside customer’s account so as not to leak any data outside the customer account. ● Long running queries ● Some queries keep running for more than 1 day ■ 2 days are the current maximum 24
  • 25. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Testing Workflow Queries ● Often uses a lot of temporary tables ● Need to maintain snapshots of partitions and table schema ● Workflow involves varieties of tasks ● Presto, Hive queries, Python, Spark codes, etc. ● Importing data from external sources, etc. ● There is almost no way to test the entire workflows 25
  • 26. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Integration Testing ● Model class changes required presto client upgrades ● Query stats model class changes ● Query plan model class changes ● Job queue and reporting system can be broken ● presto-client-ruby ● Embedding multiple model classes of both Presto 0.205 and 317 ● Query Error Retry Pattern Changes ● Not only Presto’s error code, we are retrying queries using error message patterns ■ e.g., OOM, Java exception message patterns, system, JVM crashes, etc. ● If the error message changes, we need to add more error patterns. 26
  • 27. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Production Scale Query Testing Is Also Necessary ● Example: UDF Performance Regression ● Example: TD_TIME_PARSE(time_string) ■ Parse date time strings and convert them into unixtime ● We introduced a date string cache for efficiency, but the cache size was too small for a query calling this UDF billions of times for wide-range of data ● The query slowed down from 3 minutes -> 40 minutes ■ Because of too many cache misses ● Fixed this issue by increasing the cache size ● A Lesson ● Even if 99.9999% of queries are ok, it’s not 100% yet ■ 1 query out of 1,000,000 queries = 0.0001% ■ We still need to check data-specific performance characteristics 27
  • 28. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Testing Optimization Rules ● After migrating to Presto 317, we noticed some optimization was not working ● Time-index pushdown (TD-Specific) ● Extract condition related to time column, and use this tuple domain for removing unnecessary partition scans ● Eager evaluation of nested UDFs (CURRENT_DATE, CURRENT_TIMESTAMP) are necessary to extract time ranges ● However, Presto 317 does not evaluate cast wraps in advance ■ => Failed to extract the time range and resulted in the full table scans ● We added check rules to see how many partitions are scanned 28
  • 29. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Testing Presto-PostgreSQL connector ● DataTank service ● Hosted PostgreSQL inside Treasure Data ● Accessing TD and PostgreSQL’s data using Presto ● Problem ● There is no way to run any simulation because PostgreSQL doesn’t isolate compute and storage. Any test query affects the other queries. ● Some customers are using this connector a lot ● An issue: Connecting to PostgreSQL more than hundreds of times inside a single Presto query ● Caused connection failures in Presto 317 ■ We need to introduce retry with Jittering for acquiring JDBC connections ○ https://github.com/prestosql/presto/pull/3125 ● Workaround: Increased the max connection parameter of PostgreSQL 29
  • 30. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Operating Multiple Presto Versions ● We should reduce the version differences as much as possible ● For cost-efficiency (Multi-tenancy) ● And also, because no cloud has unlimited scale. ● Our Deployment: Blue-Green Deployment ● Preparing two versions of clusters, and gradually migrating customers from the old version to the new version ● If we have 100-node cluster, we need 200 instances for blue-green deployment ● We encountered a lot of AWS insufficient capacity errors ● We are using hundreds of r4.8xlarge (240GB, 32 core) instances ● Depending on the time in the day, we can’t create 100+ instances ● We also wanted to upgrade to r5.8xlarge (next generation hypervisor), but failed because AWS doesn’t have so many r5 instances in US/JP/EU regions... 30
  • 31. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Configuration Changes ● Good ● Presto 317 no longer uses the reserved memory pool ■ If the reserved pool is used, only a single query can use large memory and blocks the other queries ● General performance improvement and new UDFs (e.g., format function) ● Several unexpected bugs and behaviors ● Cannot list/kill QUEUED queries. We noticed it after the upgrade. ● Backported fixes from Presto 319 ● More strict queuing and resource control ● Some customer experienced longer queued time of queries ● We needed to perform peak-time simulation to see the number of concurrent queries (CQs) and their queued time changes 31
  • 32. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Engineering Challenges ● Debugging Asynchronous Calls ● S3 read and write ● Transaction processing ● Async call is necessary for maximizing the I/O throughput, but reduce the code readability ● A Race Condition Bug ● Regular updates ■ Read S3 -> uploading new partitions to S3 -> registering partition entries to PlazmaDB (PostgreSQL) -> committing new partitions ● We had a race condition issue around Java’s Future task management involving RDBMS transaction, and the above update order was not ensured ■ Failed to upload some partitions (partially missing update results) ● Hard to find such a bug with peer-review, and it was hard to reproduce ■ We need to add random-delays to such critical paths 32
  • 33. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Next Presto Migration Plans ● Service limitation changes ● Max query running time: unlimited -> 2 days -> 4 hours (ideal) ■ For enabling quick JVM/cluster refresh and load balancing ● Changing query max memory size for reducing the number of query_max_memory errors ● Such limitation changes require query log analysis and simulations ● Next Migration Plans ● Upgrading from Java 8 to Java 11 ● Using Gravition2 instances (Arm64 CPU) ● Presto 338 or later ● Active Development ● Presto Query Simulator ■ Registering query sets, simulation, and reporting ● Customer-specific benchmark generator from query logs 33
  • 34. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Arm Graviton 2 ● Arm64 instances on AWS (r6g, m6g, etc.), available since June 2020 ● Experimental Results By Kai Sasaki ○ https://blog.treasuredata.com/blog/2020/03/27/high-performance-sql-aws-gravito n2-benchmarks-with-presto-and-arm-treasure-data-cdp/ 30 Percent Faster, 20 Percent Cheaper, Up to 50 Percent Better ROI 34
  • 35. Copyright 1995-2020 Arm Limited (or its affiliates). All rights reserved. Summary ● Migration of 1,000,000 queries revealed many challenges including: ● Compatibility tests ● Performance regression tests ■ Optimizer, production-scale and peak-time tests are necessary ■ How to find 0.0001% of regression? ● Difficult types of queries to test ■ Temporary tables, workflows, presto-postgresql connector ● Collecting query logs is important ● Without data, no migration can be supported ● Query clustering, SLO management, etc. 35