SlideShare a Scribd company logo
ᄆᄆਫ෸ᦇᓒଘ‫ݣ‬ຝ຅Өਫ᪢
The Application and Architecture of the Real-Time Computing Platform in DiDi
‫ݪل‬ғᄆᄆ‫ڊ‬ᤈ
Company: DiDi Chuxing
ᘳ֖ғದ๞Ꮈᑪާ
Title Director of BigData Infrastructure Department
ᄍᦖᘏғᗔ๫
Lecturer Luo Li
Agenda
Ø ᙧว݊ሿᇫ
Ø ਫ෸ᥢ‫܃ڞ‬ᯈਫ᪢
Ø 6WUHDP64/Օᕨ
Ø ਫ෸ᦇᓒଘ‫ݣ‬໛ຝ
Ø ௛ᕮ޾ᥢ‫ښ‬
Ø Background
Ø The CEP practice
Ø StreamSQL
Ø Stream Platform Infrastructure
Ø Future plans
ӱ‫ۓ‬ᇙᅩ
characteristic of business
Ø ᄆᄆ๜᫝ฎӞӻਫ෸Իฃ୚ක
Ø ᄆᄆጱහഝ޾࣋วॠᆐฎਫ෸ጱ
ள᫣ ӫ᫣ սՁ ‫ڊ‬ᑄ᫣
᨟܏᫣ …
෭பහഝ
ईᅩහഝ
Իฃහഝ
᫥ᭃහഝ
ਫ෸ᦇᓒ
ਫ෸᯻ᵞ
Ø DiDi is a realtime Trade Engine
Ø Didi's data and scenes are naturally real-time
Fast Cars
Special
Cars
Shared
Cars
Taxi
/X[XU
&DUV
…
Log Data
Collection
point Data
Transaction
Data
Trace Data
Real-time process
Real-time ETC & Collection
ၞᦇᓒದ๞ᄍᬰ
ᒫӞᴤྦྷғӱ‫ۓ‬ොᛔୌੜᵞᗭ
ᒫԫᴤྦྷғᵞӾୗय़ᵞᗭ̵ଘ‫ݣ‬۸
ᒫӣᴤྦྷғၞᦇᓒ୏‫ݎ‬64/۸
Storm/Jstorm/
Spark/Samza
ӱ‫ۓ‬ොᛔୌੜᵞᗭ
Spark Streaming
๐‫ۓ‬۸ଘ‫ݣ‬۸
Flink Streaming
ಭ‫ف‬ଫአ
चԭFlink CEPጱਫ෸ᥢ‫܃ڞ‬
ᯈ๐‫ۓ‬Ӥᕚ
चԭFlinkጱStreamSQLӤᕚ
Didi’s Real-time develop steps
First Step A lot of some Clusters
Second Step Single Big Shared Cluster
Third Step SQL On Stream process
Storm/Jstorm/
Spark/Samza
some clusters
Spark Streaming
Flink Streaming
Flink CEP
StreamSQL On Flink
ଫአ࣋ว Scenarios
- --
Ø Ø HBase
Druid ES
Ø Ø
Ø ਫ෸‫̵ڭݎ‬ਫ෸ᷚഴ̵
ਫ෸୑ଉ༄ၥᒵ
Ø හഝ՗ၾ௳ᴚ‫کڜ‬HBase
Druid ESᒵਂ‫ؙ‬
Ø Իฃፊഴ̵๐‫ۓ‬ፊ
ഴ̵ૡ‫ܔ‬ፊഴᒵ
Ø ਫ෸ԙਮᇙ஄̵‫ݪ‬๢ᇙ஄̵
ᦈ‫ܔ‬ᇙ஄ᒵ
ଫአᥢཛྷ
๐‫ۓ‬ӱ‫ۓ‬ᕚ ᵞᗭᥢཛྷ ၞᦇᓒձ‫ۓ‬හ ྯ෭॒ቘහഝᰁ
Data Scale
Applications Cluster Scale Streaming Jobs Data Scale per day
ਫ෸ᥢ‫܃ڞ‬ᯈҁCEP҂ਫ᪢
The CEP practice
ਫ෸ᥢ‫܃ڞ‬ᯈҁCEP҂
ӻ௔۸០ᲀ
Ø ʼn຤उ૱ԙਮٛု‫ݸ‬ᑁဌํӥ‫ܔ‬Ŋ
Ø ʼn຤उ૱ӥ‫ݸܔ‬ᑁဌՈള‫ܔ‬Ŋ
Ø શ꧊ᥢ‫ڞ‬
Pattern
.begin(s1)
.where(key=bubble  city=166)
.notFollowedBy(s2)
.where(key=order_send  city = 166)
.timeEnd(s3)
.timeShiftMore(s1, Time.ms(5000))
୑ଉૡ‫ܔ‬༄ၥ
Ø ᦈ‫ܔ‬ᇫாᳩ෸ᳵ‫ړ‬ᰦ
؊ኸࣁ‫ݪ‬๢૪ಶ‫ܔ‬Ŋ
Øʼnᦈ‫ܔ‬ᇫாᳩ෸ᳵੜ෸
؊ኸࣁ୏তᦇᩇŊ
Øʼn೪᫣ӨGRVᦈ‫ܔ‬ᇫாӧӞᛘ‫ݪ‬๢૪ಶ‫ܔ‬Ŋ
හഝრ ᶼ॒ቘ ‫܃‬ᯈ
DD-CEP
ᥢ‫ڜڞ‬ᤒ$3,
ਫ෸ᥢ‫܃ڞ‬ᯈ๐‫ۓ‬
CEP‫ۑ‬ᚆ	௔ᚆදᬰ
StreamSQL
StreamSQLୌᦡ
   
ӿ੄8') ‫݌‬ၞ-2,1 ᖌᤒ-2,1 64/ ,'(

ಘ઀''/
ಘ઀''/
Ø ಑᭗ᄆᄆٖ᮱Ԇၞၾ௳ᴚ‫޾ڜ‬ਫ෸ਂ‫ؙ‬
Ø ᭗ᬦಘ઀''/᧍‫ݙ‬ൈᬿ6RXUFH޾6LQN
Ø ٖᗝ-621̵ຽ‫ٵ‬෭ப̵3%໒ୗᥴຉ
Kafka
MQ
StreamSQL
Kafka
Druid
HBase
Phoenix
ElaticSearch
MYSQL
MQ
KVStore
UDFय़ଏಘ઀





UDFय़ଏಘ઀
ਁᒧԀ॒ቘ‫ڍ‬හ
෭๗॒ቘ‫ڍ‬හ
Map॒ቘ‫ڍ‬හ
ᑮᳵ֖ᗝ‫ڍ‬හ
JSON॒ቘ‫ڍ‬හ
ඪ೮चԭTTLጱ‫݌‬ၞJoin
topic1
topic2
source1
source2
topic3
JoinOperator
TTLState1 TTLState2
ၾ௳ᴚ‫ڜ‬ StreamSQL ၾ௳ᴚ‫ڜ‬
Sink
Ø inner join
Ø Left join
Ø Full join
source2 topic2
JoinOperator
TTLState Cache
ၾ௳ᴚ‫ڜ‬ ၾ௳ᴚ‫ڜ‬
topic1
୑ྍ
IO
Sink
ᖌᤒਂ‫ؙ‬
HBase/MySQL)
StreamSQL
ᖌᤒJOIN
ØStreamSQL Editor
ØSQLཛྷ຃
ØUDF‫ڍ‬හ᧔ก
Ø᧍ဩ༄ၥ
ØDEBUG
Øᇇ๜ᓕቘ
SQL IDE
ਫ෸ᦇᓒଘ‫ݣ‬ຝ຅
Stream Platform Infrastructure
ਫ෸ᦇᓒଘ‫ݣ‬ຝ຅
ଘ‫੶ݣ‬
୚ක੶
SQL IDE WEB IDE
ձ‫ۓ‬
ᓕഴ
᦬ෙ
֛ᔮ
ᤅᖭ
‫ى‬ᔮ
ፊഴ
ಸᦄ
ग़ᑄಁ
֛ᔮ
ਫ෸
ਂ‫ؙ‬
Hadoop
ବଷ ‫૲ړ‬ୗ෈կᔮᕹғHDFSᕹӞᩒრ᧣ଶғYARN
Flink Streaming
StreamSQL
ਫ෸ᥢ‫܃ڞ‬ᯈ
ҁCEP
DataStream
ӣᐿၞᦇᓒ୏‫ݎ‬ཛྷୗ
๜ࣈIDE୏‫ݎ‬
Web IDE୏‫ݎ‬
SQL IDE୏‫ݎ‬
Web۸ձ‫ۓ‬ᓕഴ
ၾᴻਮಁ๢
൉‫ف׀‬ᳪཛྷ຃
ٖᗝ݇හ᧣ս
ٖᗝ೰ຽӤಸ
Flink App
Kafka
Druid
ᤅᖭ‫ى‬ᔮ
୊෸༄ၥ
೰ຽፊഴ
෭ப༄ᔱ
ES
ᤅᖭӤಸ
೰ຽӤಸ
෭பӤಸ
හഝᤅᖭ‫ى‬ᔮ
Kafka
ᤅᖭ‫ى‬ᔮ
୊෸༄ၥ
೰ຽፊഴ
෭ப༄ᔱ
ᤅᖭӤಸ
೰ຽӤಸ
෭பӤಸ
Druid
ES
Flink App
Ø ᇿᒈୌᦡਫ෸ᦇᓒᵞᗭ҅᭿‫ع‬ᐶᕚձ‫ۓ‬ଗಟ
Ø ਫ෸ᵞᗭੱᰁᶌᬪӱ‫҅ۓ‬ᴳ֗ᗑᕶ෸୊
Ø ਧ‫᧣ګ‬ଶ࢏
Ø ᧣ଶ࣐ᤍ
Ø चԭ38᧣ଶ
Ø चԭ1RGH /DEHO޾*URXS‫݌‬᯿ᵍᐶ๢‫ګ‬
Ø ग़๢಄᮱ᗟ
Router
ਫ෸᧣ଶ࢏
RM1
NM1 NM2 NM3
Lable1 Lable1 Lable2
HDFS1
ਫ෸᧣ଶ࢏
RM2
NM4 NM5 NM6
Lable3 Lable4 Lable4
HDFS1
DC1 DC2
Hadoopବଷ
௛ᕮӨᥢ‫ښ‬
Future plans
௛ᕮ
Ø 6WUHDP64/ य़ଏᴳ֗ၞᦇᓒ୏‫ݎ‬ᳪད
Ø ਫ෸ᥢ‫܃ڞ‬ᯈҁ(3҂ ࣁӻ௔۸០ᲀ̵ਫ෸ᷚഴᒵᶾऒ៧ࣈ
Ø ਫ෸ᦇᓒଘ‫ݣ‬ Ӟᒊୗၞᦇᓒ୏‫ݎ‬ଘ‫ݣ‬
Summary
Ø StreamSQL makes the Streaming develop much easier
Ø Real-time CEP very useful in the risk control  Personalized marketing
Ø A Big Shared Stream Processing platform

More Related Content

What's hot

Asd 2015
Asd 2015Asd 2015
Asd 2015
Rim Moussa
 
SQream DB - Bigger Data On GPUs: Approaches, Challenges, Successes
SQream DB - Bigger Data On GPUs: Approaches, Challenges, SuccessesSQream DB - Bigger Data On GPUs: Approaches, Challenges, Successes
SQream DB - Bigger Data On GPUs: Approaches, Challenges, Successes
Arnon Shimoni
 
Redbooks with live links 2010 12-06
Redbooks with live links 2010 12-06Redbooks with live links 2010 12-06
Redbooks with live links 2010 12-06
Willie Favero
 
Maximize Greenplum For Any Use Cases Decoupling Compute and Storage - Greenpl...
Maximize Greenplum For Any Use Cases Decoupling Compute and Storage - Greenpl...Maximize Greenplum For Any Use Cases Decoupling Compute and Storage - Greenpl...
Maximize Greenplum For Any Use Cases Decoupling Compute and Storage - Greenpl...
VMware Tanzu
 
Microsoft cosmos
Microsoft cosmosMicrosoft cosmos
Microsoft cosmos
Karthik Murugesan
 
Data Science Across Data Sources with Apache Arrow
Data Science Across Data Sources with Apache ArrowData Science Across Data Sources with Apache Arrow
Data Science Across Data Sources with Apache Arrow
Databricks
 
Virtualized Platform Migration On A Validated System
Virtualized Platform Migration On A Validated SystemVirtualized Platform Migration On A Validated System
Virtualized Platform Migration On A Validated System
gazdagf
 
Benchmarking data warehouse systems in the cloud: new requirements & new metrics
Benchmarking data warehouse systems in the cloud: new requirements & new metricsBenchmarking data warehouse systems in the cloud: new requirements & new metrics
Benchmarking data warehouse systems in the cloud: new requirements & new metrics
Rim Moussa
 
Unlocking Your Hadoop Data with Apache Spark and CDH5
Unlocking Your Hadoop Data with Apache Spark and CDH5Unlocking Your Hadoop Data with Apache Spark and CDH5
Unlocking Your Hadoop Data with Apache Spark and CDH5
SAP Concur
 
How GPUs Enable XVA Pricing and Risk Calculations for Risk Aggregation
How GPUs Enable XVA Pricing and Risk Calculations for Risk AggregationHow GPUs Enable XVA Pricing and Risk Calculations for Risk Aggregation
How GPUs Enable XVA Pricing and Risk Calculations for Risk Aggregation
Kinetica
 
Multidimensional DB design, revolving TPC-H benchmark into OLAP bench
Multidimensional DB design, revolving TPC-H benchmark into OLAP benchMultidimensional DB design, revolving TPC-H benchmark into OLAP bench
Multidimensional DB design, revolving TPC-H benchmark into OLAP bench
Rim Moussa
 
Generative Hyperloop Design: Managing Massively Scaled Simulations Focused on...
Generative Hyperloop Design: Managing Massively Scaled Simulations Focused on...Generative Hyperloop Design: Managing Massively Scaled Simulations Focused on...
Generative Hyperloop Design: Managing Massively Scaled Simulations Focused on...
Databricks
 
Data Orchestration for AI, Big Data, and Cloud
Data Orchestration for AI, Big Data, and CloudData Orchestration for AI, Big Data, and Cloud
Data Orchestration for AI, Big Data, and Cloud
Alluxio, Inc.
 
Real-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache PinotReal-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache Pinot
Xiang Fu
 
The Future of Sharding
The Future of ShardingThe Future of Sharding
The Future of Sharding
EDB
 
Consolidate Your Technical Debt With Spark Data Sources -Tools and Techniques...
Consolidate Your Technical Debt With Spark Data Sources -Tools and Techniques...Consolidate Your Technical Debt With Spark Data Sources -Tools and Techniques...
Consolidate Your Technical Debt With Spark Data Sources -Tools and Techniques...
Databricks
 
Bigdata Machine Learning Platform
Bigdata Machine Learning PlatformBigdata Machine Learning Platform
Bigdata Machine Learning Platform
Mk Kim
 
CEP - simplified streaming architecture - Strata Singapore 2016
CEP - simplified streaming architecture - Strata Singapore 2016CEP - simplified streaming architecture - Strata Singapore 2016
CEP - simplified streaming architecture - Strata Singapore 2016
Mathieu Dumoulin
 
BigData Hadoop
BigData Hadoop BigData Hadoop
BigData Hadoop
Kumari Surabhi
 
2017 DB Trends for Powering Real-Time Systems of Engagement
2017 DB Trends for Powering Real-Time Systems of Engagement2017 DB Trends for Powering Real-Time Systems of Engagement
2017 DB Trends for Powering Real-Time Systems of Engagement
Aerospike, Inc.
 

What's hot (20)

Asd 2015
Asd 2015Asd 2015
Asd 2015
 
SQream DB - Bigger Data On GPUs: Approaches, Challenges, Successes
SQream DB - Bigger Data On GPUs: Approaches, Challenges, SuccessesSQream DB - Bigger Data On GPUs: Approaches, Challenges, Successes
SQream DB - Bigger Data On GPUs: Approaches, Challenges, Successes
 
Redbooks with live links 2010 12-06
Redbooks with live links 2010 12-06Redbooks with live links 2010 12-06
Redbooks with live links 2010 12-06
 
Maximize Greenplum For Any Use Cases Decoupling Compute and Storage - Greenpl...
Maximize Greenplum For Any Use Cases Decoupling Compute and Storage - Greenpl...Maximize Greenplum For Any Use Cases Decoupling Compute and Storage - Greenpl...
Maximize Greenplum For Any Use Cases Decoupling Compute and Storage - Greenpl...
 
Microsoft cosmos
Microsoft cosmosMicrosoft cosmos
Microsoft cosmos
 
Data Science Across Data Sources with Apache Arrow
Data Science Across Data Sources with Apache ArrowData Science Across Data Sources with Apache Arrow
Data Science Across Data Sources with Apache Arrow
 
Virtualized Platform Migration On A Validated System
Virtualized Platform Migration On A Validated SystemVirtualized Platform Migration On A Validated System
Virtualized Platform Migration On A Validated System
 
Benchmarking data warehouse systems in the cloud: new requirements & new metrics
Benchmarking data warehouse systems in the cloud: new requirements & new metricsBenchmarking data warehouse systems in the cloud: new requirements & new metrics
Benchmarking data warehouse systems in the cloud: new requirements & new metrics
 
Unlocking Your Hadoop Data with Apache Spark and CDH5
Unlocking Your Hadoop Data with Apache Spark and CDH5Unlocking Your Hadoop Data with Apache Spark and CDH5
Unlocking Your Hadoop Data with Apache Spark and CDH5
 
How GPUs Enable XVA Pricing and Risk Calculations for Risk Aggregation
How GPUs Enable XVA Pricing and Risk Calculations for Risk AggregationHow GPUs Enable XVA Pricing and Risk Calculations for Risk Aggregation
How GPUs Enable XVA Pricing and Risk Calculations for Risk Aggregation
 
Multidimensional DB design, revolving TPC-H benchmark into OLAP bench
Multidimensional DB design, revolving TPC-H benchmark into OLAP benchMultidimensional DB design, revolving TPC-H benchmark into OLAP bench
Multidimensional DB design, revolving TPC-H benchmark into OLAP bench
 
Generative Hyperloop Design: Managing Massively Scaled Simulations Focused on...
Generative Hyperloop Design: Managing Massively Scaled Simulations Focused on...Generative Hyperloop Design: Managing Massively Scaled Simulations Focused on...
Generative Hyperloop Design: Managing Massively Scaled Simulations Focused on...
 
Data Orchestration for AI, Big Data, and Cloud
Data Orchestration for AI, Big Data, and CloudData Orchestration for AI, Big Data, and Cloud
Data Orchestration for AI, Big Data, and Cloud
 
Real-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache PinotReal-time Analytics with Trino and Apache Pinot
Real-time Analytics with Trino and Apache Pinot
 
The Future of Sharding
The Future of ShardingThe Future of Sharding
The Future of Sharding
 
Consolidate Your Technical Debt With Spark Data Sources -Tools and Techniques...
Consolidate Your Technical Debt With Spark Data Sources -Tools and Techniques...Consolidate Your Technical Debt With Spark Data Sources -Tools and Techniques...
Consolidate Your Technical Debt With Spark Data Sources -Tools and Techniques...
 
Bigdata Machine Learning Platform
Bigdata Machine Learning PlatformBigdata Machine Learning Platform
Bigdata Machine Learning Platform
 
CEP - simplified streaming architecture - Strata Singapore 2016
CEP - simplified streaming architecture - Strata Singapore 2016CEP - simplified streaming architecture - Strata Singapore 2016
CEP - simplified streaming architecture - Strata Singapore 2016
 
BigData Hadoop
BigData Hadoop BigData Hadoop
BigData Hadoop
 
2017 DB Trends for Powering Real-Time Systems of Engagement
2017 DB Trends for Powering Real-Time Systems of Engagement2017 DB Trends for Powering Real-Time Systems of Engagement
2017 DB Trends for Powering Real-Time Systems of Engagement
 

Similar to The application and architecture of the real-time computing platform in di di-flink_forward_2018

Data Engineer's Lunch #81: Reverse ETL Tools for Modern Data Platforms
Data Engineer's Lunch #81: Reverse ETL Tools for Modern Data PlatformsData Engineer's Lunch #81: Reverse ETL Tools for Modern Data Platforms
Data Engineer's Lunch #81: Reverse ETL Tools for Modern Data Platforms
Anant Corporation
 
Developing Enterprise Consciousness: Building Modern Open Data Platforms
Developing Enterprise Consciousness: Building Modern Open Data PlatformsDeveloping Enterprise Consciousness: Building Modern Open Data Platforms
Developing Enterprise Consciousness: Building Modern Open Data Platforms
ScyllaDB
 
Highway to heaven - Microservices Meetup Munich
Highway to heaven - Microservices Meetup MunichHighway to heaven - Microservices Meetup Munich
Highway to heaven - Microservices Meetup Munich
Christian Deger
 
Apache Flink Adoption at Shopify
Apache Flink Adoption at ShopifyApache Flink Adoption at Shopify
Apache Flink Adoption at Shopify
Yaroslav Tkachenko
 
Preventative Maintenance of Robots in Automotive Industry
Preventative Maintenance of Robots in Automotive IndustryPreventative Maintenance of Robots in Automotive Industry
Preventative Maintenance of Robots in Automotive Industry
DataWorks Summit/Hadoop Summit
 
Running Data Platforms Like Products
Running Data Platforms Like ProductsRunning Data Platforms Like Products
Running Data Platforms Like Products
VMware Tanzu
 
PayPal datalake journey | teradata - edge of next | san diego | 2017 october ...
PayPal datalake journey | teradata - edge of next | san diego | 2017 october ...PayPal datalake journey | teradata - edge of next | san diego | 2017 october ...
PayPal datalake journey | teradata - edge of next | san diego | 2017 october ...
Deepak Chandramouli
 
How to design and implement a data ops architecture with sdc and gcp
How to design and implement a data ops architecture with sdc and gcpHow to design and implement a data ops architecture with sdc and gcp
How to design and implement a data ops architecture with sdc and gcp
Joseph Arriola
 
Building Blockchain Apps: Intro to Dash Platform
Building Blockchain Apps: Intro to Dash PlatformBuilding Blockchain Apps: Intro to Dash Platform
Building Blockchain Apps: Intro to Dash Platform
Product School
 
Geode: Much More Than a Cache
Geode: Much More Than a CacheGeode: Much More Than a Cache
Geode: Much More Than a Cache
VMware Tanzu
 
Overview and Walkthrough of the Application Programming Model with SAP Cloud ...
Overview and Walkthrough of the Application Programming Model with SAP Cloud ...Overview and Walkthrough of the Application Programming Model with SAP Cloud ...
Overview and Walkthrough of the Application Programming Model with SAP Cloud ...
SAP Cloud Platform
 
Next generation business automation with the red hat decision manager and red...
Next generation business automation with the red hat decision manager and red...Next generation business automation with the red hat decision manager and red...
Next generation business automation with the red hat decision manager and red...
Masahiko Umeno
 
Enterprise Application Migration
Enterprise Application MigrationEnterprise Application Migration
Enterprise Application Migration
VMware Tanzu
 
Sourav_Giri_Resume_2015
Sourav_Giri_Resume_2015Sourav_Giri_Resume_2015
Sourav_Giri_Resume_2015
sourav giri
 
IoT Scale Event-Stream Processing for Connected Fleet at Penske
IoT Scale Event-Stream Processing for Connected Fleet at PenskeIoT Scale Event-Stream Processing for Connected Fleet at Penske
IoT Scale Event-Stream Processing for Connected Fleet at Penske
VMware Tanzu
 
How to use hybrid cloud to migrate and deploy unified business applications i...
How to use hybrid cloud to migrate and deploy unified business applications i...How to use hybrid cloud to migrate and deploy unified business applications i...
How to use hybrid cloud to migrate and deploy unified business applications i...
Eric D. Schabell
 
Real Time Streaming Architecture at Ford
Real Time Streaming Architecture at FordReal Time Streaming Architecture at Ford
Real Time Streaming Architecture at Ford
DataWorks Summit
 
Balancing Innovation and Architecture
Balancing Innovation and ArchitectureBalancing Innovation and Architecture
Balancing Innovation and Architecture
NUS-ISS
 
RightScale Roadtrip Boston: Accelerate to Cloud
RightScale Roadtrip Boston: Accelerate to CloudRightScale Roadtrip Boston: Accelerate to Cloud
RightScale Roadtrip Boston: Accelerate to Cloud
RightScale
 
The Lyft data platform: Now and in the future
The Lyft data platform: Now and in the futureThe Lyft data platform: Now and in the future
The Lyft data platform: Now and in the future
markgrover
 

Similar to The application and architecture of the real-time computing platform in di di-flink_forward_2018 (20)

Data Engineer's Lunch #81: Reverse ETL Tools for Modern Data Platforms
Data Engineer's Lunch #81: Reverse ETL Tools for Modern Data PlatformsData Engineer's Lunch #81: Reverse ETL Tools for Modern Data Platforms
Data Engineer's Lunch #81: Reverse ETL Tools for Modern Data Platforms
 
Developing Enterprise Consciousness: Building Modern Open Data Platforms
Developing Enterprise Consciousness: Building Modern Open Data PlatformsDeveloping Enterprise Consciousness: Building Modern Open Data Platforms
Developing Enterprise Consciousness: Building Modern Open Data Platforms
 
Highway to heaven - Microservices Meetup Munich
Highway to heaven - Microservices Meetup MunichHighway to heaven - Microservices Meetup Munich
Highway to heaven - Microservices Meetup Munich
 
Apache Flink Adoption at Shopify
Apache Flink Adoption at ShopifyApache Flink Adoption at Shopify
Apache Flink Adoption at Shopify
 
Preventative Maintenance of Robots in Automotive Industry
Preventative Maintenance of Robots in Automotive IndustryPreventative Maintenance of Robots in Automotive Industry
Preventative Maintenance of Robots in Automotive Industry
 
Running Data Platforms Like Products
Running Data Platforms Like ProductsRunning Data Platforms Like Products
Running Data Platforms Like Products
 
PayPal datalake journey | teradata - edge of next | san diego | 2017 october ...
PayPal datalake journey | teradata - edge of next | san diego | 2017 october ...PayPal datalake journey | teradata - edge of next | san diego | 2017 october ...
PayPal datalake journey | teradata - edge of next | san diego | 2017 october ...
 
How to design and implement a data ops architecture with sdc and gcp
How to design and implement a data ops architecture with sdc and gcpHow to design and implement a data ops architecture with sdc and gcp
How to design and implement a data ops architecture with sdc and gcp
 
Building Blockchain Apps: Intro to Dash Platform
Building Blockchain Apps: Intro to Dash PlatformBuilding Blockchain Apps: Intro to Dash Platform
Building Blockchain Apps: Intro to Dash Platform
 
Geode: Much More Than a Cache
Geode: Much More Than a CacheGeode: Much More Than a Cache
Geode: Much More Than a Cache
 
Overview and Walkthrough of the Application Programming Model with SAP Cloud ...
Overview and Walkthrough of the Application Programming Model with SAP Cloud ...Overview and Walkthrough of the Application Programming Model with SAP Cloud ...
Overview and Walkthrough of the Application Programming Model with SAP Cloud ...
 
Next generation business automation with the red hat decision manager and red...
Next generation business automation with the red hat decision manager and red...Next generation business automation with the red hat decision manager and red...
Next generation business automation with the red hat decision manager and red...
 
Enterprise Application Migration
Enterprise Application MigrationEnterprise Application Migration
Enterprise Application Migration
 
Sourav_Giri_Resume_2015
Sourav_Giri_Resume_2015Sourav_Giri_Resume_2015
Sourav_Giri_Resume_2015
 
IoT Scale Event-Stream Processing for Connected Fleet at Penske
IoT Scale Event-Stream Processing for Connected Fleet at PenskeIoT Scale Event-Stream Processing for Connected Fleet at Penske
IoT Scale Event-Stream Processing for Connected Fleet at Penske
 
How to use hybrid cloud to migrate and deploy unified business applications i...
How to use hybrid cloud to migrate and deploy unified business applications i...How to use hybrid cloud to migrate and deploy unified business applications i...
How to use hybrid cloud to migrate and deploy unified business applications i...
 
Real Time Streaming Architecture at Ford
Real Time Streaming Architecture at FordReal Time Streaming Architecture at Ford
Real Time Streaming Architecture at Ford
 
Balancing Innovation and Architecture
Balancing Innovation and ArchitectureBalancing Innovation and Architecture
Balancing Innovation and Architecture
 
RightScale Roadtrip Boston: Accelerate to Cloud
RightScale Roadtrip Boston: Accelerate to CloudRightScale Roadtrip Boston: Accelerate to Cloud
RightScale Roadtrip Boston: Accelerate to Cloud
 
The Lyft data platform: Now and in the future
The Lyft data platform: Now and in the futureThe Lyft data platform: Now and in the future
The Lyft data platform: Now and in the future
 

Recently uploaded

HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
CAKE: Sharing Slices of Confidential Data on Blockchain
CAKE: Sharing Slices of Confidential Data on BlockchainCAKE: Sharing Slices of Confidential Data on Blockchain
CAKE: Sharing Slices of Confidential Data on Blockchain
Claudio Di Ciccio
 
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI modelsInfrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
Zilliz
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
Mariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceXMariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceX
Mariano Tinti
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
Claudio Di Ciccio
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
名前 です男
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
Wouter Lemaire
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Edge AI and Vision Alliance
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Zilliz
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 

Recently uploaded (20)

HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
CAKE: Sharing Slices of Confidential Data on Blockchain
CAKE: Sharing Slices of Confidential Data on BlockchainCAKE: Sharing Slices of Confidential Data on Blockchain
CAKE: Sharing Slices of Confidential Data on Blockchain
 
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI modelsInfrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
Mariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceXMariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceX
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 

The application and architecture of the real-time computing platform in di di-flink_forward_2018