Data pipeline and data lake for autonomous driving

•

0 likes•442 views

Yu Huang

Data pipeline, data lake, autonomous driving

Engineering

DATA PIPELINE AND DATA LAKE IN
AUTONOMOUS DRIVING
YU HUANG
SUNNYVALE, CALIFORNIA
YU.HUANG07@GMAIL.COM

OUTLINE
• TESLA
• GOOGLE WAYMO
• PLUSAI
• ALIBABA CLOUD
• NVIDIA
• NETAPP
• AMAZON-AWS
• AMAZON-TRI
• AMAZON-MOMENTA
• ECKERSON DATAOPS
• IBM

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Reference Architecture
Autonomous Driving Data Lake
Build an MDF4/Rosbag-based data ingestion and processing pipeline for Autonomous Driving and
Advanced Driver Assistance Systems (ADAS).
10
9
8
7
6
5
4
3
2
Ingest data from autonomous fleet
with AWS Outposts for local data
processing.
Ingest vehicle telemetry data in real
time using AWS IoT Core and
Amazon Kinesis Data Firehose.
Remove and transform low quality
data.
Schedule the extract, transform, load
(ETL) jobs using Apache Airflow.
Enrich data with weather conditions
based on GPS location and
timestamp.
Extract metadata using ASAM
OpenSCENARIO and store data in
Amazon DynamoDB and Amazon
Elasticsearch Service.
Store data lineage in Amazon
Neptune and catalog data using AWS
Glue Data Catalog.
Process drive data and perform deep
signal validation.
Perform automated labeling using
Amazon SageMaker Ground Truth.
Provide a search function for
particular scenarios using AWS
AppSync.
1
1
2
3
4
5
6
7
8 9
10

Radar and video data processing in MDF4 format achieves the highest scalability by leveraging AWS Fargate for Amazon ECS.

Data Pipeline Complexity
Eckerson DataOps

Analytic Models Failure to Launch
Eckerson DataOps

Streaming Data Pipelines
Eckerson DataOps

Requirements for Autonomous Navigation
Eckerson DataOps

DataOps for Model and Pipeline
Development and Operationalization
Continuous Integration / Continuous
Development（CI/CD）
Eckerson DataOps

Components and Complexities of Analytics Ecosystems
Eckerson DataOps

IBM Enterprise Data pipeline Marketecture

IBM Autonomous driving development use cases

What's hot

Reduce Costs and Build a Strong Operational Foundation with the AWS Migration...Amazon Web Services

Masterclass Live: Amazon EMRAmazon Web Services

Amazon EMR MasterclassAmazon Web Services

Webinar aws 101 a walk through the aws cloud- introduction to cloud computi...Amazon Web Services

AWS Neptune - A Fast and reliable Graph Database Built for the CloudAmazon Web Services

현대백화점 리테일테크랩과 AWS Prototyping 팀 개발자가 들려주는 인공 지능 무인 스토어 개발 여정 - 최권열 AWS 프로토타이핑...Amazon Web Services Korea

Leveraging the AWS Sales Methodology and Partner Best Practices aws-partner-s...Amazon Web Services

AWS VS AZURE VS GCP.pptxRaneesh Ramesan

AWS 기반 클라우드 아키텍처 모범사례 - 삼성전자 개발자 포털/개발자 워크스페이스 - 정영준 솔루션즈 아키텍트, AWS / 유현성 수석,...Amazon Web Services Korea

Building-a-Data-Lake-on-AWSAmazon Web Services

The Zen of DataOps – AWS Lake Formation and the Data Supply Chain PipelineAmazon Web Services

Cloud Migration, Application Modernization and Security for PartnersAmazon Web Services

Data Center Migration to the AWS CloudTom Laszewski

On-premise to Microsoft Azure Cloud Migration.Emtec Inc.

API Connect Presentationxband

AWS Builders Online Series | AWS와 함께하는 클라우드 컴퓨팅 - 강철, AWS 어카운트 매니저Amazon Web Services Korea

Building Event Driven (Micro)services with Apache KafkaGuido Schmutz

AWS Cloud Adoption Framework and WorkshopsTom Laszewski

AWS Partnership ModelAmazon Web Services

End-to-End Machine Learning with Amazon SageMakerSungmin Kim

What's hot (20)

Reduce Costs and Build a Strong Operational Foundation with the AWS Migration...

Masterclass Live: Amazon EMR

Amazon EMR Masterclass

Webinar aws 101 a walk through the aws cloud- introduction to cloud computi...

AWS Neptune - A Fast and reliable Graph Database Built for the Cloud

현대백화점 리테일테크랩과 AWS Prototyping 팀 개발자가 들려주는 인공 지능 무인 스토어 개발 여정 - 최권열 AWS 프로토타이핑...

Leveraging the AWS Sales Methodology and Partner Best Practices aws-partner-s...

AWS VS AZURE VS GCP.pptx

AWS 기반 클라우드 아키텍처 모범사례 - 삼성전자 개발자 포털/개발자 워크스페이스 - 정영준 솔루션즈 아키텍트, AWS / 유현성 수석,...

Building-a-Data-Lake-on-AWS

The Zen of DataOps – AWS Lake Formation and the Data Supply Chain Pipeline

Cloud Migration, Application Modernization and Security for Partners

Data Center Migration to the AWS Cloud

On-premise to Microsoft Azure Cloud Migration.

API Connect Presentation

AWS Builders Online Series | AWS와 함께하는 클라우드 컴퓨팅 - 강철, AWS 어카운트 매니저

Building Event Driven (Micro)services with Apache Kafka

AWS Cloud Adoption Framework and Workshops

AWS Partnership Model

End-to-End Machine Learning with Amazon SageMaker

Similar to Data pipeline and data lake for autonomous driving

GPSWKS401_Designing a Cloud Enterprise Data WarehouseAmazon Web Services

Success has Many Query Engines- Tel Aviv Summit 2018Amazon Web Services

How Disney+ uses fast data ubiquity to improve the customer experience Martin Zapletal

Building Data Lakes and Analytics on AWSAmazon Web Services

AMF305_Autonomous Driving Algorithm Development on Amazon AIAmazon Web Services

Construindo data lakes e analytics com AWSAmazon Web Services LATAM

Accelerate Productivity by Computing at the Edge - AWS Online Tech TalksAmazon Web Services

Build Data Lakes & Analytics on AWS: Patterns & Best PracticesAmazon Web Services

Build Data Lakes and Analytics on AWS: Patterns & Best PracticesAmazon Web Services

Managed Relational Databases - Amazon RDSAmazon Web Services

Hybrid as a Stepping Stone: It’s Not All or Nothing for Your Cloud Transforma...Amazon Web Services

Building Data Lakes for Analytics on AWSAmazon Web Services

Build Data Lakes and Analytics on AWS: Patterns & Best Practices - BDA305 - A...Amazon Web Services

Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...Amazon Web Services

Building Data Lakes and Analytics on AWSAmazon Web Services

What's new with Amazon Redshift - ADB203 - New York AWS SummitAmazon Web Services

AWS March 2016 Webinar Series - Building Big Data Solutions with Amazon EMR a...Amazon Web Services

Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...Amazon Web Services

Building a Modern Data Platform in the CloudAmazon Web Services

Similar to Data pipeline and data lake for autonomous driving (20)

GPSWKS401_Designing a Cloud Enterprise Data Warehouse

Success has Many Query Engines- Tel Aviv Summit 2018

How Disney+ uses fast data ubiquity to improve the customer experience

Building Data Lakes and Analytics on AWS

AMF305_Autonomous Driving Algorithm Development on Amazon AI

Construindo data lakes e analytics com AWS

Accelerate Productivity by Computing at the Edge - AWS Online Tech Talks

Build Data Lakes & Analytics on AWS: Patterns & Best Practices

Build Data Lakes and Analytics on AWS: Patterns & Best Practices

Managed Relational Databases - Amazon RDS

Hybrid as a Stepping Stone: It’s Not All or Nothing for Your Cloud Transforma...

Building Data Lakes for Analytics on AWS

Build Data Lakes and Analytics on AWS: Patterns & Best Practices - BDA305 - A...

Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...

Building Data Lakes and Analytics on AWS

What's new with Amazon Redshift - ADB203 - New York AWS Summit

AWS March 2016 Webinar Series - Building Big Data Solutions with Amazon EMR a...

Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...

Building a Modern Data Platform in the Cloud

Recently uploaded

DATA ANALYTICS PPT definition usage examplePragyanshuParadkar1

Electronically Controlled suspensions system .pdfme23b1001

Past, Present and Future of Generative AIabhishek36461

An experimental study in using natural admixture as an alternative for chemic...Chandu841456

young call girls in Rajiv Chowk🔝 9953056974 🔝 Delhi escort Service9953056974 Low Rate Call Girls In Saket, Delhi NCR

CCS355 Neural Networks & Deep Learning Unit 1 PDF notes with Question bank .pdfAsst.prof M.Gokilavani

complete construction, environmental and economics information of biomass com...asadnawaz62

Arduino_CSE ece ppt for working and principal of arduino.pptSAURABHKUMAR892774

Effects of rheological properties on mixingviprabot1

Introduction to Machine Learning Unit-3 for II MECHC Sai Kiran

Call Us ≽ 8377877756 ≼ Call Girls In Shastri Nagar (Delhi)dollysharma2066

EduAI - E learning Platform integrated with AIkoyaldeepu123

Software and Systems Engineering Standards: Verification and Validation of Sy...VICTOR MAESTRE RAMIREZ

Heart Disease Prediction using machine learning.pptxPoojaBan

young call girls in Green Park🔝 9953056974 🔝 escort Service9953056974 Low Rate Call Girls In Saket, Delhi NCR

Call Girls Narol 7397865700 Independent Call Girlsssuser7cb4ff

GDSC ASEB Gen AI study jams presentationGDSCAESB

CCS355 Neural Network & Deep Learning UNIT III notes and Question bank .pdfAsst.prof M.Gokilavani

Exploring_Network_Security_with_JA3_by_Rakesh Seal.pptxnull - The Open Security Community

INFLUENCE OF NANOSILICA ON THE PROPERTIES OF CONCRETEroselinkalist12

Recently uploaded (20)

DATA ANALYTICS PPT definition usage example

Electronically Controlled suspensions system .pdf

Past, Present and Future of Generative AI

An experimental study in using natural admixture as an alternative for chemic...

young call girls in Rajiv Chowk🔝 9953056974 🔝 Delhi escort Service

CCS355 Neural Networks & Deep Learning Unit 1 PDF notes with Question bank .pdf

complete construction, environmental and economics information of biomass com...

Arduino_CSE ece ppt for working and principal of arduino.ppt

Effects of rheological properties on mixing

Introduction to Machine Learning Unit-3 for II MECH

Call Us ≽ 8377877756 ≼ Call Girls In Shastri Nagar (Delhi)

EduAI - E learning Platform integrated with AI

Software and Systems Engineering Standards: Verification and Validation of Sy...

Heart Disease Prediction using machine learning.pptx

young call girls in Green Park🔝 9953056974 🔝 escort Service

Call Girls Narol 7397865700 Independent Call Girls

GDSC ASEB Gen AI study jams presentation

CCS355 Neural Network & Deep Learning UNIT III notes and Question bank .pdf

Exploring_Network_Security_with_JA3_by_Rakesh Seal.pptx

INFLUENCE OF NANOSILICA ON THE PROPERTIES OF CONCRETE

Data pipeline and data lake for autonomous driving

1. DATA PIPELINE AND DATA LAKE IN AUTONOMOUS DRIVING YU HUANG SUNNYVALE, CALIFORNIA YU.HUANG07@GMAIL.COM

2. OUTLINE • TESLA • GOOGLE WAYMO • PLUSAI • ALIBABA CLOUD • NVIDIA • NETAPP • AMAZON-AWS • AMAZON-TRI • AMAZON-MOMENTA • ECKERSON DATAOPS • IBM

3. Tesla

7. Nvidia

8. Nvidia

9. Nvidia

10. Nvidia

11. Nvidia

12. Nvidia

13. NetApp

14. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Reference Architecture Autonomous Driving Data Lake Build an MDF4/Rosbag-based data ingestion and processing pipeline for Autonomous Driving and Advanced Driver Assistance Systems (ADAS). 10 9 8 7 6 5 4 3 2 Ingest data from autonomous fleet with AWS Outposts for local data processing. Ingest vehicle telemetry data in real time using AWS IoT Core and Amazon Kinesis Data Firehose. Remove and transform low quality data. Schedule the extract, transform, load (ETL) jobs using Apache Airflow. Enrich data with weather conditions based on GPS location and timestamp. Extract metadata using ASAM OpenSCENARIO and store data in Amazon DynamoDB and Amazon Elasticsearch Service. Store data lineage in Amazon Neptune and catalog data using AWS Glue Data Catalog. Process drive data and perform deep signal validation. Perform automated labeling using Amazon SageMaker Ground Truth. Provide a search function for particular scenarios using AWS AppSync. 1 1 2 3 4 5 6 7 8 9 10

15. Radar and video data processing in MDF4 format achieves the highest scalability by leveraging AWS Fargate for Amazon ECS.

16.

17.

18. Autonomous driving at AWS

19. TRI

20. Momenta

21. Next Generation Data Pipelines

22. Data Pipeline Complexity Eckerson DataOps

23. Analytic Models Failure to Launch Eckerson DataOps

24. Streaming Data Pipelines Eckerson DataOps

25. Requirements for Autonomous Navigation Eckerson DataOps

26. DataOps for Model and Pipeline Development and Operationalization Continuous Integration / Continuous Development（CI/CD） Eckerson DataOps