SlideShare a Scribd company logo

How Disney+ uses fast data ubiquity to improve the customer experience

•
•
Martin Zapletal
Martin Zapletal

Disney+ uses Amazon Kinesis to drive real-time actions like providing title recommendations for customers, sending events across microservices, and delivering logs for operational analytics to improve the customer experience. In this session, you learn how Disney+ built real-time data-driven capabilities on a unified streaming platform. This platform ingests billions of events per hour in Amazon Kinesis Data Streams, processes and analyzes that data in Amazon Kinesis Data Analytics for Apache Flink, and uses Amazon Kinesis Data Firehose to deliver data to destinations without servers or code. Hear how these services helped Disney+ scale its viewing experience to tens of millions of customers with the required quality and reliability. Learn more about re:Invent 2020 at http://bit.ly/3c4NSdY

1 of 34
Download to read offline
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. How Disney+ uses fast data ubiquity to improve the customer experience Martin Zapletal Director, Software Engineering Disney+ A N T 3 0 9
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Streaming data and Disney+ Before – silos Evolution – streaming silos Now – data driven Streaming Data Platform Examples – ubiquity, platform, culture Agenda
• Variety of streaming use cases with varying needs • Dozens of millions of users • Hundreds of Amazon Kinesis data streams • Thousands of shards • Multiple regions • Billions of events • Terabytes of data Streaming data
• Microservices • Databases and data warehouses • Batch processing • Slow and limited insights • Silos Before: Silos
• Streaming, event driven, asynchronous • Custom, unique integrations and data warehouses Evolution: Streaming silos Amazon Kinesis Data Firehose Amazon Kinesis Data Streams Amazon S3 Amazon RDS Amazon Athena Amazon Redshift Amazon ECS Amazon ECS Amazon ECS AWS Lambda Amazon Kinesis Data Streams Amazon Kinesis Data Firehose Amazon S3 Amazon ECS Amazon Kinesis Data Streams
• Streaming, event driven, asynchronous • Custom, unique integrations and data warehouses Evolution: Streaming silos Amazon Kinesis Data Firehose Amazon Kinesis Data Streams Amazon S3 Amazon RDS Amazon Athena Amazon Redshift Amazon ECS Amazon ECS Amazon ECS AWS Lambda Amazon Kinesis Data Streams Amazon Kinesis Data Firehose Amazon S3 Amazon ECS Amazon Kinesis Data Streams Data format 2 Schema management 2 Data quality approach 2 Data governance 2 Tooling 2 … Data format 3 Schema management 3 Data quality approach 3 Data governance 3 Tooling 3 … Data format 1 Schema management 1 Data quality approach 1 Data governance 1 Tooling 1 …
• (Fast) data democracy • Real-time data, insights, ML • Experimentation • First-class consideration • Culture “Data / insights they need available at the time they need it” Now: Data driven
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Streaming Data Platform
Streaming Data Platform
Streaming Data Platform Analytics and ML Streaming Data Platform Amazon Kinesis Data Streams Ubiquity Platform Culture Experimentation Services Amazon Kinesis Data Firehose AWS SDK, KPL, KCL AWS Lambda Databricks / Spark Amazon Kinesis Data Analytics for Apache Flink
• Need a reliable, performant, cost-efficient event log • Kinesis, Kafka, Pulsar, and others • Amazon Kinesis Data Streams § Replicated, partitioned, ordered, distributed log § Managed § Replication to 3 AZs § Integration with other AWS services § Near real time § Scalability § Elasticity Kinesis
Examples
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Ubiquity
Data management Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
Data management Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
Schema registry Data management Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Platform
Platform Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
Reliable domain events Platform Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
Validation, routing, filtering Platform Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
Join, enrichment Platform Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
Ingestion Platform Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
Streaming application maturity § Architecture patterns § Automated testing § Performance testing and management § Elasticity and auto scaling § Deployment § Observability, alerting § Reliability and resilience § Operations simplicity § Multi-region replication and failover § Data lineage § Self-healing § Distributed tracing § Cost efficiency § Discoverability § Traffic routing § Guarantees § Streaming as a service platform § Etc. Platform
• Configurable trade-offs • Latency management • Deployment patterns Platform
• Stream elasticity • Application elasticity • Elasticity trade-offs Platform
• Reliability • Delivery semantics • End-to-end management • Failure scenarios Platform
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Culture
Culture Training Collaboration Tooling Ease of use Integrations
• Data-driven organization and data democracy • Ubiquitous data • Streaming data platform • Culture and tools • Build on top of Amazon Kinesis Conclusion
Resources § Disney Technology Blog – https://medium.com/disney-streaming § Delivering data in real-time via auto scaling Kinesis streams – https://medium.com/disney-streaming/delivering-data-in- real-time-via-auto-scaling-kinesis-streams-72a0236b2cd9 § Testing asynchronous pipelines with fs2 and weaver-test – https://medium.com/disney-streaming/testing-asynchronous- pipelines-with-fs2-and-weaver-test-f0ffd37676d § Open source project weaver-test – https://github.com/disneystreaming/weaver-test/ Credits and resources Credits • Tom LeRoux • Christian Villoslada • Petr Zapletal • Nick Burkard • Matt Jankowski • Ben Morris • Jess Geddes • Daniel Spiewak • Diego Pineda • Eric Meisel • Anthony Garo • Benoit Louy • Mark Harrison • Evan Kaplan • Olivier Melois • Rekha Bachwani • User Services team • Subscription team • Streaming Data Platform team • Data Engineering team • API Services team • Data Governance & Instrumentation team • Experimentation team • And the whole Disney+ team!
Thank you! © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Martin Zapletal Twitter @zapletal_martin LinkedIn martinzapletal
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Recommended

Self-service Events & Decentralised Governance with AsyncAPI: A Real World Ex...
Self-service Events & Decentralised Governance with AsyncAPI: A Real World Ex...Self-service Events & Decentralised Governance with AsyncAPI: A Real World Ex...
Self-service Events & Decentralised Governance with AsyncAPI: A Real World Ex...
HostedbyConfluent
 
Logging infrastructure for Microservices using StreamSets Data Collector
Logging infrastructure for Microservices using StreamSets Data CollectorLogging infrastructure for Microservices using StreamSets Data Collector
Logging infrastructure for Microservices using StreamSets Data Collector
Cask Data
 
Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...
Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...
Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...
HostedbyConfluent
 
Stream Processing Live Traffic Data with Kafka Streams
Stream Processing Live Traffic Data with Kafka StreamsStream Processing Live Traffic Data with Kafka Streams
Stream Processing Live Traffic Data with Kafka Streams
Tom Van den Bulck
 
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
HostedbyConfluent
 
Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...
Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...
Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...
HostedbyConfluent
 
Blueprint Series: Architecture Patterns for Implementing Serverless Microserv...
Blueprint Series: Architecture Patterns for Implementing Serverless Microserv...Blueprint Series: Architecture Patterns for Implementing Serverless Microserv...
Blueprint Series: Architecture Patterns for Implementing Serverless Microserv...
Matt Stubbs
 
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
confluent
 

Recommended

Self-service Events & Decentralised Governance with AsyncAPI: A Real World Ex...
Self-service Events & Decentralised Governance with AsyncAPI: A Real World Ex...Self-service Events & Decentralised Governance with AsyncAPI: A Real World Ex...
Self-service Events & Decentralised Governance with AsyncAPI: A Real World Ex...
HostedbyConfluent
 
Logging infrastructure for Microservices using StreamSets Data Collector
Logging infrastructure for Microservices using StreamSets Data CollectorLogging infrastructure for Microservices using StreamSets Data Collector
Logging infrastructure for Microservices using StreamSets Data Collector
Cask Data
 
Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...
Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...
Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...
HostedbyConfluent
 
Stream Processing Live Traffic Data with Kafka Streams
Stream Processing Live Traffic Data with Kafka StreamsStream Processing Live Traffic Data with Kafka Streams
Stream Processing Live Traffic Data with Kafka Streams
Tom Van den Bulck
 
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...
HostedbyConfluent
 
Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...
Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...
Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...
HostedbyConfluent
 
Blueprint Series: Architecture Patterns for Implementing Serverless Microserv...
Blueprint Series: Architecture Patterns for Implementing Serverless Microserv...Blueprint Series: Architecture Patterns for Implementing Serverless Microserv...
Blueprint Series: Architecture Patterns for Implementing Serverless Microserv...
Matt Stubbs
 
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
confluent
 
AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...
AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...
AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...
Amazon Web Services
 
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
Apache Flink @ Alibaba - Seattle Apache Flink MeetupApache Flink @ Alibaba - Seattle Apache Flink Meetup
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
Bowen Li
 
Data Driven Enterprise with Apache Kafka
Data Driven Enterprise with Apache KafkaData Driven Enterprise with Apache Kafka
Data Driven Enterprise with Apache Kafka
confluent
 
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
Monal Daxini
 
Blueprint Series: Expedia Partner Solutions, Data Platform
Blueprint Series: Expedia Partner Solutions, Data PlatformBlueprint Series: Expedia Partner Solutions, Data Platform
Blueprint Series: Expedia Partner Solutions, Data Platform
Matt Stubbs
 
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
HostedbyConfluent
 
AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)
AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)
AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)
Amazon Web Services
 
Streaming data for real time analysis
Streaming data for real time analysisStreaming data for real time analysis
Streaming data for real time analysis
Amazon Web Services
 
Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive
Hadoop made fast - Why Virtual Reality Needed Stream Processing to SurviveHadoop made fast - Why Virtual Reality Needed Stream Processing to Survive
Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive
confluent
 
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
Amazon Web Services
 
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud ServicesBuild a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
confluent
 
Asynchronous micro-services and the unified log
Asynchronous micro-services and the unified logAsynchronous micro-services and the unified log
Asynchronous micro-services and the unified log
Alexander Dean
 
Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation
Kafka Summit SF 2017 - Riot's Journey to Global Kafka AggregationKafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation
Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation
confluent
 
So You’ve Inherited Kafka? Now What? (Alon Gavra, AppsFlyer) Kafka Summit Lon...
So You’ve Inherited Kafka? Now What? (Alon Gavra, AppsFlyer) Kafka Summit Lon...So You’ve Inherited Kafka? Now What? (Alon Gavra, AppsFlyer) Kafka Summit Lon...
So You’ve Inherited Kafka? Now What? (Alon Gavra, AppsFlyer) Kafka Summit Lon...
confluent
 
Apache kafka-a distributed streaming platform
Apache kafka-a distributed streaming platformApache kafka-a distributed streaming platform
Apache kafka-a distributed streaming platform
confluent
 
Data Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDBData Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDB
confluent
 
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Kai Wähner
 
Maximize the Business Value of Machine Learning and Data Science with Kafka (...
Maximize the Business Value of Machine Learning and Data Science with Kafka (...Maximize the Business Value of Machine Learning and Data Science with Kafka (...
Maximize the Business Value of Machine Learning and Data Science with Kafka (...
confluent
 
Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...
Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...
Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...
Flink Forward
 
Battle-tested event-driven patterns for your microservices architecture - Sca...
Battle-tested event-driven patterns for your microservices architecture - Sca...Battle-tested event-driven patterns for your microservices architecture - Sca...
Battle-tested event-driven patterns for your microservices architecture - Sca...
Natan Silnitsky
 
Getting Started with Amazon Kinesis
Getting Started with Amazon KinesisGetting Started with Amazon Kinesis
Getting Started with Amazon Kinesis
Amazon Web Services
 
Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...
Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...
Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...
Amazon Web Services
 

More Related Content

What's hot

AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...
AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...
AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...
Amazon Web Services
 
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
Apache Flink @ Alibaba - Seattle Apache Flink MeetupApache Flink @ Alibaba - Seattle Apache Flink Meetup
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
Bowen Li
 
Data Driven Enterprise with Apache Kafka
Data Driven Enterprise with Apache KafkaData Driven Enterprise with Apache Kafka
Data Driven Enterprise with Apache Kafka
confluent
 
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
Monal Daxini
 
Blueprint Series: Expedia Partner Solutions, Data Platform
Blueprint Series: Expedia Partner Solutions, Data PlatformBlueprint Series: Expedia Partner Solutions, Data Platform
Blueprint Series: Expedia Partner Solutions, Data Platform
Matt Stubbs
 
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
HostedbyConfluent
 
AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)
AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)
AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)
Amazon Web Services
 
Streaming data for real time analysis
Streaming data for real time analysisStreaming data for real time analysis
Streaming data for real time analysis
Amazon Web Services
 
Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive
Hadoop made fast - Why Virtual Reality Needed Stream Processing to SurviveHadoop made fast - Why Virtual Reality Needed Stream Processing to Survive
Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive
confluent
 
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
Amazon Web Services
 
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud ServicesBuild a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
confluent
 
Asynchronous micro-services and the unified log
Asynchronous micro-services and the unified logAsynchronous micro-services and the unified log
Asynchronous micro-services and the unified log
Alexander Dean
 
Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation
Kafka Summit SF 2017 - Riot's Journey to Global Kafka AggregationKafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation
Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation
confluent
 
So You’ve Inherited Kafka? Now What? (Alon Gavra, AppsFlyer) Kafka Summit Lon...
So You’ve Inherited Kafka? Now What? (Alon Gavra, AppsFlyer) Kafka Summit Lon...So You’ve Inherited Kafka? Now What? (Alon Gavra, AppsFlyer) Kafka Summit Lon...
So You’ve Inherited Kafka? Now What? (Alon Gavra, AppsFlyer) Kafka Summit Lon...
confluent
 
Apache kafka-a distributed streaming platform
Apache kafka-a distributed streaming platformApache kafka-a distributed streaming platform
Apache kafka-a distributed streaming platform
confluent
 
Data Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDBData Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDB
confluent
 
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Kai Wähner
 
Maximize the Business Value of Machine Learning and Data Science with Kafka (...
Maximize the Business Value of Machine Learning and Data Science with Kafka (...Maximize the Business Value of Machine Learning and Data Science with Kafka (...
Maximize the Business Value of Machine Learning and Data Science with Kafka (...
confluent
 
Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...
Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...
Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...
Flink Forward
 
Battle-tested event-driven patterns for your microservices architecture - Sca...
Battle-tested event-driven patterns for your microservices architecture - Sca...Battle-tested event-driven patterns for your microservices architecture - Sca...
Battle-tested event-driven patterns for your microservices architecture - Sca...
Natan Silnitsky
 

What's hot (20)

AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...
AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...
AWS re:Invent 2016: How Toyota Racing Development Makes Racing Decisions in R...
 
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
Apache Flink @ Alibaba - Seattle Apache Flink MeetupApache Flink @ Alibaba - Seattle Apache Flink Meetup
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
 
Data Driven Enterprise with Apache Kafka
Data Driven Enterprise with Apache KafkaData Driven Enterprise with Apache Kafka
Data Driven Enterprise with Apache Kafka
 
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
 
Blueprint Series: Expedia Partner Solutions, Data Platform
Blueprint Series: Expedia Partner Solutions, Data PlatformBlueprint Series: Expedia Partner Solutions, Data Platform
Blueprint Series: Expedia Partner Solutions, Data Platform
 
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
 
AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)
AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)
AWS re:Invent 2016: Streaming ETL for RDS and DynamoDB (DAT315)
 
Streaming data for real time analysis
Streaming data for real time analysisStreaming data for real time analysis
Streaming data for real time analysis
 
Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive
Hadoop made fast - Why Virtual Reality Needed Stream Processing to SurviveHadoop made fast - Why Virtual Reality Needed Stream Processing to Survive
Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive
 
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
AWS re:Invent 2016: Tableau Rules of Engagement in the Cloud (STG306)
 
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud ServicesBuild a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
Build a Bridge to Cloud with Apache Kafka® for Data Analytics Cloud Services
 
Asynchronous micro-services and the unified log
Asynchronous micro-services and the unified logAsynchronous micro-services and the unified log
Asynchronous micro-services and the unified log
 
Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation
Kafka Summit SF 2017 - Riot's Journey to Global Kafka AggregationKafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation
Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation
 
So You’ve Inherited Kafka? Now What? (Alon Gavra, AppsFlyer) Kafka Summit Lon...
So You’ve Inherited Kafka? Now What? (Alon Gavra, AppsFlyer) Kafka Summit Lon...So You’ve Inherited Kafka? Now What? (Alon Gavra, AppsFlyer) Kafka Summit Lon...
So You’ve Inherited Kafka? Now What? (Alon Gavra, AppsFlyer) Kafka Summit Lon...
 
Apache kafka-a distributed streaming platform
Apache kafka-a distributed streaming platformApache kafka-a distributed streaming platform
Apache kafka-a distributed streaming platform
 
Data Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDBData Streaming with Apache Kafka & MongoDB
Data Streaming with Apache Kafka & MongoDB
 
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
 
Maximize the Business Value of Machine Learning and Data Science with Kafka (...
Maximize the Business Value of Machine Learning and Data Science with Kafka (...Maximize the Business Value of Machine Learning and Data Science with Kafka (...
Maximize the Business Value of Machine Learning and Data Science with Kafka (...
 
Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...
Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...
Flink Forward San Francisco 2018: Ken Krugler - "Building a scalable focused ...
 
Battle-tested event-driven patterns for your microservices architecture - Sca...
Battle-tested event-driven patterns for your microservices architecture - Sca...Battle-tested event-driven patterns for your microservices architecture - Sca...
Battle-tested event-driven patterns for your microservices architecture - Sca...
 

Similar to How Disney+ uses fast data ubiquity to improve the customer experience

Getting Started with Amazon Kinesis
Getting Started with Amazon KinesisGetting Started with Amazon Kinesis
Getting Started with Amazon Kinesis
Amazon Web Services
 
Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...
Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...
Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...
Amazon Web Services
 
Building Data Lakes and Analytics on AWS
Building Data Lakes and Analytics on AWSBuilding Data Lakes and Analytics on AWS
Building Data Lakes and Analytics on AWS
Amazon Web Services
 
Building Data Lakes and Analytics on AWS
Building Data Lakes and Analytics on AWSBuilding Data Lakes and Analytics on AWS
Building Data Lakes and Analytics on AWS
Amazon Web Services
 
Build Data Lakes and Analytics on AWS: Patterns & Best Practices - BDA305 - A...
Build Data Lakes and Analytics on AWS: Patterns & Best Practices - BDA305 - A...Build Data Lakes and Analytics on AWS: Patterns & Best Practices - BDA305 - A...
Build Data Lakes and Analytics on AWS: Patterns & Best Practices - BDA305 - A...
Amazon Web Services
 
AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020
AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020
AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020
Amazon Web Services Korea
 
Building your Datalake on AWS
Building your Datalake on AWSBuilding your Datalake on AWS
Building your Datalake on AWS
Amazon Web Services
 
Build Data Lakes and Analytics on AWS: Patterns & Best Practices
Build Data Lakes and Analytics on AWS: Patterns & Best PracticesBuild Data Lakes and Analytics on AWS: Patterns & Best Practices
Build Data Lakes and Analytics on AWS: Patterns & Best Practices
Amazon Web Services
 
Build Data Lakes & Analytics on AWS: Patterns & Best Practices
Build Data Lakes & Analytics on AWS: Patterns & Best PracticesBuild Data Lakes & Analytics on AWS: Patterns & Best Practices
Build Data Lakes & Analytics on AWS: Patterns & Best Practices
Amazon Web Services
 
Getting Started with Amazon Kinesis | AWS Public Sector Summit 2016
Getting Started with Amazon Kinesis | AWS Public Sector Summit 2016Getting Started with Amazon Kinesis | AWS Public Sector Summit 2016
Getting Started with Amazon Kinesis | AWS Public Sector Summit 2016
Amazon Web Services
 
BDA305 Building Data Lakes and Analytics on AWS
BDA305 Building Data Lakes and Analytics on AWSBDA305 Building Data Lakes and Analytics on AWS
BDA305 Building Data Lakes and Analytics on AWS
Amazon Web Services
 
AWS Data Immersion Webinar Week - Entenda como ampliar suas possibilidades de...
AWS Data Immersion Webinar Week - Entenda como ampliar suas possibilidades de...AWS Data Immersion Webinar Week - Entenda como ampliar suas possibilidades de...
AWS Data Immersion Webinar Week - Entenda como ampliar suas possibilidades de...
Amazon Web Services LATAM
 
Construindo data lakes e analytics com AWS
Construindo data lakes e analytics com AWSConstruindo data lakes e analytics com AWS
Construindo data lakes e analytics com AWS
Amazon Web Services LATAM
 
Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...
Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...
Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...
Amazon Web Services
 
Building a Data Processing Pipeline on AWS
Building a Data Processing Pipeline on AWSBuilding a Data Processing Pipeline on AWS
Building a Data Processing Pipeline on AWS
Amazon Web Services
 
BDA303 Serverless big data architectures: Design patterns and best practices
BDA303 Serverless big data architectures: Design patterns and best practicesBDA303 Serverless big data architectures: Design patterns and best practices
BDA303 Serverless big data architectures: Design patterns and best practices
Amazon Web Services
 
Building a Data Processing Pipeline on AWS - AWS Summit SG 2017
Building a Data Processing Pipeline on AWS - AWS Summit SG 2017Building a Data Processing Pipeline on AWS - AWS Summit SG 2017
Building a Data Processing Pipeline on AWS - AWS Summit SG 2017
Amazon Web Services
 
AWS Summit Singapore - Architecting a Serverless Data Lake on AWS
AWS Summit Singapore - Architecting a Serverless Data Lake on AWSAWS Summit Singapore - Architecting a Serverless Data Lake on AWS
AWS Summit Singapore - Architecting a Serverless Data Lake on AWS
Amazon Web Services
 
How TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPTHow TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPT
Amazon Web Services
 

Similar to How Disney+ uses fast data ubiquity to improve the customer experience (20)

Getting Started with Amazon Kinesis
Getting Started with Amazon KinesisGetting Started with Amazon Kinesis
Getting Started with Amazon Kinesis
 
Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...
Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...
Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...
 
Building Data Lakes and Analytics on AWS
Building Data Lakes and Analytics on AWSBuilding Data Lakes and Analytics on AWS
Building Data Lakes and Analytics on AWS
 
Building Data Lakes and Analytics on AWS
Building Data Lakes and Analytics on AWSBuilding Data Lakes and Analytics on AWS
Building Data Lakes and Analytics on AWS
 
Build Data Lakes and Analytics on AWS: Patterns & Best Practices - BDA305 - A...
Build Data Lakes and Analytics on AWS: Patterns & Best Practices - BDA305 - A...Build Data Lakes and Analytics on AWS: Patterns & Best Practices - BDA305 - A...
Build Data Lakes and Analytics on AWS: Patterns & Best Practices - BDA305 - A...
 
AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020
AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020
AWS를 통한 데이터 분석 및 처리의 새로운 혁신 기법 - 김윤건, AWS사업개발 담당:: AWS Summit Online Korea 2020
 
Building your Datalake on AWS
Building your Datalake on AWSBuilding your Datalake on AWS
Building your Datalake on AWS
 
Build Data Lakes and Analytics on AWS: Patterns & Best Practices
Build Data Lakes and Analytics on AWS: Patterns & Best PracticesBuild Data Lakes and Analytics on AWS: Patterns & Best Practices
Build Data Lakes and Analytics on AWS: Patterns & Best Practices
 
Build Data Lakes & Analytics on AWS: Patterns & Best Practices
Build Data Lakes & Analytics on AWS: Patterns & Best PracticesBuild Data Lakes & Analytics on AWS: Patterns & Best Practices
Build Data Lakes & Analytics on AWS: Patterns & Best Practices
 
Getting Started with Amazon Kinesis | AWS Public Sector Summit 2016
Getting Started with Amazon Kinesis | AWS Public Sector Summit 2016Getting Started with Amazon Kinesis | AWS Public Sector Summit 2016
Getting Started with Amazon Kinesis | AWS Public Sector Summit 2016
 
BDA305 Building Data Lakes and Analytics on AWS
BDA305 Building Data Lakes and Analytics on AWSBDA305 Building Data Lakes and Analytics on AWS
BDA305 Building Data Lakes and Analytics on AWS
 
AWS Data Immersion Webinar Week - Entenda como ampliar suas possibilidades de...
AWS Data Immersion Webinar Week - Entenda como ampliar suas possibilidades de...AWS Data Immersion Webinar Week - Entenda como ampliar suas possibilidades de...
AWS Data Immersion Webinar Week - Entenda como ampliar suas possibilidades de...
 
Data_Analytics_and_AI_ML
Data_Analytics_and_AI_MLData_Analytics_and_AI_ML
Data_Analytics_and_AI_ML
 
Construindo data lakes e analytics com AWS
Construindo data lakes e analytics com AWSConstruindo data lakes e analytics com AWS
Construindo data lakes e analytics com AWS
 
Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...
Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...
Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...
 
Building a Data Processing Pipeline on AWS
Building a Data Processing Pipeline on AWSBuilding a Data Processing Pipeline on AWS
Building a Data Processing Pipeline on AWS
 
BDA303 Serverless big data architectures: Design patterns and best practices
BDA303 Serverless big data architectures: Design patterns and best practicesBDA303 Serverless big data architectures: Design patterns and best practices
BDA303 Serverless big data architectures: Design patterns and best practices
 
Building a Data Processing Pipeline on AWS - AWS Summit SG 2017
Building a Data Processing Pipeline on AWS - AWS Summit SG 2017Building a Data Processing Pipeline on AWS - AWS Summit SG 2017
Building a Data Processing Pipeline on AWS - AWS Summit SG 2017
 
AWS Summit Singapore - Architecting a Serverless Data Lake on AWS
AWS Summit Singapore - Architecting a Serverless Data Lake on AWSAWS Summit Singapore - Architecting a Serverless Data Lake on AWS
AWS Summit Singapore - Architecting a Serverless Data Lake on AWS
 
How TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPTHow TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPT
 

More from Martin Zapletal

Customer experience at disney+ through data perspective
Customer experience at disney+ through data perspectiveCustomer experience at disney+ through data perspective
Customer experience at disney+ through data perspective
Martin Zapletal
 
Intelligent System Optimizations
Intelligent System OptimizationsIntelligent System Optimizations
Intelligent System Optimizations
Martin Zapletal
 
Intelligent Distributed Systems Optimizations
Intelligent Distributed Systems OptimizationsIntelligent Distributed Systems Optimizations
Intelligent Distributed Systems Optimizations
Martin Zapletal
 
Data in Motion: Streaming Static Data Efficiently 2
Data in Motion: Streaming Static Data Efficiently 2Data in Motion: Streaming Static Data Efficiently 2
Data in Motion: Streaming Static Data Efficiently 2
Martin Zapletal
 
Data in Motion: Streaming Static Data Efficiently
Data in Motion: Streaming Static Data EfficientlyData in Motion: Streaming Static Data Efficiently
Data in Motion: Streaming Static Data Efficiently
Martin Zapletal
 
Machine learning at Scale with Apache Spark
Machine learning at Scale with Apache SparkMachine learning at Scale with Apache Spark
Machine learning at Scale with Apache Spark
Martin Zapletal
 
Cassandra as an event sourced journal for big data analytics Cassandra Summit...
Cassandra as an event sourced journal for big data analytics Cassandra Summit...Cassandra as an event sourced journal for big data analytics Cassandra Summit...
Cassandra as an event sourced journal for big data analytics Cassandra Summit...
Martin Zapletal
 
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
Martin Zapletal
 
Large volume data analysis on the Typesafe Reactive Platform
Large volume data analysis on the Typesafe Reactive PlatformLarge volume data analysis on the Typesafe Reactive Platform
Large volume data analysis on the Typesafe Reactive Platform
Martin Zapletal
 
Apache spark - Installation
Apache spark - InstallationApache spark - Installation
Apache spark - Installation
Martin Zapletal
 
Apache spark - Spark's distributed programming model
Apache spark - Spark's distributed programming modelApache spark - Spark's distributed programming model
Apache spark - Spark's distributed programming model
Martin Zapletal
 
Apache spark - History and market overview
Apache spark - History and market overviewApache spark - History and market overview
Apache spark - History and market overview
Martin Zapletal
 

More from Martin Zapletal (12)

Customer experience at disney+ through data perspective
Customer experience at disney+ through data perspectiveCustomer experience at disney+ through data perspective
Customer experience at disney+ through data perspective
 
Intelligent System Optimizations
Intelligent System OptimizationsIntelligent System Optimizations
Intelligent System Optimizations
 
Intelligent Distributed Systems Optimizations
Intelligent Distributed Systems OptimizationsIntelligent Distributed Systems Optimizations
Intelligent Distributed Systems Optimizations
 
Data in Motion: Streaming Static Data Efficiently 2
Data in Motion: Streaming Static Data Efficiently 2Data in Motion: Streaming Static Data Efficiently 2
Data in Motion: Streaming Static Data Efficiently 2
 
Data in Motion: Streaming Static Data Efficiently
Data in Motion: Streaming Static Data EfficientlyData in Motion: Streaming Static Data Efficiently
Data in Motion: Streaming Static Data Efficiently
 
Machine learning at Scale with Apache Spark
Machine learning at Scale with Apache SparkMachine learning at Scale with Apache Spark
Machine learning at Scale with Apache Spark
 
Cassandra as an event sourced journal for big data analytics Cassandra Summit...
Cassandra as an event sourced journal for big data analytics Cassandra Summit...Cassandra as an event sourced journal for big data analytics Cassandra Summit...
Cassandra as an event sourced journal for big data analytics Cassandra Summit...
 
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
Large volume data analysis on the Typesafe Reactive Platform - Big Data Scala...
 
Large volume data analysis on the Typesafe Reactive Platform
Large volume data analysis on the Typesafe Reactive PlatformLarge volume data analysis on the Typesafe Reactive Platform
Large volume data analysis on the Typesafe Reactive Platform
 
Apache spark - Installation
Apache spark - InstallationApache spark - Installation
Apache spark - Installation
 
Apache spark - Spark's distributed programming model
Apache spark - Spark's distributed programming modelApache spark - Spark's distributed programming model
Apache spark - Spark's distributed programming model
 
Apache spark - History and market overview
Apache spark - History and market overviewApache spark - History and market overview
Apache spark - History and market overview
 

Recently uploaded

Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
Alison B. Lowndes
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
DianaGray10
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Ana-Maria Mihalceanu
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
UiPathCommunity
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
ThousandEyes
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Product School
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Ramesh Iyer
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
Product School
 
Generating a custom Ruby SDK for your web service or Rails API using Smithy
Generating a custom Ruby SDK for your web service or Rails API using SmithyGenerating a custom Ruby SDK for your web service or Rails API using Smithy
Generating a custom Ruby SDK for your web service or Rails API using Smithy
g2nightmarescribd
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Albert Hoitingh
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
OnBoard
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
Kari Kakkonen
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 

Recently uploaded (20)

Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
 
Generating a custom Ruby SDK for your web service or Rails API using Smithy
Generating a custom Ruby SDK for your web service or Rails API using SmithyGenerating a custom Ruby SDK for your web service or Rails API using Smithy
Generating a custom Ruby SDK for your web service or Rails API using Smithy
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
 
DevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA ConnectDevOps and Testing slides at DASA Connect
DevOps and Testing slides at DASA Connect
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 

How Disney+ uses fast data ubiquity to improve the customer experience

  • 1.
  • 2. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. How Disney+ uses fast data ubiquity to improve the customer experience Martin Zapletal Director, Software Engineering Disney+ A N T 3 0 9
  • 3. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Streaming data and Disney+ Before – silos Evolution – streaming silos Now – data driven Streaming Data Platform Examples – ubiquity, platform, culture Agenda
  • 4.
  • 5. • Variety of streaming use cases with varying needs • Dozens of millions of users • Hundreds of Amazon Kinesis data streams • Thousands of shards • Multiple regions • Billions of events • Terabytes of data Streaming data
  • 6. • Microservices • Databases and data warehouses • Batch processing • Slow and limited insights • Silos Before: Silos
  • 7. • Streaming, event driven, asynchronous • Custom, unique integrations and data warehouses Evolution: Streaming silos Amazon Kinesis Data Firehose Amazon Kinesis Data Streams Amazon S3 Amazon RDS Amazon Athena Amazon Redshift Amazon ECS Amazon ECS Amazon ECS AWS Lambda Amazon Kinesis Data Streams Amazon Kinesis Data Firehose Amazon S3 Amazon ECS Amazon Kinesis Data Streams
  • 8. • Streaming, event driven, asynchronous • Custom, unique integrations and data warehouses Evolution: Streaming silos Amazon Kinesis Data Firehose Amazon Kinesis Data Streams Amazon S3 Amazon RDS Amazon Athena Amazon Redshift Amazon ECS Amazon ECS Amazon ECS AWS Lambda Amazon Kinesis Data Streams Amazon Kinesis Data Firehose Amazon S3 Amazon ECS Amazon Kinesis Data Streams Data format 2 Schema management 2 Data quality approach 2 Data governance 2 Tooling 2 … Data format 3 Schema management 3 Data quality approach 3 Data governance 3 Tooling 3 … Data format 1 Schema management 1 Data quality approach 1 Data governance 1 Tooling 1 …
  • 9. • (Fast) data democracy • Real-time data, insights, ML • Experimentation • First-class consideration • Culture “Data / insights they need available at the time they need it” Now: Data driven
  • 10. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Streaming Data Platform
  • 12. Streaming Data Platform Analytics and ML Streaming Data Platform Amazon Kinesis Data Streams Ubiquity Platform Culture Experimentation Services Amazon Kinesis Data Firehose AWS SDK, KPL, KCL AWS Lambda Databricks / Spark Amazon Kinesis Data Analytics for Apache Flink
  • 13. • Need a reliable, performant, cost-efficient event log • Kinesis, Kafka, Pulsar, and others • Amazon Kinesis Data Streams § Replicated, partitioned, ordered, distributed log § Managed § Replication to 3 AZs § Integration with other AWS services § Near real time § Scalability § Elasticity Kinesis
  • 15. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Ubiquity
  • 16. Data management Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
  • 17. Data management Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
  • 18. Schema registry Data management Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
  • 19. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Platform
  • 20. Platform Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
  • 21. Reliable domain events Platform Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
  • 22. Validation, routing, filtering Platform Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
  • 23. Join, enrichment Platform Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
  • 24. Ingestion Platform Amazon ECS Amazon DynamoDB Amazon ElastiCache Kinesis Data Streams DynamoDB Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Kinesis Data Streams Amazon ECS Amazon ECS Databricks / Spark Amazon S3 Databricks / Spark Amazon S3 Kinesis Data Streams Amazon S3 Databricks / Spark Kinesis Data Streams Kinesis Data Firehose Amazon ES Amazon Kinesis Data Analytics for Apache Flink Kinesis Data Streams
  • 25. Streaming application maturity § Architecture patterns § Automated testing § Performance testing and management § Elasticity and auto scaling § Deployment § Observability, alerting § Reliability and resilience § Operations simplicity § Multi-region replication and failover § Data lineage § Self-healing § Distributed tracing § Cost efficiency § Discoverability § Traffic routing § Guarantees § Streaming as a service platform § Etc. Platform
  • 26. • Configurable trade-offs • Latency management • Deployment patterns Platform
  • 27. • Stream elasticity • Application elasticity • Elasticity trade-offs Platform
  • 28. • Reliability • Delivery semantics • End-to-end management • Failure scenarios Platform
  • 29. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Culture
  • 30. Culture Training Collaboration Tooling Ease of use Integrations
  • 31. • Data-driven organization and data democracy • Ubiquitous data • Streaming data platform • Culture and tools • Build on top of Amazon Kinesis Conclusion
  • 32. Resources § Disney Technology Blog – https://medium.com/disney-streaming § Delivering data in real-time via auto scaling Kinesis streams – https://medium.com/disney-streaming/delivering-data-in- real-time-via-auto-scaling-kinesis-streams-72a0236b2cd9 § Testing asynchronous pipelines with fs2 and weaver-test – https://medium.com/disney-streaming/testing-asynchronous- pipelines-with-fs2-and-weaver-test-f0ffd37676d § Open source project weaver-test – https://github.com/disneystreaming/weaver-test/ Credits and resources Credits • Tom LeRoux • Christian Villoslada • Petr Zapletal • Nick Burkard • Matt Jankowski • Ben Morris • Jess Geddes • Daniel Spiewak • Diego Pineda • Eric Meisel • Anthony Garo • Benoit Louy • Mark Harrison • Evan Kaplan • Olivier Melois • Rekha Bachwani • User Services team • Subscription team • Streaming Data Platform team • Data Engineering team • API Services team • Data Governance & Instrumentation team • Experimentation team • And the whole Disney+ team!
  • 33. Thank you! © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Martin Zapletal Twitter @zapletal_martin LinkedIn martinzapletal
  • 34. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.