Submit Search
Upload
Building Effective Near-Real-Time Analytics with Spark Streaming and Kudu
•
3 likes
•
1,183 views
Jeremy Beard
Follow
Presented at the New York Hadoop User Group, May 2016
Read less
Read more
Software
Report
Share
Report
Share
1 of 23
Download Now
Download to read offline
Recommended
Using Kafka and Kudu for fast, low-latency SQL analytics on streaming data
Using Kafka and Kudu for fast, low-latency SQL analytics on streaming data
Mike Percy
Building Efficient Pipelines in Apache Spark
Building Efficient Pipelines in Apache Spark
Jeremy Beard
Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize...
Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize...
Data Con LA
High concurrency, Low latency analytics using Spark/Kudu
High concurrency, Low latency analytics using Spark/Kudu
Chris George
Introduction to Apache Kudu
Introduction to Apache Kudu
Shravan (Sean) Pabba
Exponea - Kafka and Hadoop as components of architecture
Exponea - Kafka and Hadoop as components of architecture
MartinStrycek
A Closer Look at Apache Kudu
A Closer Look at Apache Kudu
Andriy Zabavskyy
cloudera Apache Kudu Updatable Analytical Storage for Modern Data Platform
cloudera Apache Kudu Updatable Analytical Storage for Modern Data Platform
Rakuten Group, Inc.
More Related Content
What's hot
Intro to Apache Kudu (short) - Big Data Application Meetup
Intro to Apache Kudu (short) - Big Data Application Meetup
Mike Percy
Introduction to Apache Kudu
Introduction to Apache Kudu
Jeff Holoman
A brave new world in mutable big data relational storage (Strata NYC 2017)
A brave new world in mutable big data relational storage (Strata NYC 2017)
Todd Lipcon
Low latency high throughput streaming using Apache Apex and Apache Kudu
Low latency high throughput streaming using Apache Apex and Apache Kudu
DataWorks Summit
Apache Kudu (Incubating): New Hadoop Storage for Fast Analytics on Fast Data ...
Apache Kudu (Incubating): New Hadoop Storage for Fast Analytics on Fast Data ...
Cloudera, Inc.
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
Data Con LA
Apache Flink & Kudu: a connector to develop Kappa architectures
Apache Flink & Kudu: a connector to develop Kappa architectures
Nacho García Fernández
SQL Engines for Hadoop - The case for Impala
SQL Engines for Hadoop - The case for Impala
markgrover
Data Science at Scale Using Apache Spark and Apache Hadoop
Data Science at Scale Using Apache Spark and Apache Hadoop
Cloudera, Inc.
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoop
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoop
jdcryans
Intro to Apache Spark
Intro to Apache Spark
Cloudera, Inc.
Application Architectures with Hadoop
Application Architectures with Hadoop
hadooparchbook
Introducing Kudu
Introducing Kudu
Jeremy Beard
Enabling the Active Data Warehouse with Apache Kudu
Enabling the Active Data Warehouse with Apache Kudu
Grant Henke
Introduction to Kudu: Hadoop Storage for Fast Analytics on Fast Data - Rüdige...
Introduction to Kudu: Hadoop Storage for Fast Analytics on Fast Data - Rüdige...
Dataconomy Media
Introducing Kudu, Big Data Warehousing Meetup
Introducing Kudu, Big Data Warehousing Meetup
Caserta
Kudu - Fast Analytics on Fast Data
Kudu - Fast Analytics on Fast Data
Ryan Bosshart
Kudu: New Hadoop Storage for Fast Analytics on Fast Data
Kudu: New Hadoop Storage for Fast Analytics on Fast Data
Cloudera, Inc.
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
Yahoo Developer Network
Introducing Apache Kudu (Incubating) - Montreal HUG May 2016
Introducing Apache Kudu (Incubating) - Montreal HUG May 2016
Mladen Kovacevic
What's hot
(20)
Intro to Apache Kudu (short) - Big Data Application Meetup
Intro to Apache Kudu (short) - Big Data Application Meetup
Introduction to Apache Kudu
Introduction to Apache Kudu
A brave new world in mutable big data relational storage (Strata NYC 2017)
A brave new world in mutable big data relational storage (Strata NYC 2017)
Low latency high throughput streaming using Apache Apex and Apache Kudu
Low latency high throughput streaming using Apache Apex and Apache Kudu
Apache Kudu (Incubating): New Hadoop Storage for Fast Analytics on Fast Data ...
Apache Kudu (Incubating): New Hadoop Storage for Fast Analytics on Fast Data ...
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
Big Data Day LA 2016/ NoSQL track - Apache Kudu: Fast Analytics on Fast Data,...
Apache Flink & Kudu: a connector to develop Kappa architectures
Apache Flink & Kudu: a connector to develop Kappa architectures
SQL Engines for Hadoop - The case for Impala
SQL Engines for Hadoop - The case for Impala
Data Science at Scale Using Apache Spark and Apache Hadoop
Data Science at Scale Using Apache Spark and Apache Hadoop
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoop
Kudu: Resolving Transactional and Analytic Trade-offs in Hadoop
Intro to Apache Spark
Intro to Apache Spark
Application Architectures with Hadoop
Application Architectures with Hadoop
Introducing Kudu
Introducing Kudu
Enabling the Active Data Warehouse with Apache Kudu
Enabling the Active Data Warehouse with Apache Kudu
Introduction to Kudu: Hadoop Storage for Fast Analytics on Fast Data - Rüdige...
Introduction to Kudu: Hadoop Storage for Fast Analytics on Fast Data - Rüdige...
Introducing Kudu, Big Data Warehousing Meetup
Introducing Kudu, Big Data Warehousing Meetup
Kudu - Fast Analytics on Fast Data
Kudu - Fast Analytics on Fast Data
Kudu: New Hadoop Storage for Fast Analytics on Fast Data
Kudu: New Hadoop Storage for Fast Analytics on Fast Data
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
February 2016 HUG: Apache Kudu (incubating): New Apache Hadoop Storage for Fa...
Introducing Apache Kudu (Incubating) - Montreal HUG May 2016
Introducing Apache Kudu (Incubating) - Montreal HUG May 2016
Viewers also liked
Moving Beyond Lambda Architectures with Apache Kudu
Moving Beyond Lambda Architectures with Apache Kudu
Cloudera, Inc.
Apache Spark streaming and HBase
Apache Spark streaming and HBase
Carol McDonald
Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...
Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...
Spark Summit
Huohua: A Distributed Time Series Analysis Framework For Spark
Huohua: A Distributed Time Series Analysis Framework For Spark
Jen Aman
Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...
Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...
Spark Summit
Intro to Apache Spark
Intro to Apache Spark
Mammoth Data
Alpine academy apache spark series #1 introduction to cluster computing wit...
Alpine academy apache spark series #1 introduction to cluster computing wit...
Holden Karau
Sa introduction to big data pipelining with cassandra & spark west mins...
Sa introduction to big data pipelining with cassandra & spark west mins...
Simon Ambridge
Rethinking Streaming Analytics For Scale
Rethinking Streaming Analytics For Scale
Helena Edelson
How to deploy Apache Spark to Mesos/DCOS
How to deploy Apache Spark to Mesos/DCOS
Legacy Typesafe (now Lightbend)
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...
DataWorks Summit/Hadoop Summit
Akka in Production - ScalaDays 2015
Akka in Production - ScalaDays 2015
Evan Chan
Airstream: Spark Streaming At Airbnb
Airstream: Spark Streaming At Airbnb
Jen Aman
Apache kudu
Apache kudu
Asim Jalis
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
Robert "Chip" Senkbeil
Reactive app using actor model & apache spark
Reactive app using actor model & apache spark
Rahul Kumar
Bulletproof Jobs: Patterns For Large-Scale Spark Processing
Bulletproof Jobs: Patterns For Large-Scale Spark Processing
Spark Summit
Netflix - Productionizing Spark On Yarn For ETL At Petabyte Scale
Netflix - Productionizing Spark On Yarn For ETL At Petabyte Scale
Jen Aman
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
Natalino Busa
Reactive dashboard’s using apache spark
Reactive dashboard’s using apache spark
Rahul Kumar
Viewers also liked
(20)
Moving Beyond Lambda Architectures with Apache Kudu
Moving Beyond Lambda Architectures with Apache Kudu
Apache Spark streaming and HBase
Apache Spark streaming and HBase
Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...
Unified Framework for Real Time, Near Real Time and Offline Analysis of Video...
Huohua: A Distributed Time Series Analysis Framework For Spark
Huohua: A Distributed Time Series Analysis Framework For Spark
Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...
Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...
Intro to Apache Spark
Intro to Apache Spark
Alpine academy apache spark series #1 introduction to cluster computing wit...
Alpine academy apache spark series #1 introduction to cluster computing wit...
Sa introduction to big data pipelining with cassandra & spark west mins...
Sa introduction to big data pipelining with cassandra & spark west mins...
Rethinking Streaming Analytics For Scale
Rethinking Streaming Analytics For Scale
How to deploy Apache Spark to Mesos/DCOS
How to deploy Apache Spark to Mesos/DCOS
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...
Near Real-Time Network Anomaly Detection and Traffic Analysis using Spark bas...
Akka in Production - ScalaDays 2015
Akka in Production - ScalaDays 2015
Airstream: Spark Streaming At Airbnb
Airstream: Spark Streaming At Airbnb
Apache kudu
Apache kudu
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
Reactive app using actor model & apache spark
Reactive app using actor model & apache spark
Bulletproof Jobs: Patterns For Large-Scale Spark Processing
Bulletproof Jobs: Patterns For Large-Scale Spark Processing
Netflix - Productionizing Spark On Yarn For ETL At Petabyte Scale
Netflix - Productionizing Spark On Yarn For ETL At Petabyte Scale
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
Reactive dashboard’s using apache spark
Reactive dashboard’s using apache spark
Similar to Building Effective Near-Real-Time Analytics with Spark Streaming and Kudu
Lambda architecture on Spark, Kafka for real-time large scale ML
Lambda architecture on Spark, Kafka for real-time large scale ML
huguk
Spark One Platform Webinar
Spark One Platform Webinar
Cloudera, Inc.
YARN
YARN
Alex Moundalexis
Lessons Learned: Using Spark and Microservices
Lessons Learned: Using Spark and Microservices
Alexis Seigneurin
Faster Batch Processing with Cloudera 5.7: Hive-on-Spark is ready for production
Faster Batch Processing with Cloudera 5.7: Hive-on-Spark is ready for production
Cloudera, Inc.
Real Time Analytics with Dse
Real Time Analytics with Dse
DataStax Academy
Kafka for DBAs
Kafka for DBAs
Gwen (Chen) Shapira
Effective Spark on Multi-Tenant Clusters
Effective Spark on Multi-Tenant Clusters
DataWorks Summit/Hadoop Summit
Spark introduction and architecture
Spark introduction and architecture
Sohil Jain
Spark introduction and architecture
Spark introduction and architecture
Sohil Jain
Real Time Data Processing Using Spark Streaming
Real Time Data Processing Using Spark Streaming
Hari Shreedharan
Simplifying Hadoop with RecordService, A Secure and Unified Data Access Path ...
Simplifying Hadoop with RecordService, A Secure and Unified Data Access Path ...
Cloudera, Inc.
Leveraging Databricks for Spark Pipelines
Leveraging Databricks for Spark Pipelines
Rose Toomey
Leveraging Databricks for Spark pipelines
Leveraging Databricks for Spark pipelines
Rose Toomey
Building High-Throughput, Low-Latency Pipelines in Kafka
Building High-Throughput, Low-Latency Pipelines in Kafka
confluent
Apache Spark Fundamentals
Apache Spark Fundamentals
Zahra Eskandari
Event Detection Pipelines with Apache Kafka
Event Detection Pipelines with Apache Kafka
DataWorks Summit
Building a Hadoop Data Warehouse with Impala
Building a Hadoop Data Warehouse with Impala
Swiss Big Data User Group
Decoupling Decisions with Apache Kafka
Decoupling Decisions with Apache Kafka
Grant Henke
Building a Hadoop Data Warehouse with Impala
Building a Hadoop Data Warehouse with Impala
huguk
Similar to Building Effective Near-Real-Time Analytics with Spark Streaming and Kudu
(20)
Lambda architecture on Spark, Kafka for real-time large scale ML
Lambda architecture on Spark, Kafka for real-time large scale ML
Spark One Platform Webinar
Spark One Platform Webinar
YARN
YARN
Lessons Learned: Using Spark and Microservices
Lessons Learned: Using Spark and Microservices
Faster Batch Processing with Cloudera 5.7: Hive-on-Spark is ready for production
Faster Batch Processing with Cloudera 5.7: Hive-on-Spark is ready for production
Real Time Analytics with Dse
Real Time Analytics with Dse
Kafka for DBAs
Kafka for DBAs
Effective Spark on Multi-Tenant Clusters
Effective Spark on Multi-Tenant Clusters
Spark introduction and architecture
Spark introduction and architecture
Spark introduction and architecture
Spark introduction and architecture
Real Time Data Processing Using Spark Streaming
Real Time Data Processing Using Spark Streaming
Simplifying Hadoop with RecordService, A Secure and Unified Data Access Path ...
Simplifying Hadoop with RecordService, A Secure and Unified Data Access Path ...
Leveraging Databricks for Spark Pipelines
Leveraging Databricks for Spark Pipelines
Leveraging Databricks for Spark pipelines
Leveraging Databricks for Spark pipelines
Building High-Throughput, Low-Latency Pipelines in Kafka
Building High-Throughput, Low-Latency Pipelines in Kafka
Apache Spark Fundamentals
Apache Spark Fundamentals
Event Detection Pipelines with Apache Kafka
Event Detection Pipelines with Apache Kafka
Building a Hadoop Data Warehouse with Impala
Building a Hadoop Data Warehouse with Impala
Decoupling Decisions with Apache Kafka
Decoupling Decisions with Apache Kafka
Building a Hadoop Data Warehouse with Impala
Building a Hadoop Data Warehouse with Impala
Recently uploaded
Unlocking AI:Navigating Open Source vs. Commercial Frontiers
Unlocking AI:Navigating Open Source vs. Commercial Frontiers
Raphaël Semeteys
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
University of Antwerp
Einstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdf
CloudMetic
8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdf
OffsiteNOC
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
telebusocialmarketin
Revolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM Grid
Mathew Thomas
VuNet software organisation powerpoint deck
VuNet software organisation powerpoint deck
Naval Singh
Practical Advice for FDA’s 510(k) Requirements.pdf
Practical Advice for FDA’s 510(k) Requirements.pdf
ICS
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
Priyadarshini T
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Inc
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Randy Shoup
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
jackiepotts6
Enterprise Content Managements Solutions
Enterprise Content Managements Solutions
IQBG inc
openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scale
Shane Coughlan
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
MyFAA
20140812 - OBD2 Solution
20140812 - OBD2 Solution
Jamie (Taka) Wang
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to start
Maxim Salnikov
Mobile App Development company Houston
Mobile App Development company Houston
jennysmithusa549
User Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller Resume
Kaylee Miller
Mobile App Development process | Expert Tips
Mobile App Development process | Expert Tips
michealwillson701
Recently uploaded
(20)
Unlocking AI:Navigating Open Source vs. Commercial Frontiers
Unlocking AI:Navigating Open Source vs. Commercial Frontiers
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
Einstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdf
8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdf
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Revolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM Grid
VuNet software organisation powerpoint deck
VuNet software organisation powerpoint deck
Practical Advice for FDA’s 510(k) Requirements.pdf
Practical Advice for FDA’s 510(k) Requirements.pdf
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
Enterprise Content Managements Solutions
Enterprise Content Managements Solutions
openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scale
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
20140812 - OBD2 Solution
20140812 - OBD2 Solution
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to start
Mobile App Development company Houston
Mobile App Development company Houston
User Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller Resume
Mobile App Development process | Expert Tips
Mobile App Development process | Expert Tips
Building Effective Near-Real-Time Analytics with Spark Streaming and Kudu
1.
1© Cloudera, Inc.
All rights reserved. Building Effective Near-Real-Time Analytics with Spark Streaming and Kudu Jeremy Beard | Senior Solutions Architect, Cloudera May 2016
2.
2© Cloudera, Inc.
All rights reserved. Myself • Jeremy Beard • Senior Solutions Architect at Cloudera • 3.5 years at Cloudera • 6 years data warehousing before that • jeremy@cloudera.com
3.
3© Cloudera, Inc.
All rights reserved. Agenda • What do we mean by near-real-time analytics? • Which components can we use from the Cloudera stack? • How do these components fit together? • How do we implement the Spark Streaming to Kudu path? • What if I don’t want to write all that code?
4.
4© Cloudera, Inc.
All rights reserved. Defining near-real-time analytics (for this talk) • Ability to analyze events happening right now in the real world • And in the context of all the history that has gone before it • By “near” we mean this is human scale (seconds), not machine scale (ns/us) • Closer to real time is possible in CDH, but is more custom development • SQL is the lingua franca of analytics • Millions of people know it or the tools that run on it • Say what you want to get not how you want to get it
5.
5© Cloudera, Inc.
All rights reserved. Components from the Cloudera stack • Four components come together to make this possible • Apache Kafka • Apache Spark • Apache Kudu (incubating) • Apache Impala (incubating) • First we’ll discuss what they are, then how they fit together
6.
6© Cloudera, Inc.
All rights reserved. Apache Kafka • Publish-subscribe system • Publish messages into topics • Subscribe to messages arriving in topics • Very high throughput • Very low latency • Distributed for fault tolerance and scale • Supported by Cloudera
7.
7© Cloudera, Inc.
All rights reserved. Apache Spark • Modern distributed data processing engine • Heavy utilizer of memory for speed • Rich and intuitive API • Spark Streaming • Module for running a continuous loop of Spark transformations • Each iteration is a micro-batch, usually in the single-digit seconds • Supported by Cloudera (with some exceptions for experimental features)
8.
8© Cloudera, Inc.
All rights reserved. Apache Kudu (incubating) • New open-source columnar storage layer • Data model of tables with finite typed columns • Very fast random I/O • Very fast scans • Developed from scratch in C++ • Client APIs for C++, Java, Python • First developed in Cloudera, now at Apache Software Foundation • Currently in beta, not yet supported by Cloudera, not production ready
9.
9© Cloudera, Inc.
All rights reserved. Apache Impala (incubating) • Open-source SQL query engine • Built for one purpose: really fast analytics SQL • High concurrency • Queries data stored in HDFS, HBase, and now Kudu • Standard JDBC/ODBC interface for SQL editors and BI tools • Uses JIT query compilation and modern CPU instructions • First developed in Cloudera, now at Apache Software Foundation • Fully supported by Cloudera and in production at many of our customers
10.
10© Cloudera, Inc.
All rights reserved. Near-real-time analytics on the Cloudera stack
11.
11© Cloudera, Inc.
All rights reserved. Implementing Spark Streaming to Kudu • We define what we want Spark to do each micro-batch • Spark then takes care of running the micro-batches for us • We have limited time to process a micro-batch • Storage lookups must be key lookups or very short scans • A lot of repetitive boilerplate code to get up and running
12.
12© Cloudera, Inc.
All rights reserved. Typical stages of a Spark Streaming to Kudu pipeline • Sourcing from a queue of data • Translating into a structured format • Deriving the storage records • Planning how to update the storage layer • Applying the planned mutations to the storage layer
13.
13© Cloudera, Inc.
All rights reserved. Queue sourcing • Each micro-batch we first have to bring in data to process • This is near-real-time, so we expect a queue of messages waiting to be processed • Kafka fits this requirement very well • Native no-data-loss integration with Spark Streaming • Partitioned topics automatically parallelize across Spark executors • Fault recovery simple because Kafka does not drop consumed messages • In Spark Streaming this is the creation of a DStream object • For Kafka use KafkaUtils.createDirectStream()
14.
14© Cloudera, Inc.
All rights reserved. Translation • Arriving messages could be in any format (XML, CSV, binary, proprietary, etc.) • We need them in a common structured record format to effectively transform it • When messages arrive, translate them first • Avro’s GenericRecord is a widely adopted in-memory record format • In Spark Streaming job use DStream.map() to define translation
15.
15© Cloudera, Inc.
All rights reserved. Derivation • We need to create the records that we want to write to the storage layer • Often not identical to the arriving records • Derive the storage records from the arriving records • Spark SQL can define transformation, but much more plumbing code required • May also require deriving from existing records in the storage layer • Enrichment using reference data is a common example
16.
16© Cloudera, Inc.
All rights reserved. Planning • With derived storage records in hand we need to plan the storage mutations • When existing records are never updated it is straight-forward • Just plan inserts • When updates for a key can occur it is a bit harder • Plan insert if key does not exist, plan update if key does exist • When all versions of a key are kept it can be a lot more complicated • Insert arriving record, update metadata on existing records (e.g. end date)
17.
17© Cloudera, Inc.
All rights reserved. Storing • With the planned mutations for the micro-batch, we apply them to the storage • For Kudu this requires using the Kudu client Java API • Applied mutations are immediately visible to Impala users • Use RDD.forEachPartition() so that you can open a Kudu connection per JVM • Alternatively write to Solr, can be a good option where SQL is not required • Alternatively write to HBase, but storage is too slow for analytics queries • Alternatively write to HDFS, but storage does not support updates or deletes
18.
18© Cloudera, Inc.
All rights reserved. Performance considerations • Repartition the arriving records across all the cores of the Spark job • If using Spark SQL, lower the number of shuffle partitions from default 200 • Use Spark Streaming backpressure to optimize micro-batch size • If using Kafka, also use spark.streaming.kafka.maxRatePerPartition • Experiment with micro-batch lengths to balance latency vs. throughput • Ensure storage lookup predicates are at least by key, or face full table scans • Avoid connecting and disconnecting from storage every micro-batch • Singleton pattern can help to keep a connection per JVM • Avoid instantiating objects for each record where they could be reused • Batch mutations for higher throughput
19.
19© Cloudera, Inc.
All rights reserved. New on Cloudera Labs: Envelope • A pre-developed Spark Streaming application that implements these stages • Pipelines are defined as simple configuration using a properties file • Custom implementations of stages can be referenced in the configuration • Available on Cloudera Labs (cloudera.com/labs) • Not supported by Cloudera, not production ready
20.
20© Cloudera, Inc.
All rights reserved. Envelope built-in functionality • Queue source for Kafka • Translators for delimited text, key-value pairs, and binary Avro • Lookup of existing storage records • Deriver for Spark SQL transformations • Planners for appends, upserts, and history tracking • Storage system for Kudu • Support for many of the described performance considerations • All stage implementations are also pluggable with user-provided classes
21.
21© Cloudera, Inc.
All rights reserved. Example pipeline: Traffic
22.
22© Cloudera, Inc.
All rights reserved. Example pipeline: FIX
23.
23© Cloudera, Inc.
All rights reserved. Thank you jeremy@cloudera.com
Download Now