This document discusses building data pipelines for both static and streaming data using Apache Spark and DataStax Enterprise (DSE). For static data, it recommends using optimized data storage formats, distributed and scalable technologies like Spark, interactive analysis tools like notebooks, and DSE for persistent storage. For streaming data, it recommends using scalable distributed technologies, Kafka to decouple producers and consumers, and DSE for real-time analytics and persistent storage across datacenters.

1. Simon Ambridge
Data Pipelines With Spark & DSE
An Introduction To Building Agile, Flexible and Scalable Big Data and Data Science Pipelines
Version 0.8

2. Certified Apache Cassandra and DataStax enthusiast who enjoys
explaining that the traditional approaches to data management just
don’t cut it anymore in the new always on, no single point of failure,
high volume, high velocity, real time distributed data management
world.
Previously 25 years implementing Oracle relational data management
solutions. Certified in Exadata, Oracle Cloud, Oracle Essbase, Oracle
Linux and OBIEE
simon.ambridge@datastax.com
@stratman1958
Simon Ambridge
Pre-Sales Solution Engineer, Datastax UK

3. Introduction To Big Data Pipelines

4. Big, Static Data
Fast, Streaming Data
Big Data Pipelining: Classification
Big Data Pipelines can mean different things to different people
Repeated analysis on a static but massive dataset
• An element of research – e.g. genomics, clinical trial,
demographic data
• Typically repetitive, iterative, shared amongst data
scientists for analysis
Real-time analytics on streaming data
• Industrialised or commercial processes – sensors, tick
data, bioinformatics, transactional data, real-time
personalisation
• Happening in real-time, data cannot be dropped or lost

5. Static Datasets
All You Can Eat?
Really.

6. Static Data Analytics : Traditional Tools
Repeated iterations, at each stage
Run/debug cycle can be slow
Sampling Modeling InterpretTuning Reporting
Re-sample
Typical traditional ‘static’ data analysis model
Data
Results

7. Static Data Analytics : Scale Up Challenges
Sampling and analysis often run on a single machine
• CPU and memory limitations
Limited sampling of a large dataset because of data size limitations
• Multiple iterations over large datasets is frequently not an ideal
approach

8. Static Data Analytics : Traditional Scaling
DATA (GB)
DATA (MB)
DATA (TB)
Small datasets, small servers
Large datasets, large servers

9. Static Data Analytics: Big Data Problems
Data is getting Really Big!
• Data volumes are getting larger!
• The number of data sources is exploding!
• More data is arriving faster!
Scaling up is becoming impractical
• Physical limits
• Datalimits
• The validity of the analysis becomes obsolete, faster

10. Static Data Analytics : Big Data Needs
We need scalable infrastructure + distributed technologies
• Data volumes can be scaled
• Distribute the data across multiple low-cost machines
• Faster processing
• More complex processing
• No single point of failure

11. Static Data Analytics : DSE Delivers
Building a distributed data processing framework can be a complex task!
It needs to be:
• Scalable
• Fast in-memory processing
• Replicated for resiliency
• Batch and real-time data feeds
• Ad-hoc queries
DataStax delivers an integrated analytics platform

12. Cassandra: THE Web, IoT & Cloud Database
What is Apache Cassandra?
• Very fast
• Extremely resilient
• Across multiple data centres
• No single point of failure
• Continuous Availability, Disaster Avoidance
• Linear scale
• Easy to operate
Enterprise Cassandra platform from Datastax

13. DataStax
Enterprise
DataStax Enterprise: Editions
DataStax Enterprise Standard
• DSE Standard is DataStax’s entry level
commercial database offering
• Represents the minimum recommended to
deploy Cassandra in a production environment
DataStax Enterprise Max
• DSE Max is DataStax’s advanced commercial
database offering
• Designed for production Cassandra
environments that have mixed workload
requirements

14. Spark: THE Analytics Engine
What is Apache Spark?
• Distributed in-memory analytic processing
• Batch and streaming analytics
• Fast - 10x-100x faster than Hadoop MapReduce
• Rich Scala, Java and Python APIs
Tightly integrated with DSE

15. Spark: Dayton Gray Sort Contest
Dayton Gray benchmark - tests how fast a system can sort 100 TB
of data (1 trillion records)
• Previous world record held by Hadoop MapReduce cluster of 2100
nodes, in 72 minutes
• 2014: Spark completed the benchmark in 23 minutes on just 206 EC2
nodes = 3X faster using 10X fewer machines
• Spark sorted 1 PB (10 trillion records) on 190 machines in < 4 hours.
Previous Hadoop MapReduce time of 16 hours on 3800 machines = 4X
faster using 20X fewer machines

16. DataStax Enterprise: Analytics Integration
Cassandra Cluster
Spark Cluster
ETL
Spark Cluster
• Tight integration
• Data locality
• Microsecond response times
X
• Apache Cassandra for Distributed Persistent Storage
• Integrated Apache Spark for Distributed Real-Time Analytics
• Analytics nodes close to data - no ETL required
X
• Loose integration
• Data separate from processing
• Millisecond response times
“Latency
when
transferring
data
is
unavoidable.
The
trick
is
to
reduce
the
latency
to
as
close
to
zero
as
possible…”

17. Static Data Analytics : Requirements
Valid data pipeline analysis methods must be:
Auditable
• Reproducible
• Documented
Controlled
• Version control
Collaborative
• Accessible

18. Notebooks: Features
What are Notebooks?
• Drive your data analysis from the browser
• Highly interactive
• Tight integration with Apache Spark
• Handy tools for analysts:
• Reproducible visual analysis
• Code in Scala, CQL, SparkSQL, Python
• Charting – pie, bar, line etc
• Extensible with custom libraries

19. Example: Spark Notebook
Cells
Markdown
Output
Controls

20. Static Data Analytics : Approach
Example architecture & requirements
1. Optimised source data format
2. Distributed in-memory analytics
3. Interactive and flexible data analysis tool
4. Persistent data store
5. Visualisation tools

21. Static Data Analytics : Example
ADAM
Notebook Persistent Storage
OLTP Database Visualisation
Genome research platform - ADST (Agile Data Science Toolkit)

22. Static Data Analytics : Pipeline Process Flow
3. Persistent data storage
2. Interactive, flexible and reproducible analysis
1. Source data
4. Visualise and analyse

23. Static Data Analytics : Pipeline Scalability
• Add more (physical or virtual) nodes as
required to add capacity
• Container tools ease configuration
management and deployment
• Scale out quickly

24. Static Data Analytics : Now
• No longer an iterative process constrained by hardware limitations
• Now a more scalable, resilient, dynamic, interactive process, easily shareable
Analyse
The new model for large-scale static data analytics
Share
X
Load
SCALE & DISTRIBUTE PROCESSING

25. Real-Time Datasets
If it’s Not “Now”, Then It’s Probably Already Too Late

26. Big Data Pipelining: Why Real-Time?
• React to customers faster and with more accuracy
• Reduce risk through more accurate understanding of the market
• Optimise return on marketing investment
• Faster time to market
• Improve efficiency
In a highly connected world
In most cases ‘real-time’ data changing at <1s intervals

27. Big Data Pipelining: Real-Time Analytics
• Capture, prepare, and process fast streaming data
• Different approach from traditional batch processing
• The speed of now – cannot wait
• Immediate insight, instant decisions
What problem are we trying to solve?

28. Big Data Pipelining: Real-Time Use Cases
Sensor data (IoT)
Transactional data
User Experience
Social media
Use cases for streaming analytics

29. Big Data Analytics: Streams
Data tidal waves!Netflix
• Ingests Petabytes of data per day
• Over 1 TRILLION transactions per day (>10 m per second) into DSE
Data streams?
Data torrent?

30. Big Data Pipelining: Real-Time architecture
Analytics in real-time, at scale
Fast processing, distributed, in-memory
Increasingly using a technology stack comprising Kafka, Spark and Cassandra
• Scalable
• Distributed
• Resilient
Streaming analytics architecture - what do we need?

31. Kafka: Architecture
How Does Kafka Work?
Kafka “De-couples” producers and consumers in data pipelines
’Producers’ send messages to the Kafka cluster, which in turn serves them up to
’Consumers’
• Kafka maintains feeds of messages in categories called topics
• A Kafka cluster is comprised of one or more servers called a broker
Producer
Producer
Producer
Consumer
Consumer
Consumer
Kafka
Cluster

32. Kafka: Streaming With Spark
Kafka writes, Spark reads
• Topics can have multiple partitions
• Each topic partition stored as a log (an ordered set of messages)
• Messages are simply byte arrays, so can store any object in any format
• Each message in a partition is assigned a unique offset
Spark consumes messages as a stream, in micro batches, saved as RDD’s
1 2 3 4 5 6 7 8
Partition 0
1 2 3 4 5 6 7 8
Partition 1
1 2 3 4 5 6
Partition 0
Temperature Topic
Rainfall Topic
Temperature Consumer
Rainfall Consumer
Temperature Consumer

33. DataStax Enterprise: Streaming Schematic
Sensor
Network
Signal
Aggregation
Services
Messaging Queue
Sensor Data Queue
Management
Broker
Broker
Collection
Service
Data Storage
OLTP PersistenceLayer
Streaming Data
Ingest

34. DataStax Enterprise: Streaming Analytics
Real-time
Analytics
Persistent Storage
OLTP Database
!$£€!
Personalisation
Actionable insight Monitoring
Web / Analytics / BI

35. DataStax Enterprise: Multi-DC Uses
DC: EUROPEDC: USA
Real-time active-active geo-replication
across physical datacentres
4 3
25
1
4 3
25
1
8
1
2
3
4
5
6
7
1
2
3
OLTP:
Cassandra
5
4
Analytics:
Cassandra + Spark
Replication
Replication
Workload separation via virtual datacentres

36. Real-Time Analytics: DSE Multi-DC
Workload Management and Separation With DSE
Analytics / BI
Analytics
Datacentre
OLTP
Datacentre
100% Uptime, Global Scale
OLTP
Real-Time Analytics
Mixed Load OLTP and Analytics Platform
Replication
Replication
JDBC
ODBC
Separation of OLTP
from Analytics
Social Media
IoT
Personalisation & Persistence
Personalisation
!$£€!
Actionable insight
Monitoring
App, Web

37. DSE & Analytics : Summary
Static, Massive Data
Scalable Data Pipelines
1. Optimised data storage formats
2. Scalable, distributed technologies
3. Flexible and interactive analysis tools
4. Resilient, persistent Storage
Real-Time Streaming Data
Scalable Data Pipelines
1. Scalable, distributed technologies
2. De-coupled Producers and Consumers
3. Real-Time analytics
4. Resilient, persistent Storage
Spark
Mesos
Akka
Cassandra
Kafka

39. Thank you!