Patrick McFadin, profile picture
Uploaded byPatrick McFadin
PDF, PPTX3,196 views

Apache cassandra and spark. you got the the lighter, let's start the fire

The document discusses the integration of Apache Cassandra and Apache Spark, highlighting their scalability, resilience, and performance enhancements for data processing. It details the features of both technologies, including the capabilities of Spark's RDDs and the use of the Spark-Cassandra connector for seamless data access. Additionally, it provides technical examples of creating and accessing tables in Cassandra while utilizing Spark for advanced data analysis and manipulation.

Embed presentation

Download as PDF, PPTX
©2013 DataStax Confidential. Do not distribute without consent. @PatrickMcFadin Patrick McFadin
 Chief Evangelist for Apache Cassandra Apache Cassandra and Spark You got the lighter, let’s spark the fire 1
6 years. How’s it going?
Cassandra 3.0 & 3.1 Spring and Fall
Cassandra is… • Shared nothing • Masterless peer-to-peer • Great scaling story • Resilient to failure
Cassandra for Applications APACHE CASSANDRA
A Data Ocean or Pond., Lake An In-Memory Database A Key-Value Store A magical database unicorn that farts rainbows
Apache Spark
Apache Spark • 10x faster on disk,100x faster in memory than Hadoop MR • Works out of the box on EMR • Fault Tolerant Distributed Datasets • Batch, iterative and streaming analysis • In Memory Storage and Disk • Integrates with Most File and Storage Options Up to 100× faster (2-10× on disk) 2-5× less code
Spark Components Spark Core Spark SQL structured Spark Streaming real-time MLlib machine learning GraphX graph
org.apache.spark.rdd.RDD Resilient Distributed Dataset (RDD) •Created through transformations on data (map,filter..) or other RDDs •Immutable •Partitioned •Reusable
RDD Operations •Transformations - Similar to scala collections API •Produce new RDDs •filter, flatmap, map, distinct, groupBy, union, zip, reduceByKey, subtract •Actions •Require materialization of the records to generate a value •collect: Array[T], count, fold, reduce..
Analytic Analytic Search Transformation Action RDD Operations
Cassandra and Spark
Cassandra & Spark: A Great Combo Datastax: spark-cassandra-connector:
 https://github.com/datastax/spark-cassandra-connector •Both are Easy to Use •Spark Can Help You Bridge Your Hadoop and Cassandra Systems •Use Spark Libraries, Caching on-top of Cassandra-stored Data •Combine Spark Streaming with Cassandra Storage
Spark On Cassandra •Server-Side filters (where clauses) •Cross-table operations (JOIN, UNION, etc.) •Data locality-aware (speed) •Data transformation, aggregation, etc. •Natural Time Series Integration
Apache Spark and Cassandra Open Source Stack Cassandra
Spark Cassandra Connector 18
Spark Cassandra Connector *Cassandra tables exposed as Spark RDDs *Read from and write to Cassandra *Mapping of C* tables and rows to Scala objects *All Cassandra types supported and converted to Scala types *Server side data selection *Virtual Nodes support *Use with Scala or Java *Compatible with, Spark 1.1.0, Cassandra 2.1 & 2.0
Type Mapping CQL Type Scala Type ascii String bigint Long boolean Boolean counter Long decimal BigDecimal, java.math.BigDecimal double Double float Float inet java.net.InetAddress int Int list Vector, List, Iterable, Seq, IndexedSeq, java.util.List map Map, TreeMap, java.util.HashMap set Set, TreeSet, java.util.HashSet text, varchar String timestamp Long, java.util.Date, java.sql.Date, org.joda.time.DateTime timeuuid java.util.UUID uuid java.util.UUID varint BigInt, java.math.BigInteger *nullable values Option
Connecting to Cassandra // Import Cassandra-specific functions on SparkContext and RDD objects import com.datastax.driver.spark._ // Spark connection options val conf = new SparkConf(true) .setMaster("spark://192.168.123.10:7077") .setAppName("cassandra-demo") .set("cassandra.connection.host", "192.168.123.10") // initial contact .set("cassandra.username", "cassandra") .set("cassandra.password", "cassandra") val sc = new SparkContext(conf)
Accessing Data CREATE TABLE test.words (word text PRIMARY KEY, count int); INSERT INTO test.words (word, count) VALUES ('bar', 30); INSERT INTO test.words (word, count) VALUES ('foo', 20); // Use table as RDD val rdd = sc.cassandraTable("test", "words") // rdd: CassandraRDD[CassandraRow] = CassandraRDD[0] rdd.toArray.foreach(println) // CassandraRow[word: bar, count: 30] // CassandraRow[word: foo, count: 20] rdd.columnNames // Stream(word, count) rdd.size // 2 val firstRow = rdd.first // firstRow: CassandraRow = CassandraRow[word: bar, count: 30] firstRow.getInt("count") // Int = 30 *Accessing table above as RDD:
Saving Data val newRdd = sc.parallelize(Seq(("cat", 40), ("fox", 50))) // newRdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[2] newRdd.saveToCassandra("test", "words", Seq("word", "count")) SELECT * FROM test.words; word | count ------+------- bar | 30 foo | 20 cat | 40 fox | 50 (4 rows) *RDD above saved to Cassandra:
Spark Cassandra Connector https://github.com/datastax/spark-­‐cassandra-­‐connector Keyspace Table Cassandra Spark RDD[CassandraRow] RDD[Tuples] Bundled  and  Supported  with  DSE  4.5!
Spark Cassandra Connector uses the DataStax Java Driver to Read from and Write to C* Spark C* Full Token Range Each Executor Maintains a connection to the C* Cluster Spark Executor DataStax Java Driver Tokens 1-1000 Tokens 1001 -2000 Tokens … RDD’s read into different splits based on sets of tokens Spark Cassandra Connector
Co-locate Spark and C* for Best Performance C* C*C* C* Spark
 Worker Spark
 Worker Spark Master Spark Worker Running Spark Workers on the same nodes as your C* Cluster will save network hops when reading and writing
Analytics Workload Isolation Cassandra + Spark DC Cassandra Only DC Online App Analytical App Mixed Load Cassandra Cluster
Data Locality
Example 1: Weather Station • Weather station collects data • Cassandra stores in sequence • Application reads in sequence
Data Model • Weather Station Id and Time are unique • Store as many as needed CREATE TABLE temperature ( weather_station text, year int, month int, day int, hour int, temperature double, PRIMARY KEY (weather_station,year,month,day,hour) ); INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘10010:99999’,2005,12,1,7,-5.6); INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘10010:99999’,2005,12,1,8,-5.1); INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘10010:99999’,2005,12,1,9,-4.9); INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘10010:99999’,2005,12,1,10,-5.3);
Primary key relationship PRIMARY KEY (weather_station,year,month,day,hour)
Primary key relationship PRIMARY KEY (weather_station,year,month,day,hour) Partition Key
Primary key relationship PRIMARY KEY (weather_station,year,month,day,hour) Partition Key Clustering Columns
Primary key relationship PRIMARY KEY (weather_station,year,month,day,hour) Partition Key Clustering Columns 10010:99999
2005:12:1:7 -5.6 Primary key relationship PRIMARY KEY (weather_station,year,month,day,hour) Partition Key Clustering Columns 10010:99999 -5.3-4.9-5.1 2005:12:1:8 2005:12:1:9 2005:12:1:10
Partition keys 10010:99999 Murmur3 Hash Token = 7224631062609997448 722266:13850 Murmur3 Hash Token = -6804302034103043898 INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘10010:99999’,2005,12,1,7,-5.6); INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘722266:13850’,2005,12,1,7,-5.6); Consistent hash. 128 bit number between 2-63 and 264
Partition keys 10010:99999 Murmur3 Hash Token = 15 722266:13850 Murmur3 Hash Token = 77 INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘10010:99999’,2005,12,1,7,-5.6); INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘722266:13850’,2005,12,1,7,-5.6); For this example, let’s make it a reasonable number
Writes & WAN replication 10.0.0.1 00-25 10.0.0.4 76-100 10.0.0.2 26-50 10.0.0.3 51-75 DC1 DC1: RF=3 Node Primary Replica Replica 10.0.0.1 00-25 76-100 51-75 10.0.0.2 26-50 00-25 76-100 10.0.0.3 51-75 26-50 00-25 10.0.0.4 76-100 51-75 26-50 10.10.0.1 00-25 10.10.0.4 76-100 10.10.0.2 26-50 10.10.0.3 51-75 DC2 Node Primary Replica Replica 10.0.0.1 00-25 76-100 51-75 10.0.0.2 26-50 00-25 76-100 10.0.0.3 51-75 26-50 00-25 10.0.0.4 76-100 51-75 26-50 DC2: RF=3 Client Insert Data Partition Key = 15 Asynchronous Local Replication Asynchronous WAN Replication
Locality 10.0.0.1 00-25 10.0.0.4 76-100 10.0.0.2 26-50 10.0.0.3 51-75 DC1 DC1: RF=3 Node Primary Replica Replica 10.0.0.1 00-25 76-100 51-75 10.0.0.2 26-50 00-25 76-100 10.0.0.3 51-75 26-50 00-25 10.0.0.4 76-100 51-75 26-50 10.10.0.1 00-25 10.10.0.4 76-100 10.10.0.2 26-50 10.10.0.3 51-75 DC2 Node Primary Replica Replica 10.0.0.1 00-25 76-100 51-75 10.0.0.2 26-50 00-25 76-100 10.0.0.3 51-75 26-50 00-25 10.0.0.4 76-100 51-75 26-50 DC2: RF=3 Client Get Data Partition Key = 15 Client Get Data Partition Key = 15
Data Locality weatherstation_id=‘10010:99999’ ? 1000 Node Cluster You are here!
Spark Reads on Cassandra Awesome animation by DataStax’s own Russel Spitzer
Spark RDDs Represent a Large Amount of Data Partitioned into Chunks RDD 1 2 3 4 5 6 7 8 9Node 2 Node 1 Node 3 Node 4
Node 2 Node 1 Spark RDDs Represent a Large Amount of Data Partitioned into Chunks RDD 2 346 7 8 9 Node 3 Node 4 1 5
Node 2 Node 1 RDD 2 346 7 8 9 Node 3 Node 4 1 5 Spark RDDs Represent a Large Amount of Data Partitioned into Chunks
Cassandra Data is Distributed By Token Range
Cassandra Data is Distributed By Token Range 0 500
Cassandra Data is Distributed By Token Range 0 500 999
Cassandra Data is Distributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4
Cassandra Data is Distributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4 Without vnodes
Cassandra Data is Distributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4 With vnodes
Node 1 120-220 300-500 780-830 0-50 spark.cassandra.input.split.size 50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions
Node 1 120-220 300-500 0-50 spark.cassandra.input.split.size 50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 1 780-830
1 Node 1 120-220 300-500 0-50 spark.cassandra.input.split.size 50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830
2 1 Node 1 300-500 0-50 spark.cassandra.input.split.size 50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830
2 1 Node 1 300-500 0-50 spark.cassandra.input.split.size 50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830
2 1 Node 1 300-400 0-50 spark.cassandra.input.split.size 50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 400-500
21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 400-500
21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 400-500 3
21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3 400-500
21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3
4 21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3
4 21 Node 1 0-50 spark.cassandra.input.split.size 50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3
421 Node 1 spark.cassandra.input.split.size 50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 3
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50780-830 Node 1
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
4 spark.cassandra.input.page.row.size 50 Data is Retrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
Co-locate Spark and C* for Best Performance C* C*C* C* Spark
 Worker Spark
 Worker Spark Master Spark Worker Running Spark Workers on the same nodes as your C* Cluster will save network hops when reading and writing
Weather Station Analysis • Weather station collects data • Cassandra stores in sequence • Spark rolls up data into new tables Windsor California July 1, 2014 High: 73.4 Low : 51.4
Roll-up table(SparkSQL example) CREATE TABLE daily_high_low ( weatherstation text, date text, high_temp double, low_temp double, PRIMARY KEY ((weatherstation,date)) ); • Weather Station Id and Date are unique • High and low temp for each day SparkSQL> INSERT INTO TABLE > daily_high_low > SELECT > weatherstation, to_date(year, day, hour) date, max(temperature) high_temp, min(temperature) low_temp > FROM temperature > GROUP BY weatherstation_id, year, month, day; OK Time taken: 2.345 seconds functions aggregations
What just happened • Data is read from temperature table • Transformed • Inserted into the daily_high_low table Table: temperature Table: daily_high_low Read data from table Transform Insert data into table
Spark Streaming
zillions of bytes gigabytes per second Spark Versus Spark Streaming
Analytic Analytic Search Spark Streaming Kinesis,'S3'
DStream - Micro Batches μBatch (ordinary RDD) μBatch (ordinary RDD) μBatch (ordinary RDD) Processing of DStream = Processing of μBatches, RDDs DStream • Continuous sequence of micro batches • More complex processing models are possible with less effort • Streaming computations as a series of deterministic batch computations on small time intervals
Spark Streaming Example val conf = new SparkConf(loadDefaults = true) .set("spark.cassandra.connection.host", "127.0.0.1") .setMaster("spark://127.0.0.1:7077") val sc = new SparkContext(conf) val table: CassandraRDD[CassandraRow] = sc.cassandraTable("keyspace", "tweets") 
 val ssc = new StreamingContext(sc, Seconds(30))
 
 val stream = KafkaUtils.createStream[String, String, StringDecoder, StringDecoder](
 ssc, kafka.kafkaParams, Map(topic -> 1), StorageLevel.MEMORY_ONLY)
 
 stream.map(_._2).countByValue().saveToCassandra("demo", "wordcount")
 
 ssc.start()
 ssc.awaitTermination() Initialization Transformations and Action CassandraRDD Stream Initialization
Now what? Cassandra Only DC Cassandra + Spark DC Spark Jobs Spark Streaming
You can do this at home! https://github.com/killrweather/killrweather On your USB!
Thank you! Bring the questions Follow me on twitter @PatrickMcFadin

More Related Content

PDF
An Introduction to time series with Team Apache
byPatrick McFadin
 
PDF
Nike Tech Talk: Double Down on Apache Cassandra and Spark
byPatrick McFadin
 
PDF
Laying down the smack on your data pipelines
byPatrick McFadin
 
PDF
Owning time series with team apache Strata San Jose 2015
byPatrick McFadin
 
PDF
Successful Architectures for Fast Data
byPatrick McFadin
 
PDF
Cassandra Fundamentals - C* 2.0
byRussell Spitzer
 
PDF
Time series with apache cassandra strata
byPatrick McFadin
 
PDF
Cassandra and Spark: Optimizing for Data Locality-(Russell Spitzer, DataStax)
bySpark Summit
 
An Introduction to time series with Team Apache
byPatrick McFadin
 
Nike Tech Talk: Double Down on Apache Cassandra and Spark
byPatrick McFadin
 
Laying down the smack on your data pipelines
byPatrick McFadin
 
Owning time series with team apache Strata San Jose 2015
byPatrick McFadin
 
Successful Architectures for Fast Data
byPatrick McFadin
 
Cassandra Fundamentals - C* 2.0
byRussell Spitzer
 
Time series with apache cassandra strata
byPatrick McFadin
 
Cassandra and Spark: Optimizing for Data Locality-(Russell Spitzer, DataStax)
bySpark Summit
 

What's hot

PDF
Time series with Apache Cassandra - Long version
byPatrick McFadin
 
PDF
A Cassandra + Solr + Spark Love Triangle Using DataStax Enterprise
byPatrick McFadin
 
PDF
Introduction to data modeling with apache cassandra
byPatrick McFadin
 
PDF
Lessons from Cassandra & Spark (Matthias Niehoff & Stephan Kepser, codecentri...
byDataStax
 
PDF
Cassandra 2.0 and timeseries
byPatrick McFadin
 
PDF
Introduction to cassandra 2014
byPatrick McFadin
 
PDF
Analytics with Cassandra & Spark
byMatthias Niehoff
 
PPTX
Analytics with Cassandra, Spark & MLLib - Cassandra Essentials Day
byMatthias Niehoff
 
PDF
Spark and Cassandra 2 Fast 2 Furious
byRussell Spitzer
 
PDF
Maximum Overdrive: Tuning the Spark Cassandra Connector (Russell Spitzer, Dat...
byDataStax
 
PDF
Spark cassandra connector.API, Best Practices and Use-Cases
byDuyhai Doan
 
PDF
Zero to Streaming: Spark and Cassandra
byRussell Spitzer
 
PDF
Big data analytics with Spark & Cassandra
byMatthias Niehoff
 
PPTX
Maximum Overdrive: Tuning the Spark Cassandra Connector
byRussell Spitzer
 
PDF
Cassandra Basics, Counters and Time Series Modeling
byVassilis Bekiaris
 
PDF
Spark Streaming with Cassandra
byJacek Lewandowski
 
PDF
Escape From Hadoop: Spark One Liners for C* Ops
byRussell Spitzer
 
PDF
Cassandra and Spark: Optimizing for Data Locality
byRussell Spitzer
 
PDF
Spark Cassandra Connector: Past, Present, and Future
byRussell Spitzer
 
PDF
Apache Spark and DataStax Enablement
byVincent Poncet
 
Time series with Apache Cassandra - Long version
byPatrick McFadin
 
A Cassandra + Solr + Spark Love Triangle Using DataStax Enterprise
byPatrick McFadin
 
Introduction to data modeling with apache cassandra
byPatrick McFadin
 
Lessons from Cassandra & Spark (Matthias Niehoff & Stephan Kepser, codecentri...
byDataStax
 
Cassandra 2.0 and timeseries
byPatrick McFadin
 
Introduction to cassandra 2014
byPatrick McFadin
 
Analytics with Cassandra & Spark
byMatthias Niehoff
 
Analytics with Cassandra, Spark & MLLib - Cassandra Essentials Day
byMatthias Niehoff
 
Spark and Cassandra 2 Fast 2 Furious
byRussell Spitzer
 
Maximum Overdrive: Tuning the Spark Cassandra Connector (Russell Spitzer, Dat...
byDataStax
 
Spark cassandra connector.API, Best Practices and Use-Cases
byDuyhai Doan
 
Zero to Streaming: Spark and Cassandra
byRussell Spitzer
 
Big data analytics with Spark & Cassandra
byMatthias Niehoff
 
Maximum Overdrive: Tuning the Spark Cassandra Connector
byRussell Spitzer
 
Cassandra Basics, Counters and Time Series Modeling
byVassilis Bekiaris
 
Spark Streaming with Cassandra
byJacek Lewandowski
 
Escape From Hadoop: Spark One Liners for C* Ops
byRussell Spitzer
 
Cassandra and Spark: Optimizing for Data Locality
byRussell Spitzer
 
Spark Cassandra Connector: Past, Present, and Future
byRussell Spitzer
 
Apache Spark and DataStax Enablement
byVincent Poncet
 

Viewers also liked

PDF
Seattle C* Meetup: Hardening cassandra for compliance or paranoia
byzznate
 
PDF
Making money with open source and not losing your soul: A practical guide
byPatrick McFadin
 
PDF
Cassandra EU - Data model on fire
byPatrick McFadin
 
PDF
Cassandra Meetup: Real-time Analytics using Cassandra, Spark and Shark at Ooyala
byDataStax Academy
 
PPTX
Running 400-node Cassandra + Spark Clusters in Azure (Anubhav Kale, Microsoft...
byDataStax
 
PDF
Real data models of silicon valley
byPatrick McFadin
 
PDF
Cassandra 3.0 advanced preview
byPatrick McFadin
 
PDF
Cassandra nice use cases and worst anti patterns
byDuyhai Doan
 
PDF
Open source or proprietary, choose wisely!
byPatrick McFadin
 
PDF
Traveler's Guide to Cassandra
byDataStax Academy
 
PDF
Storing time series data with Apache Cassandra
byPatrick McFadin
 
PPTX
Cassandra concepts, patterns and anti-patterns
byDave Gardner
 
PDF
Advanced data modeling with apache cassandra
byPatrick McFadin
 
PDF
Reactive dashboard’s using apache spark
byRahul Kumar
 
PDF
Webinar - Bringing Game Changing Insights with Graph Databases
byDataStax
 
PDF
Cassandra Introduction & Features
byDataStax Academy
 
PDF
Apache cassandra & apache spark for time series data
byPatrick McFadin
 
Seattle C* Meetup: Hardening cassandra for compliance or paranoia
byzznate
 
Making money with open source and not losing your soul: A practical guide
byPatrick McFadin
 
Cassandra EU - Data model on fire
byPatrick McFadin
 
Cassandra Meetup: Real-time Analytics using Cassandra, Spark and Shark at Ooyala
byDataStax Academy
 
Running 400-node Cassandra + Spark Clusters in Azure (Anubhav Kale, Microsoft...
byDataStax
 
Real data models of silicon valley
byPatrick McFadin
 
Cassandra 3.0 advanced preview
byPatrick McFadin
 
Cassandra nice use cases and worst anti patterns
byDuyhai Doan
 
Open source or proprietary, choose wisely!
byPatrick McFadin
 
Traveler's Guide to Cassandra
byDataStax Academy
 
Storing time series data with Apache Cassandra
byPatrick McFadin
 
Cassandra concepts, patterns and anti-patterns
byDave Gardner
 
Advanced data modeling with apache cassandra
byPatrick McFadin
 
Reactive dashboard’s using apache spark
byRahul Kumar
 
Webinar - Bringing Game Changing Insights with Graph Databases
byDataStax
 
Cassandra Introduction & Features
byDataStax Academy
 
Apache cassandra & apache spark for time series data
byPatrick McFadin
 

Similar to Apache cassandra and spark. you got the the lighter, let's start the fire

PDF
Big Data Day LA 2015 - Sparking up your Cassandra Cluster- Analytics made Awe...
byData Con LA
 
PDF
Lightning fast analytics with Spark and Cassandra
bynickmbailey
 
PDF
Spark cassandra integration, theory and practice
byDuyhai Doan
 
PDF
PySpark Cassandra - Amsterdam Spark Meetup
byFrens Jan Rumph
 
PPTX
Lightning Fast Analytics with Cassandra and Spark
byTim Vincent
 
PPTX
Realtime Data Pipeline with Spark Streaming and Cassandra with Mesos (Rahul K...
byDataStax
 
PPTX
Real time data pipeline with spark streaming and cassandra with mesos
byRahul Kumar
 
PPTX
Spark + Cassandra = Real Time Analytics on Operational Data
byVictor Coustenoble
 
PDF
Manchester Hadoop Meetup: Cassandra Spark internals
byChristopher Batey
 
PDF
Cassandra and Spark
bydatastaxjp
 
PDF
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
byHelena Edelson
 
PPTX
Spark Cassandra Connector: Past, Present and Furure
byDataStax Academy
 
PDF
Analyzing Time Series Data with Apache Spark and Cassandra
byPatrick McFadin
 
PDF
Cassandra & Spark for IoT
byMatthias Niehoff
 
PPTX
5 Ways to Use Spark to Enrich your Cassandra Environment
byJim Hatcher
 
PPTX
Lightning fast analytics with Cassandra and Spark
byVictor Coustenoble
 
PPTX
Analyzing Time-Series Data with Apache Spark and Cassandra - StampedeCon 2016
byStampedeCon
 
PDF
Spark and cassandra (Hulu Talk)
byJon Haddad
 
PDF
Escape from Hadoop
byDataStax Academy
 
PDF
Spark Cassandra Connector Dataframes
byRussell Spitzer
 
Big Data Day LA 2015 - Sparking up your Cassandra Cluster- Analytics made Awe...
byData Con LA
 
Lightning fast analytics with Spark and Cassandra
bynickmbailey
 
Spark cassandra integration, theory and practice
byDuyhai Doan
 
PySpark Cassandra - Amsterdam Spark Meetup
byFrens Jan Rumph
 
Lightning Fast Analytics with Cassandra and Spark
byTim Vincent
 
Realtime Data Pipeline with Spark Streaming and Cassandra with Mesos (Rahul K...
byDataStax
 
Real time data pipeline with spark streaming and cassandra with mesos
byRahul Kumar
 
Spark + Cassandra = Real Time Analytics on Operational Data
byVictor Coustenoble
 
Manchester Hadoop Meetup: Cassandra Spark internals
byChristopher Batey
 
Cassandra and Spark
bydatastaxjp
 
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
byHelena Edelson
 
Spark Cassandra Connector: Past, Present and Furure
byDataStax Academy
 
Analyzing Time Series Data with Apache Spark and Cassandra
byPatrick McFadin
 
Cassandra & Spark for IoT
byMatthias Niehoff
 
5 Ways to Use Spark to Enrich your Cassandra Environment
byJim Hatcher
 
Lightning fast analytics with Cassandra and Spark
byVictor Coustenoble
 
Analyzing Time-Series Data with Apache Spark and Cassandra - StampedeCon 2016
byStampedeCon
 
Spark and cassandra (Hulu Talk)
byJon Haddad
 
Escape from Hadoop
byDataStax Academy
 
Spark Cassandra Connector Dataframes
byRussell Spitzer
 

More from Patrick McFadin

PDF
Help! I want to contribute to an Open Source project but my boss says no.
byPatrick McFadin
 
PDF
Cassandra 2.0 better, faster, stronger
byPatrick McFadin
 
PDF
Building Antifragile Applications with Apache Cassandra
byPatrick McFadin
 
PDF
Cassandra at scale
byPatrick McFadin
 
PDF
The world's next top data model
byPatrick McFadin
 
PDF
Become a super modeler
byPatrick McFadin
 
PDF
The data model is dead, long live the data model
byPatrick McFadin
 
PDF
Cassandra Virtual Node talk
byPatrick McFadin
 
PPT
Toronto jaspersoft meetup
byPatrick McFadin
 
PDF
Cassandra data modeling talk
byPatrick McFadin
 
Help! I want to contribute to an Open Source project but my boss says no.
byPatrick McFadin
 
Cassandra 2.0 better, faster, stronger
byPatrick McFadin
 
Building Antifragile Applications with Apache Cassandra
byPatrick McFadin
 
Cassandra at scale
byPatrick McFadin
 
The world's next top data model
byPatrick McFadin
 
Become a super modeler
byPatrick McFadin
 
The data model is dead, long live the data model
byPatrick McFadin
 
Cassandra Virtual Node talk
byPatrick McFadin
 
Toronto jaspersoft meetup
byPatrick McFadin
 
Cassandra data modeling talk
byPatrick McFadin
 

Apache cassandra and spark. you got the the lighter, let's start the fire

  • 1.
    ©2013 DataStax Confidential.Do not distribute without consent. @PatrickMcFadin Patrick McFadin
 Chief Evangelist for Apache Cassandra Apache Cassandra and Spark You got the lighter, let’s spark the fire 1
  • 2.
  • 3.
    Cassandra 3.0 &3.1 Spring and Fall
  • 4.
    Cassandra is… • Sharednothing • Masterless peer-to-peer • Great scaling story • Resilient to failure
  • 5.
  • 6.
    A Data Oceanor Pond., Lake An In-Memory Database A Key-Value Store A magical database unicorn that farts rainbows
  • 7.
  • 8.
    Apache Spark • 10xfaster on disk,100x faster in memory than Hadoop MR • Works out of the box on EMR • Fault Tolerant Distributed Datasets • Batch, iterative and streaming analysis • In Memory Storage and Disk • Integrates with Most File and Storage Options Up to 100× faster (2-10× on disk) 2-5× less code
  • 9.
    Spark Components Spark Core SparkSQL structured Spark Streaming real-time MLlib machine learning GraphX graph
  • 11.
    org.apache.spark.rdd.RDD Resilient Distributed Dataset(RDD) •Created through transformations on data (map,filter..) or other RDDs •Immutable •Partitioned •Reusable
  • 12.
    RDD Operations •Transformations -Similar to scala collections API •Produce new RDDs •filter, flatmap, map, distinct, groupBy, union, zip, reduceByKey, subtract •Actions •Require materialization of the records to generate a value •collect: Array[T], count, fold, reduce..
  • 13.
  • 14.
  • 15.
    Cassandra & Spark:A Great Combo Datastax: spark-cassandra-connector:
 https://github.com/datastax/spark-cassandra-connector •Both are Easy to Use •Spark Can Help You Bridge Your Hadoop and Cassandra Systems •Use Spark Libraries, Caching on-top of Cassandra-stored Data •Combine Spark Streaming with Cassandra Storage
  • 16.
    Spark On Cassandra •Server-Sidefilters (where clauses) •Cross-table operations (JOIN, UNION, etc.) •Data locality-aware (speed) •Data transformation, aggregation, etc. •Natural Time Series Integration
  • 17.
    Apache Spark andCassandra Open Source Stack Cassandra
  • 18.
  • 19.
    Spark Cassandra Connector *Cassandratables exposed as Spark RDDs *Read from and write to Cassandra *Mapping of C* tables and rows to Scala objects *All Cassandra types supported and converted to Scala types *Server side data selection *Virtual Nodes support *Use with Scala or Java *Compatible with, Spark 1.1.0, Cassandra 2.1 & 2.0
  • 20.
    Type Mapping CQL TypeScala Type ascii String bigint Long boolean Boolean counter Long decimal BigDecimal, java.math.BigDecimal double Double float Float inet java.net.InetAddress int Int list Vector, List, Iterable, Seq, IndexedSeq, java.util.List map Map, TreeMap, java.util.HashMap set Set, TreeSet, java.util.HashSet text, varchar String timestamp Long, java.util.Date, java.sql.Date, org.joda.time.DateTime timeuuid java.util.UUID uuid java.util.UUID varint BigInt, java.math.BigInteger *nullable values Option
  • 21.
    Connecting to Cassandra //Import Cassandra-specific functions on SparkContext and RDD objects import com.datastax.driver.spark._ // Spark connection options val conf = new SparkConf(true) .setMaster("spark://192.168.123.10:7077") .setAppName("cassandra-demo") .set("cassandra.connection.host", "192.168.123.10") // initial contact .set("cassandra.username", "cassandra") .set("cassandra.password", "cassandra") val sc = new SparkContext(conf)
  • 22.
    Accessing Data CREATE TABLEtest.words (word text PRIMARY KEY, count int); INSERT INTO test.words (word, count) VALUES ('bar', 30); INSERT INTO test.words (word, count) VALUES ('foo', 20); // Use table as RDD val rdd = sc.cassandraTable("test", "words") // rdd: CassandraRDD[CassandraRow] = CassandraRDD[0] rdd.toArray.foreach(println) // CassandraRow[word: bar, count: 30] // CassandraRow[word: foo, count: 20] rdd.columnNames // Stream(word, count) rdd.size // 2 val firstRow = rdd.first // firstRow: CassandraRow = CassandraRow[word: bar, count: 30] firstRow.getInt("count") // Int = 30 *Accessing table above as RDD:
  • 23.
    Saving Data val newRdd= sc.parallelize(Seq(("cat", 40), ("fox", 50))) // newRdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[2] newRdd.saveToCassandra("test", "words", Seq("word", "count")) SELECT * FROM test.words; word | count ------+------- bar | 30 foo | 20 cat | 40 fox | 50 (4 rows) *RDD above saved to Cassandra:
  • 24.
    Spark Cassandra Connector https://github.com/datastax/spark-­‐cassandra-­‐connector KeyspaceTable Cassandra Spark RDD[CassandraRow] RDD[Tuples] Bundled  and  Supported  with  DSE  4.5!
  • 25.
    Spark Cassandra Connectoruses the DataStax Java Driver to Read from and Write to C* Spark C* Full Token Range Each Executor Maintains a connection to the C* Cluster Spark Executor DataStax Java Driver Tokens 1-1000 Tokens 1001 -2000 Tokens … RDD’s read into different splits based on sets of tokens Spark Cassandra Connector
  • 26.
    Co-locate Spark andC* for Best Performance C* C*C* C* Spark
 Worker Spark
 Worker Spark Master Spark Worker Running Spark Workers on the same nodes as your C* Cluster will save network hops when reading and writing
  • 27.
    Analytics Workload Isolation Cassandra +Spark DC Cassandra Only DC Online App Analytical App Mixed Load Cassandra Cluster
  • 28.
  • 29.
    Example 1: WeatherStation • Weather station collects data • Cassandra stores in sequence • Application reads in sequence
  • 30.
    Data Model • WeatherStation Id and Time are unique • Store as many as needed CREATE TABLE temperature ( weather_station text, year int, month int, day int, hour int, temperature double, PRIMARY KEY (weather_station,year,month,day,hour) ); INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘10010:99999’,2005,12,1,7,-5.6); INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘10010:99999’,2005,12,1,8,-5.1); INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘10010:99999’,2005,12,1,9,-4.9); INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘10010:99999’,2005,12,1,10,-5.3);
  • 31.
    Primary key relationship PRIMARYKEY (weather_station,year,month,day,hour)
  • 32.
    Primary key relationship PRIMARYKEY (weather_station,year,month,day,hour) Partition Key
  • 33.
    Primary key relationship PRIMARYKEY (weather_station,year,month,day,hour) Partition Key Clustering Columns
  • 34.
    Primary key relationship PRIMARYKEY (weather_station,year,month,day,hour) Partition Key Clustering Columns 10010:99999
  • 35.
    2005:12:1:7 -5.6 Primary key relationship PRIMARYKEY (weather_station,year,month,day,hour) Partition Key Clustering Columns 10010:99999 -5.3-4.9-5.1 2005:12:1:8 2005:12:1:9 2005:12:1:10
  • 36.
    Partition keys 10010:99999 Murmur3Hash Token = 7224631062609997448 722266:13850 Murmur3 Hash Token = -6804302034103043898 INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘10010:99999’,2005,12,1,7,-5.6); INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘722266:13850’,2005,12,1,7,-5.6); Consistent hash. 128 bit number between 2-63 and 264
  • 37.
    Partition keys 10010:99999 Murmur3Hash Token = 15 722266:13850 Murmur3 Hash Token = 77 INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘10010:99999’,2005,12,1,7,-5.6); INSERT INTO temperature(weather_station,year,month,day,hour,temperature) VALUES (‘722266:13850’,2005,12,1,7,-5.6); For this example, let’s make it a reasonable number
  • 38.
    Writes & WANreplication 10.0.0.1 00-25 10.0.0.4 76-100 10.0.0.2 26-50 10.0.0.3 51-75 DC1 DC1: RF=3 Node Primary Replica Replica 10.0.0.1 00-25 76-100 51-75 10.0.0.2 26-50 00-25 76-100 10.0.0.3 51-75 26-50 00-25 10.0.0.4 76-100 51-75 26-50 10.10.0.1 00-25 10.10.0.4 76-100 10.10.0.2 26-50 10.10.0.3 51-75 DC2 Node Primary Replica Replica 10.0.0.1 00-25 76-100 51-75 10.0.0.2 26-50 00-25 76-100 10.0.0.3 51-75 26-50 00-25 10.0.0.4 76-100 51-75 26-50 DC2: RF=3 Client Insert Data Partition Key = 15 Asynchronous Local Replication Asynchronous WAN Replication
  • 39.
    Locality 10.0.0.1 00-25 10.0.0.4 76-100 10.0.0.2 26-50 10.0.0.3 51-75 DC1 DC1: RF=3 Node PrimaryReplica Replica 10.0.0.1 00-25 76-100 51-75 10.0.0.2 26-50 00-25 76-100 10.0.0.3 51-75 26-50 00-25 10.0.0.4 76-100 51-75 26-50 10.10.0.1 00-25 10.10.0.4 76-100 10.10.0.2 26-50 10.10.0.3 51-75 DC2 Node Primary Replica Replica 10.0.0.1 00-25 76-100 51-75 10.0.0.2 26-50 00-25 76-100 10.0.0.3 51-75 26-50 00-25 10.0.0.4 76-100 51-75 26-50 DC2: RF=3 Client Get Data Partition Key = 15 Client Get Data Partition Key = 15
  • 40.
  • 41.
    Spark Reads onCassandra Awesome animation by DataStax’s own Russel Spitzer
  • 42.
    Spark RDDs Represent aLarge Amount of Data Partitioned into Chunks RDD 1 2 3 4 5 6 7 8 9Node 2 Node 1 Node 3 Node 4
  • 43.
    Node 2 Node 1 SparkRDDs Represent a Large Amount of Data Partitioned into Chunks RDD 2 346 7 8 9 Node 3 Node 4 1 5
  • 44.
    Node 2 Node 1 RDD 2 346 78 9 Node 3 Node 4 1 5 Spark RDDs Represent a Large Amount of Data Partitioned into Chunks
  • 45.
    Cassandra Data isDistributed By Token Range
  • 46.
    Cassandra Data isDistributed By Token Range 0 500
  • 47.
    Cassandra Data isDistributed By Token Range 0 500 999
  • 48.
    Cassandra Data isDistributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4
  • 49.
    Cassandra Data isDistributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4 Without vnodes
  • 50.
    Cassandra Data isDistributed By Token Range 0 500 Node 1 Node 2 Node 3 Node 4 With vnodes
  • 51.
    Node 1 120-220 300-500 780-830 0-50 spark.cassandra.input.split.size 50 Reporteddensity is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions
  • 52.
    Node 1 120-220 300-500 0-50 spark.cassandra.input.split.size 50 Reporteddensity is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 1 780-830
  • 53.
    1 Node 1 120-220 300-500 0-50 spark.cassandra.input.split.size 50 Reporteddensity is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830
  • 54.
    2 1 Node 1 300-500 0-50 spark.cassandra.input.split.size50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830
  • 55.
    2 1 Node 1 300-500 0-50 spark.cassandra.input.split.size50 Reported density is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830
  • 56.
    2 1 Node 1 300-400 0-50 spark.cassandra.input.split.size 50 Reporteddensity is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 400-500
  • 57.
    21 Node 1 0-50 spark.cassandra.input.split.size 50 Reporteddensity is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 400-500
  • 58.
    21 Node 1 0-50 spark.cassandra.input.split.size 50 Reporteddensity is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 400-500 3
  • 59.
    21 Node 1 0-50 spark.cassandra.input.split.size 50 Reporteddensity is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3 400-500
  • 60.
    21 Node 1 0-50 spark.cassandra.input.split.size 50 Reporteddensity is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3
  • 61.
    4 21 Node 1 0-50 spark.cassandra.input.split.size 50 Reporteddensity is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3
  • 62.
    4 21 Node 1 0-50 spark.cassandra.input.split.size 50 Reporteddensity is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 780-830 3
  • 63.
    421 Node 1 spark.cassandra.input.split.size 50 Reporteddensity is 0.5 The Connector Uses Information on the Node to Make 
 Spark Partitions 3
  • 64.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50780-830 Node 1
  • 65.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50
  • 66.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50
  • 67.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows
  • 68.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows
  • 69.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows 50 CQL Rows
  • 70.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows
  • 71.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows
  • 72.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows
  • 73.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 74.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 75.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 76.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 780 and token(pk) <= 830 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 77.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 78.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 79.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 80.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 81.
    4 spark.cassandra.input.page.row.size 50 Data isRetrieved Using the DataStax Java Driver 0-50 780-830 Node 1 SELECT * FROM keyspace.table WHERE token(pk) > 0 and token(pk) <= 50 50 CQL Rows50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows 50 CQL Rows
  • 82.
    Co-locate Spark andC* for Best Performance C* C*C* C* Spark
 Worker Spark
 Worker Spark Master Spark Worker Running Spark Workers on the same nodes as your C* Cluster will save network hops when reading and writing
  • 83.
    Weather Station Analysis •Weather station collects data • Cassandra stores in sequence • Spark rolls up data into new tables Windsor California July 1, 2014 High: 73.4 Low : 51.4
  • 84.
    Roll-up table(SparkSQL example) CREATETABLE daily_high_low ( weatherstation text, date text, high_temp double, low_temp double, PRIMARY KEY ((weatherstation,date)) ); • Weather Station Id and Date are unique • High and low temp for each day SparkSQL> INSERT INTO TABLE > daily_high_low > SELECT > weatherstation, to_date(year, day, hour) date, max(temperature) high_temp, min(temperature) low_temp > FROM temperature > GROUP BY weatherstation_id, year, month, day; OK Time taken: 2.345 seconds functions aggregations
  • 85.
    What just happened •Data is read from temperature table • Transformed • Inserted into the daily_high_low table Table: temperature Table: daily_high_low Read data from table Transform Insert data into table
  • 86.
  • 87.
    zillions of bytesgigabytes per second Spark Versus Spark Streaming
  • 88.
  • 89.
    DStream - MicroBatches μBatch (ordinary RDD) μBatch (ordinary RDD) μBatch (ordinary RDD) Processing of DStream = Processing of μBatches, RDDs DStream • Continuous sequence of micro batches • More complex processing models are possible with less effort • Streaming computations as a series of deterministic batch computations on small time intervals
  • 90.
    Spark Streaming Example valconf = new SparkConf(loadDefaults = true) .set("spark.cassandra.connection.host", "127.0.0.1") .setMaster("spark://127.0.0.1:7077") val sc = new SparkContext(conf) val table: CassandraRDD[CassandraRow] = sc.cassandraTable("keyspace", "tweets") 
 val ssc = new StreamingContext(sc, Seconds(30))
 
 val stream = KafkaUtils.createStream[String, String, StringDecoder, StringDecoder](
 ssc, kafka.kafkaParams, Map(topic -> 1), StorageLevel.MEMORY_ONLY)
 
 stream.map(_._2).countByValue().saveToCassandra("demo", "wordcount")
 
 ssc.start()
 ssc.awaitTermination() Initialization Transformations and Action CassandraRDD Stream Initialization
  • 91.
    Now what? Cassandra Only DC Cassandra +Spark DC Spark Jobs Spark Streaming
  • 92.
    You can dothis at home! https://github.com/killrweather/killrweather On your USB!
  • 93.
    Thank you! Bring thequestions Follow me on twitter @PatrickMcFadin