In this in-depth workshop you will gain hands on experience with using Spark and Cassandra inside the DataStax Enterprise Platform. The focus of the workshop will be working through data analytics exercises to understand the major developer developer considerations. You will also gain an understanding of the internals behind the integration that allow for large scale data loading and analysis. It will also review some of the major machine learning libraries in Spark as an example of data analysis. The workshop will start with a review the basics of how Spark and Cassandra are integrated. Then we will work through a series of exercises that will show how to perform large scale Data Analytics with Spark and Cassandra. A major part of the workshop will be to understand effective data modeling techniques in Cassandra that allow for fast parallel loading of the data into Spark to perform large scale analytics on that data. The exercises will also look at how to how to use the open source Spark Notebook to run interactive data analytics with the DataStax Enterprise Platform.

1. Large Scale Data Analytics

2. Large Scale
Data Analytics
Ryan Knight
@Knight_Cloud
Solution Engineer - DataStax
Paco Nathan
@pacoid
Evil Mad Scientist - O’Reilly Media

3. Demo of Streaming in the Real World -
Spark At Scale Project
3© 2015. All Rights Reserved.
•Based on Real World Use Cases
•Simulate a real world streaming use case
•Test throughput of Spark Streaming
•Best Practices for scaling
•https://github.com/retroryan/SparkAtScale

4. Spark At Scale
Demo Application
4© 2015. All Rights Reserved.
DataStax Enterprise Platform
DataStax Enterprise Platform

5. How do we Scale for Load and Traffic?

6. Data Modeling using Event Sourcing
6© 2015. All Rights Reserved.
•Append-Only Logging
•Database of Facts
•Snapshots or Roll-Ups
•Why Delete Data any more?
•Replay Events

7. Scala for Large Scale Data Analytics
7© 2015. All Rights Reserved.
•Functional Paradigm is ideal for Data Analytics
•Strongly Typed - Enforce Schema at Every Later
•Immutable by Default - Event Logging
•Declarative instead of Imperative - Focus on
Transformation not Implementation

8. Key to Scaling - Configuring Kafka Topics
8© 2015. All Rights Reserved.
•Number of Partitions per Topic — Degree of parallelism
•Directly Affects Spark Streaming Parallelism
•bin/kafka-topics.sh --create --zookeeper localhost:2181 --
replication-factor 1 --partitions 5 --topic ratings

9. Populating Kafka Topics
9© 2015. All Rights Reserved.
val record = new ProducerRecord[String, String]
(feederExtension.kafkaTopic, partNum, key,
nxtRating.toString)
val future = feederExtension.producer.send(record, new
Callback {

10. 10© 2015. All Rights Reserved.

11. Spark Streaming with Kafka Direct Approach
11© 2015. All Rights Reserved.
•Use Kafka Direct Approach (No Receivers)
•Queries Kafka Directly
•Automatically Parallelizes based on Kafka Partitions
•Exactly Once Processing - Only Move Offset after
Processing
•Resiliency without copying data

12. 12© 2015. All Rights Reserved.

13. Spark Streaming Monitoring
13© 2015. All Rights Reserved.
Processing Time
>
Batch Duration
=
Total Delay Grows
Out Of Memory
Errors

14. © 2014 DataStax, All Rights Reserved.
Confidential
DataStax Enterprise Platform
Workload Segregation w/out ETL
14
Cassandra Mode
OLTP Database
Analytics Mode
Streaming and Analytics
Search Mode
All Data Searchable
C*
C
C
S A
A

15. DataStax Enterprise Platform
Workload Segregation w/out ETL
15© 2015. All Rights Reserved.

16. DataStax Enterprise Platform
Integrated Spark Analytics
16© 2015. All Rights Reserved.

17. DataStax Analytics
17© 2015. All Rights Reserved.
•Simplified Deployment and Management
•HA Spark Master with automatic leader election
•Detects when Spark Master is down with gossip
•Uses Paxos to elect Spark Master
•Stores Spark Worker metadata in Cassandra
•No need to run Zookeeper

18. Spark Notebook
18© 2015. All Rights Reserved.
C*
C
C A
AA
Notebook
Notebook
Notebook
Spark Notebook Server
Cassandra Cluster with Spark Connector

19. Apache Spark Notebook
19© 2015. All Rights Reserved.
•Reactive / Dynamic Graphs base on Scala, SQL and
DataFrames
•Spark Streaming
• Examples notebooks covering visualization, machine
learning, streaming, graph analysis, genomics analysis
•SVG / Sliders - interactive graphs
•Tune and Configure Each Notebook Separately
•https://github.com/andypetrella/spark-notebook

20. databricks.gitbooks.io/databricks-spark-reference-applications/
content/twitter_classifier/README.html
Demo: Twitter Streaming Language Classifier

21. Streaming:
collect tweets
Twitter API
HDFS:
dataset
Spark SQL:
ETL, queries
MLlib:
train classifier
Spark:
featurize
HDFS:
model
Streaming:
score tweets
language
filter
Demo: Twitter Streaming Language Classifier
Cassandra
Cassandra

22. 1. extract text
from the tweet
https://twitter.com/andy_bf/
status/16222269370011648
"Ceci n'est pas un tweet"
2. sequence text
as bigrams
tweet.sliding(2).toSeq ("Ce", "ec", "ci", …, )
3. convert
bigrams into
numbers
seq.map(_.hashCode()) (2178, 3230, 3174, …, )
4. index into
sparse tf vector
seq.map(_.hashCode() % 1000) (178, 230, 174, …, )
5. increment
feature count
Vector.sparse(1000, …) (1000, [102, 104, …],
[0.0455, 0.0455, …])
Demo: Twitter Streaming Language Classifier
From tweets to ML features,
approximated as sparse
vectors:

23. KMeans: Formal Definition (ignore this)

24. KMeans: How it really works…

25. KMeans: How it really works…

26. Demo: Twitter Streaming Language Classifier
Sample Code + Output:
https://github.com/retroryan/twitter_classifier
val sc = new SparkContext(new SparkConf())
val ssc = new StreamingContext(conf, Seconds(5))
val tweets = TwitterUtils.createStream(ssc, Utils.getAuth)
val statuses = tweets.map(_.getText)
val model = new KMeansModel(ssc.sparkContext.objectFile[Vector]
(modelFile.toString).collect())
val filteredTweets = statuses
.filter(t =>
model.predict(Utils.featurize(t)) == clust)
filteredTweets.print()
ssc.start()
ssc.awaitTermination()
CLUSTER 1:
TLあんまり見ないけど
@くれたっら
いつでもくっるよ٩(δωδ)۶
そういえばディスガイアも今日か
CLUSTER 4:
‫صدام‬ ‫بعد‬ ‫روحت‬ ‫العروبه‬ ‫قالوا‬
‫العروبه‬ ‫تحيى‬ ‫سلمان‬ ‫مع‬ ‫واقول‬
RT @vip588: √ ‫مي‬ ‫فولو‬ √ ‫متابعني‬ ‫زيادة‬ √ ‫االن‬ ‫للمتواجدين‬ vip588
√ ‫ما‬ ‫يلتزم‬ ‫ما‬ ‫اللي‬ √ ‫رتويت‬ ‫عمل‬ ‫للي‬ ‫فولو‬ √ ‫للتغريدة‬ ‫رتويت‬ √ ‫باك‬ ‫فولو‬
‫بيستفيد‬ …
‫سورة‬ ‫ن‬

27. Thank you