The document discusses using Apache Cassandra and Apache Spark together for time series analysis of weather data. It provides an overview of Cassandra and how it is suitable for storing time series data modeled by primary key. It then discusses using Spark to perform analytics on the Cassandra data by rolling it up into aggregated tables for things like daily high/low temperatures and precipitation amounts. Example Cassandra table schemas and Spark queries are shown to illustrate the end-to-end workflow of ingesting raw weather data, storing it in Cassandra, and analyzing it using Spark.

1. @chbatey
Christopher Batey
Technical Evangelist for Apache Cassandra
Time series analysis with Spark and
Cassandra

2. @chbatey
Who am I?
• Technical Evangelist for Apache Cassandra
•Founder of Stubbed Cassandra
•Help out Apache Cassandra users
• DataStax
•Builds enterprise ready version of Apache
Cassandra
• Previous: Cassandra backed apps at BSkyB

3. @chbatey
Agenda
• Motivation
• Cassandra
• Replication
• Fault tolerance
• Data modelling
• Spark
• Use cases
• Stream processing
• Time series example: Weather station data

4. @chbatey
OLTP OLAP Batch

5. Weather data streaming
Incoming
weather
events
Apache Kafka
Producer
Consumer
NodeGuardian
Dashboard

6. @chbatey

7. @chbatey

8. @chbatey
Run this your self
• https://github.com/killrweather/killrweather

9. @chbatey
Cassandra

10. @chbatey
Cassandra for Applications
APACHE
CASSANDRA

11. @chbatey
Common use cases
•Ordered data such as time series
-Event stores
-Financial transactions
-IoT e.g Sensor data

12. @chbatey
Common use cases
•Ordered data such as time series
-Event stores
-Financial transactions
-IoT e.g Sensor data
•Non functional requirements:
-Linear scalability
-High throughout durable writes
-Multi datacenter including active-active
-Analytics without ETL

13. @chbatey
Cassandra
Cassandra
• Distributed masterless
database (Dynamo)
• Column family data model
(Google BigTable)

14. @chbatey
Datacenter and rack aware
Europe
• Distributed master less
database (Dynamo)
• Column family data model
(Google BigTable)
• Multi data centre replication
built in from the start
USA

15. @chbatey
Cassandra
Online
• Distributed master less
database (Dynamo)
• Column family data model
(Google BigTable)
• Multi data centre replication
built in from the start
• Analytics with Apache SparkAnalytics

16. @chbatey
Dynamo 101

17. @chbatey
Dynamo 101
• The parts Cassandra took
- Consistent hashing
- Replication
- Gossip
- Hinted handoff
- Anti-entropy repair
• And the parts it left behind
- Key/Value
- Vector clocks

18. @chbatey
Picking the right nodes
• You don’t want a full table scan on a 1000 node cluster!
• Dynamo to the rescue: Consistent Hashing

19. @chbatey
Murmer3 Example
• Data:
• Murmer3 Hash Values:
jim age: 36 car: ford gender: M
carol age: 37 car: bmw gender: F
johnny age: 12 gender: M
suzy: age: 10 gender: F
Primary Key Murmur3 hash value
jim 350
carol 998
johnny 50
suzy 600
Primary Key
Real hash range: -9223372036854775808 to 9223372036854775807

20. @chbatey
Murmer3 Example
Four node cluster:
Node Murmur3 start range Murmur3 end range
A 0 249
B 250 499
C 500 749
D 750 999

21. @chbatey
Pictures are better
A
B
C
D
999
249
499
750
749
0
250
500
B
CD
A

22. @chbatey
Murmer3 Example
Data is distributed as:
Node Start range End range Primary
key
Hash value
A 0 249 johnny 50
B 250 499 jim 350
C 500 749 suzy 600
D 750 999 carol 998

23. @chbatey
Replication

24. @chbatey
Replication strategy
• Simple
- Give it to the next node in the ring
- Don’t use this in production
• NetworkTopology
- Every Cassandra node knows its DC and Rack
- Replicas won’t be put on the same rack unless Replication Factor > # of racks
- Unfortunately Cassandra can’t create servers and racks on the fly to fix this :(

25. @chbatey
Replication
DC1 DC2
client
RF3 RF3
C
RC
WRITE
CL = 1 We have replication!

26. 26

27. @chbatey
Tunable Consistency
•Data is replicated N times
•Every query that you execute you give a consistency
-ALL
-QUORUM
-LOCAL_QUORUM
-ONE
• Christos Kalantzis Eventual Consistency != Hopeful Consistency: http://
youtu.be/A6qzx_HE3EU?list=PLqcm6qE9lgKJzVvwHprow9h7KMpb5hcUU

28. @chbatey
Scaling shouldn’t be hard
• Throw more nodes at a cluster
• Bootstrapping + joining the ring
• For large data sets this can take some time

29. @chbatey
Spark Time

30. @chbatey
Scalability & Performance
• Scalability
- No single point of failure
- No special nodes that become the bottle neck
- Work/data can be re-distributed
• Operational Performance i.e single digit ms
- Single node for query
- Single disk seek per query

31. @chbatey
But but…
• Sometimes you don’t need a answers in milliseconds
• Reports / analysis
• Data models done wrong - how do I fix it?
• New requirements for old data?
• Ad-hoc operational queries
• Managers always want counts / maxs

32. @chbatey

33. @chbatey
Apache Spark
• 10x faster on disk,100x faster in memory than Hadoop
MR
• Works out of the box on EMR
• Fault tolerant distributed datasets
• Batch, iterative and streaming analysis
• In memory storage and disk
• Integrates with most file and storage options

34. @chbatey
Part of most Big Data Platforms
Analytic
Search
• All Major Hadoop Distributions Include
Spark
• Spark Is Also Integrated With Non-
Hadoop Big Data Platforms like DSE
• Spark Applications Can Be Written Once
and Deployed Anywhere
SQL
Machine
Learning
Streaming Graph
Core
Deploy Spark Apps Anywhere

35. @chbatey
Components
Shark
or
Spark SQL
Streaming ML
Spark (General execution engine)
Graph
Cassandra
Compatible

36. @chbatey
org.apache.spark.rdd.RDD
• Resilient Distributed Dataset (RDD)
• Created through transformations on data (map,filter..) or other RDDs
• Immutable
• Partitioned
• Reusable

37. @chbatey
RDD Operations
• Transformations - Similar to Scala collections API
• Produce new RDDs
• filter, flatmap, map, distinct, groupBy, union, zip, reduceByKey, subtract
• Actions
• Require materialization of the records to generate a value
• collect: Array[T], count, fold, reduce..

38. @chbatey
Word count
val file: RDD[String] = sc.textFile("hdfs://...")
val counts: RDD[(String, Int)] = file.flatMap(line =>
line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")

39. zillions of bytes gigabytes per second
Spark Versus Spark Streaming

40. DStream - Micro Batches
μBatch (ordinary RDD) μBatch (ordinary RDD) μBatch (ordinary RDD)
Processing of DStream = Processing of μBatches, RDDs
DStream
• Continuous sequence of micro batches
• More complex processing models are possible with less effort
• Streaming computations as a series of deterministic batch
computations on small time intervals

41. @chbatey
Time series example time

42. @chbatey
Deployment
• Spark worker in each of the
Cassandra nodes
• Partitions made up of LOCAL
cassandra data
S C
S C
S C
S C

43. Weather Station Analysis
• Weather station collects data
• Cassandra stores in sequence
• Spark rolls up data into new
tables
Windsor California
July 1, 2014
High: 73.4F
Low : 51.4F

44. raw_weather_data
CREATE TABLE raw_weather_data (
weather_station text, // Composite of Air Force Datsav3 station number and NCDC WBAN numbe
year int, // Year collected
month int, // Month collected
day int, // Day collected
hour int, // Hour collected
temperature double, // Air temperature (degrees Celsius)
dewpoint double, // Dew point temperature (degrees Celsius)
pressure double, // Sea level pressure (hectopascals)
wind_direction int, // Wind direction in degrees. 0-359
wind_speed double, // Wind speed (meters per second)
sky_condition int, // Total cloud cover (coded, see format documentation)
sky_condition_text text, // Non-coded sky conditions
one_hour_precip double, // One-hour accumulated liquid precipitation (millimeters)
six_hour_precip double, // Six-hour accumulated liquid precipitation (millimeters)
PRIMARY KEY ((weather_station), year, month, day, hour)
) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC);
Reverses data in the storage engine.

45. Primary key relationship
PRIMARY KEY (weatherstation_id,year,month,day,hour)

46. Primary key relationship
PRIMARY KEY (weatherstation_id,year,month,day,hour)
Partition Key

47. Primary key relationship
PRIMARY KEY (weatherstation_id,year,month,day,hour)
Partition Key Clustering Columns

48. Primary key relationship
PRIMARY KEY (weatherstation_id,year,month,day,hour)
Partition Key Clustering Columns
10010:99999

49. 2005:12:1:7
-5.6
Primary key relationship
PRIMARY KEY (weatherstation_id,year,month,day,hour)
Partition Key Clustering Columns
10010:99999
-5.3-4.9-5.1
2005:12:1:8 2005:12:1:9 2005:12:1:10

50. Data Locality
weatherstation_id=‘10010:99999’ ?
1000 Node Cluster
You are here!

51. Query patterns
• Range queries
• “Slice” operation on disk
SELECT weatherstation,hour,temperature
FROM raw_weather_data
WHERE weatherstation_id=‘10010:99999'
AND year = 2005 AND month = 12 AND day = 1
AND hour >= 7 AND hour <= 10;
Single seek on disk
2005:12:1:12
-5.4
2005:12:1:11
-4.9-5.3-4.9-5.1
2005:12:1:7
-5.6
2005:12:1:8 2005:12:1:9
10010:99999
2005:12:1:10
Partition key for locality

52. Query patterns
• Range queries
• “Slice” operation on disk
Programmers like this
Sorted by event_time
2005:12:1:7
-5.6
2005:12:1:8
-5.1
2005:12:1:9
-4.9
10010:99999
10010:99999
10010:99999
weather_station hour temperature
2005:12:1:10
-5.3
10010:99999
SELECT weatherstation,hour,temperature
FROM raw_weather_data
WHERE weatherstation_id=‘10010:99999'
AND year = 2005 AND month = 12 AND day = 1
AND hour >= 7 AND hour <= 10;

53. weather_station
CREATE TABLE weather_station (
id text PRIMARY KEY, // Composite of Air Force Datsav3 station number and NCDC WBAN number
name text, // Name of reporting station
country_code text, // 2 letter ISO Country ID
state_code text, // 2 letter state code for US stations
call_sign text, // International station call sign
lat double, // Latitude in decimal degrees
long double, // Longitude in decimal degrees
elevation double // Elevation in meters
);
Lookup table

54. daily_aggregate_temperature
CREATE TABLE daily_aggregate_temperature (
weather_station text,
year int,
month int,
day int,
high double,
low double,
mean double,
variance double,
stdev double,
PRIMARY KEY ((weather_station), year, month, day)
) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC);
SELECT high, low FROM daily_aggregate_temperature
WHERE weather_station='010010:99999'
AND year=2005 AND month=12 AND day=3;
high | low
------+------
1.8 | -1.5

55. daily_aggregate_precip
CREATE TABLE daily_aggregate_precip (
weather_station text,
year int,
month int,
day int,
precipitation counter,
PRIMARY KEY ((weather_station), year, month, day)
) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC);
SELECT precipitation FROM daily_aggregate_precip
WHERE weather_station='010010:99999'
AND year=2005 AND month=12 AND day>=1 AND day <= 7;
0
10
20
30
40
1 2 3 4 5 6 7
17
26
2
0
33
12
0

56. Result
wsid | year | month | day | high | low
--------------+------+-------+-----+------+------
725300:94846 | 2012 | 9 | 30 | 18.9 | 10.6
725300:94846 | 2012 | 9 | 29 | 25.6 | 9.4
725300:94846 | 2012 | 9 | 28 | 19.4 | 11.7
725300:94846 | 2012 | 9 | 27 | 17.8 | 7.8
725300:94846 | 2012 | 9 | 26 | 22.2 | 13.3
725300:94846 | 2012 | 9 | 25 | 25 | 11.1
725300:94846 | 2012 | 9 | 24 | 21.1 | 4.4
725300:94846 | 2012 | 9 | 23 | 15.6 | 5
725300:94846 | 2012 | 9 | 22 | 15 | 7.2
725300:94846 | 2012 | 9 | 21 | 18.3 | 9.4
725300:94846 | 2012 | 9 | 20 | 21.7 | 11.7
725300:94846 | 2012 | 9 | 19 | 22.8 | 5.6
725300:94846 | 2012 | 9 | 18 | 17.2 | 9.4
725300:94846 | 2012 | 9 | 17 | 25 | 12.8
725300:94846 | 2012 | 9 | 16 | 25 | 10.6
725300:94846 | 2012 | 9 | 15 | 26.1 | 11.1
725300:94846 | 2012 | 9 | 14 | 23.9 | 11.1
725300:94846 | 2012 | 9 | 13 | 26.7 | 13.3
725300:94846 | 2012 | 9 | 12 | 29.4 | 17.2
725300:94846 | 2012 | 9 | 11 | 28.3 | 11.7
725300:94846 | 2012 | 9 | 10 | 23.9 | 12.2
725300:94846 | 2012 | 9 | 9 | 21.7 | 12.8
725300:94846 | 2012 | 9 | 8 | 22.2 | 12.8
725300:94846 | 2012 | 9 | 7 | 25.6 | 18.9
725300:94846 | 2012 | 9 | 6 | 30 | 20.6
725300:94846 | 2012 | 9 | 5 | 30 | 17.8
725300:94846 | 2012 | 9 | 4 | 32.2 | 21.7
725300:94846 | 2012 | 9 | 3 | 30.6 | 21.7
725300:94846 | 2012 | 9 | 2 | 27.2 | 21.7
725300:94846 | 2012 | 9 | 1 | 27.2 | 21.7
SELECT wsid, year, month, day, high, low
FROM daily_aggregate_temperature
WHERE wsid = '725300:94846'
AND year=2012 AND month=9 ;

57. Weather Station Stream Analysis
• Weather station collects data
• Data processed in stream
• Data stored in Cassandra
Windsor California
Today
Rainfall total: 1.2cm
High: 73.4F
Low : 51.4F

58. Incoming data from Kafka
725030:14732,2008,01,01,00,5.0,-3.9,1020.4,270,4.6,2,0.0,0.0

59. @chbatey
Creating a Stream

60. @chbatey
Saving the raw data

61. @chbatey
Building an aggregate
CREATE TABLE daily_aggregate_precip (
weather_station text,
year int,
month int,
day int,
precipitation counter,
PRIMARY KEY ((weather_station), year, month, day)
) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC);
CQL Counter

62. Weather data streaming
Load
Generator or
Data import
Apache Kafka
Producer
Consumer
NodeGuardian
Dashboard

63. @chbatey

64. @chbatey

65. @chbatey
Run this your self
• https://github.com/killrweather/killrweather

66. @chbatey
Summary
• Cassandra
- always-on operational database
• Spark
- Batch analytics
- Stream processing and saving back to Cassandra

67. @chbatey
Thanks for listening
• Follow me on twitter @chbatey
• Cassandra + Fault tolerance posts a plenty:
• http://christopher-batey.blogspot.co.uk/
• Cassandra resources: http://planetcassandra.org/
• Full free day of Cassandra talks/training:
• http://www.eventbrite.com/e/cassandra-day-london-2015-
april-22nd-2015-tickets-15053026006?aff=meetup1