Cassandra London - C* Spark Connector

1. ©2013 DataStax Confidential. Do not distribute without consent.
@chbatey
Christopher Batey
C* Spark Connector

2. @chbatey
Cassandra London Needs you
• We are always looking for Cassandra speakers to share
their experience and have created a Speakers Program
full of benefits! If you are interested please contact us for
details. Talk to Ale

3. @chbatey
Overview
• Reading data from C* into Spark
• Writing data to C*: effective batching

4. @chbatey
C* Table -> RDD

5. Spark RDDs
Represent a Large
Amount of Data
Partitioned into Chunks
RDD
1 2 3
4 5 6
7 8 9Node 2
Node 1 Node 3
Node 4

6. Node 2
Node 1
Spark RDDs
Represent a Large
Amount of Data
Partitioned into Chunks
RDD
2
346
7 8 9
Node 3
Node 4
1 5

7. Cassandra Data is Distributed By Token Range

8. Cassandra Data is Distributed By Token Range
0
500
999

9. Cassandra Data is Distributed By Token Range
0
500
Node 1
Node 2
Node 3
Node 4

10. Cassandra Data is Distributed By Token Range
0
500
Node 1
Node 2
Node 3
Node 4
Without vnodes

11. Cassandra Data is Distributed By Token Range
0
500
Node 1
Node 2
Node 3
Node 4
With vnodes

12. @chbatey
Goals
• Spark partitions made up of token ranges on the same
node
• Tasks to be executed on workers co-located with that
node
• Same(ish) amount of data in each Spark partition

13. Node 1
120-220
300-500
780-830
0-50
•spark.cassandra.input.split.size_in_mb 64
•system.size_estimates (# partitions & mean size)
•tokens per spark partition
The Connector Uses Information on the Node to Make
Spark Partitions

14. Node 1
120-220
300-500
0-50
The Connector Uses Information on the Node to Make
Spark Partitions
1
780-830

15. 1
Node 1
120-220
300-500
0-50
The Connector Uses Information on the Node to Make
Spark Partitions
780-830

16. 2
1
Node 1 300-500
0-50
The Connector Uses Information on the Node to Make
Spark Partitions
780-830

17. 2
1
Node 1 300-500
0-50
The Connector Uses Information on the Node to Make
Spark Partitions
780-830

18. 2
1
Node 1
300-400
0-50
The Connector Uses Information on the Node to Make
Spark Partitions
780-830
400-500

19. 21
Node 1
0-50
The Connector Uses Information on the Node to Make
Spark Partitions
780-830
400-500

20. 21
Node 1
0-50
The Connector Uses Information on the Node to Make
Spark Partitions
780-830
400-500
3

21. 21
Node 1
0-50
The Connector Uses Information on the Node to Make
Spark Partitions
780-830
3
400-500

22. 21
Node 1
0-50
The Connector Uses Information on the Node to Make
Spark Partitions
780-830
3

23. 4
21
Node 1
0-50
The Connector Uses Information on the Node to Make
Spark Partitions
780-830
3

24. 4
21
Node 1
0-50
The Connector Uses Information on the Node to Make
Spark Partitions
780-830
3

25. 421
Node 1
The Connector Uses Information on the Node to Make
Spark Partitions
3

26. @chbatey
Key classes
• CassandraTableScanRDD, CassandraRDD
- getPreferredLocations
• CassandraTableRowReaderProvider
- DataSizeEstimates - goes to C*
• CassandraPartitioner
- Gets ring information from the driver
• CassandraPartition
- endpoints
- tokenRanges

27. 4
spark.cassandra.input.fetch.size_in_rows 50
Data is Retrieved Using the DataStax Java Driver
0-50780-830
Node 1

28. 4
spark.cassandra.input.fetch.size_in_rows 50
Data is Retrieved Using the DataStax Java Driver
0-50
780-830
Node 1
SELECT * FROM keyspace.table WHERE
token(pk) > 780 and token(pk) <= 830
SELECT * FROM keyspace.table WHERE
token(pk) > 0 and token(pk) <= 50

29. 4
spark.cassandra.input.fetch.size_in_rows 50
Data is Retrieved Using the DataStax Java Driver
0-50
780-830
Node 1
SELECT * FROM keyspace.table WHERE
token(pk) > 780 and token(pk) <= 830
SELECT * FROM keyspace.table WHERE
token(pk) > 0 and token(pk) <= 50

30. 4
spark.cassandra.input.fetch.size_in_rows 50
Data is Retrieved Using the DataStax Java Driver
0-50
780-830
Node 1
SELECT * FROM keyspace.table WHERE
token(pk) > 780 and token(pk) <= 830
SELECT * FROM keyspace.table WHERE
token(pk) > 0 and token(pk) <= 50
50 CQL Rows

31. 4
spark.cassandra.input.fetch.size_in_rows 50
Data is Retrieved Using the DataStax Java Driver
0-50
780-830
Node 1
SELECT * FROM keyspace.table WHERE
token(pk) > 780 and token(pk) <= 830
SELECT * FROM keyspace.table WHERE
token(pk) > 0 and token(pk) <= 50
50 CQL Rows

32. 4
spark.cassandra.input.fetch.size_in_rows 50
Data is Retrieved Using the DataStax Java Driver
0-50
780-830
Node 1
SELECT * FROM keyspace.table WHERE
token(pk) > 780 and token(pk) <= 830
SELECT * FROM keyspace.table WHERE
token(pk) > 0 and token(pk) <= 50
50 CQL Rows
50 CQL Rows

33. 4
spark.cassandra.input.fetch.size_in_rows 50
Data is Retrieved Using the DataStax Java Driver
0-50
780-830
Node 1
SELECT * FROM keyspace.table WHERE
token(pk) > 780 and token(pk) <= 830
SELECT * FROM keyspace.table WHERE
token(pk) > 0 and token(pk) <= 50
50 CQL Rows50 CQL Rows

34. 4
spark.cassandra.input.fetch.size_in_rows 50
Data is Retrieved Using the DataStax Java Driver
0-50
780-830
Node 1
SELECT * FROM keyspace.table WHERE
token(pk) > 780 and token(pk) <= 830
SELECT * FROM keyspace.table WHERE
token(pk) > 0 and token(pk) <= 50
50 CQL Rows50 CQL Rows
50 CQL Rows

35. 4
spark.cassandra.input.fetch.size_in_rows 50
Data is Retrieved Using the DataStax Java Driver
0-50
780-830
Node 1
SELECT * FROM keyspace.table WHERE
token(pk) > 780 and token(pk) <= 830
SELECT * FROM keyspace.table WHERE
token(pk) > 0 and token(pk) <= 50
50 CQL Rows50 CQL Rows
50 CQL Rows

36. 4
spark.cassandra.input.fetch.size_in_rows 50
Data is Retrieved Using the DataStax Java Driver
0-50
780-830
Node 1
SELECT * FROM keyspace.table WHERE
token(pk) > 780 and token(pk) <= 830
SELECT * FROM keyspace.table WHERE
token(pk) > 0 and token(pk) <= 50
50 CQL Rows50 CQL Rows
50 CQL Rows
50 CQL Rows

37. 4
spark.cassandra.input.page.row.size 50
Data is Retrieved Using the DataStax Java Driver
0-50
780-830
Node 1
SELECT * FROM keyspace.table WHERE
token(pk) > 0 and token(pk) <= 50
50 CQL Rows50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows

38. 4
spark.cassandra.input.page.row.size 50
Data is Retrieved Using the DataStax Java Driver
0-50
780-830
Node 1
SELECT * FROM keyspace.table WHERE
token(pk) > 0 and token(pk) <= 50
50 CQL Rows50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows

39. 4
spark.cassandra.input.page.row.size 50
Data is Retrieved Using the DataStax Java Driver
0-50
780-830
Node 1
SELECT * FROM keyspace.table WHERE
token(pk) > 0 and token(pk) <= 50
50 CQL Rows50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows
50 CQL Rows

40. @chbatey
Paging

41. @chbatey
Other bits and bobs
• LocalNodeFirstLoadBalancingPolicy

42. @chbatey
Then we’re into Spark land
• Spark partitions are made up of C* partitions that exist
on the same node
• C* connector tells Spark which workers to use via
information from the C* driver

43. @chbatey
RDD -> C* Table

44. Node 2
Node 1
RDD
2
346
7 8 9
Node 3
Node 4
1 5
The Spark Cassandra
Connector saveToCassandra
method can be called on
almost all RDDs
rdd.saveToCassandra("Keyspace","Table")

45. Node 11
Java
Driver

46. Node 11
Java
Driver
1,1,1
1,2,1
2,1,1
3,8,1
3,2,1
3,4,1
3,5,1
3,1,1
1,4,1
5,4,1
2,4,1
8,4,1
9,4,1
3,9,1

47. Node 11
Java
Driver
1,1,1
1,2,1
2,1,1
3,8,1
3,2,1
3,4,1
3,5,1
3,1,1
1,4,1
5,4,1
2,4,1
8,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
3,9,1

48. Node 11
Java
Driver
1,1,1
1,2,1
2,1,1
3,8,1
3,2,1
3,4,1
3,5,1
3,1,1
1,4,1
5,4,1
2,4,1
8,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
3,9,1
PK=1

49. Node 11
Java
Driver
1,1,1 1,2,1
2,1,1
3,8,1
3,2,1
3,4,1
3,5,1
3,1,1
1,4,1
5,4,1
2,4,1
8,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
3,9,1
PK=1

50. Node 11
Java
Driver
1,1,1 1,2,1
2,1,1
3,8,1
3,2,1
3,4,1
3,5,1
3,1,1
1,4,1
5,4,1
2,4,1
8,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
3,9,1
PK=1
PK=2

51. Node 11
Java
Driver
1,1,1 1,2,1
2,1,1
3,8,1
3,2,1
3,4,1
3,5,1
3,1,1
1,4,1
5,4,1
2,4,1
8,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
3,9,1
PK=1
PK=2

52. Node 11
Java
Driver
1,1,1 1,2,1
2,1,1
3,8,13,2,1 3,4,1 3,5,1
3,1,1
1,4,1
5,4,1
2,4,1
8,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
3,9,1
PK=1
PK=2
PK=3

53. Node 11
Java
Driver
1,1,1 1,2,1
2,1,1
1,4,1
5,4,1
2,4,1
8,4,1
9,4,1
11,4,3,9,1
3,1,1
spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
PK=1
PK=2

54. Node 11
Java
Driver
1,1,1 1,2,1
2,1,1
3,1,1
1,4,1
5,4,1
2,4,1
8,4,1
9,4,1
11,4,3,9,1 spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
PK=1
PK=2
PK=3

55. Node 11
Java
Driver
1,1,1 1,2,1
2,1,1
3,1,1
1,4,1
5,4,1
2,4,1
8,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
3,9,1
PK=1
PK=2
PK=3

56. Node 11
Java
Driver
2,1,1
3,1,1
5,4,1
2,4,1
8,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
3,9,1
PK=2
PK=3

57. Node 11
Java
Driver
2,1,1
3,1,1
5,4,1
2,4,1
8,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
3,9,1
PK=2
PK=3
PK=5

58. Node 11
Java
Driver
2,1,1
3,1,1
5,4,1
2,4,1
8,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
3,9,1
PK=2
PK=3
PK=5

59. Node 11
Java
Driver
2,1,1
3,1,1
5,4,1
2,4,18,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
3,9,13,9,1
PK=2
PK=3
PK=5

60. Node 11
Java
Driver
2,1,1
3,1,1
5,4,1
2,4,18,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
3,9,13,9,1
Write Acknowledged
PK=2
PK=3
PK=5

61. Node 11
Java
Driver
2,1,1
3,1,1
5,4,1
2,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
8,4,1
3,9,1
PK=2
PK=3
PK=5

62. Node 11
Java
Driver
3,1,1
5,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
8,4,1
3,9,1
PK=3
PK=5

63. Node 11
Java
Driver
3,1,1
5,4,1
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
8,4,1
3,9,1
PK=8
PK=3
PK=5

64. Node 11
Java
Driver
9,4,1
11,4, spark.cassandra.output.batch.grouping.key partition
spark.cassandra.output.batch.size.rows 4
spark.cassandra.output.batch.grouping.buffer.size 3
spark.cassandra.output.concurrent.writes 2
3,1,1
5,4,1
8,4,1
3,9,1
PK=8
PK=3
PK=5

65. @chbatey
Summary
• Reading - data locality is key
• Joining - repartition by C*
• Writing - batching by C* partition is key