OSCON TALK: Becoming Friends with Cassandra and Spark

1. BECOMING FRIENDS WITH CASSANDRA & SPARK DANI TRAPHAGEN & JON HADDAD YOU SPARK C*

2. BECOMING FRIENDS WITH CASSANDRA & SPARK DANI TRAPHAGEN & JON HADDAD YOU SPARKC*

3. HOUSEKEEPING

4. RAISE YOUR HAND IF YOU DON’T HAVE THE VM OSCON2016.ZIP

5. 1.copy the vm files to a place of your choosing 2.open virtual ovf VM INSTRUCTIONS

6. 3.import the .ovf as prompted

7. 3.open the packer ovf in VirtualBox

8. 4.check out the vm

9. LET’S GET STARTED

10. WHAT ARE WE GOING TO COVER? 1. CASSANDRA ARCHITECTURE, CQL, DATA MODELING 2. SPARK DATAFRAMES

11. RDBMS & YOU

12. SQLITE, PYTHON SCRIPTS, LOG FILES SUCH AS? SMALL DATA

13. MOST WEB SITES RDBMS MEDIUM DATA

14. CAN RDBMS WORK FOR BIG DATA? YOU BIG DATA

15. VERTICAL SCALE

16. VERTICAL SCALE STARTING MY BUSINESS YAY!

17. VERTICAL SCALE STARTING MY BUSINESS YAY!

18. VERTICAL SCALE STARTING MY BUSINESS YAY! OH, WHOA, THINGS ARE KICKING UP

21. ACID IS A LIE

22. ACID IS A LIE ATOMICITY

23. ACID IS A LIE ATOMICITY CONSISTENCY

24. ACID IS A LIE ATOMICITY CONSISTENCY ISOLATION

25. ACID IS A LIE ATOMICITY CONSISTENCY ISOLATION DURABILITY

26. ACID IS A LIE ATOMICITY CONSISTENCY ISOLATION DURABILITY

27. ASYNC REPLICATION != CONSISTENCY

28. ASYNC REPLICATION != CONSISTENCY CLIENT

29. ASYNC REPLICATION != CONSISTENCY CLIENT

30. ASYNC REPLICATION != CONSISTENCY CLIENT MASTER

31. ASYNC REPLICATION != CONSISTENCY CLIENT MASTER SLAVE

32. ASYNC REPLICATION != CONSISTENCY CLIENT MASTER SLAVE

33. ASYNC REPLICATION != CONSISTENCY CLIENT MASTER SLAVE REPLICATION LAG

34. CONSISTENT? ASYNC REPLICATION != CONSISTENCY CLIENT MASTER SLAVE REPLICATION LAG

35. CONSISTENT? ASYNC REPLICATION != CONSISTENCY CLIENT MASTER SLAVE REPLICATION LAG IDK?

36. CONSISTENT? ASYNC REPLICATION != CONSISTENCY CLIENT MASTER SLAVE REPLICATION LAG LOL NO! IDK?

37. THIRD NORMAL FORM DOESN’T SCALE ▸ UNPREDICTABLE ▸ DATA > MEMORY? ▸ DISK SEEKS ALL DAY ▸ USERS = ANGRY

38. THIRD NORMAL FORM DOESN’T SCALE AWFUL ▸ UNPREDICTABLE ▸ DATA > MEMORY? ▸ DISK SEEKS ALL DAY ▸ USERS = ANGRY

39. SHARDING

40. SHARDING CLIENT

41. SHARDING CLIENT

42. SHARDING CLIENT NIGHTMARE

43. AVAILABILITY?

44. AVAILABILITY? NOT WITH THESE KNUCKLEHEADS

45. CONCLUSION: SCALING IS HARD

46. FRIEND #1: CASSANDRA

47. FRIEND #1: CASSANDRA

48. ARCHITECTURE

49. ARCHITECTURE PEER TO PEER ▸ With Cassandra there is no Master Slave Hierarchy ▸ Every node is the captain of it’s own ship ▸ Processes within Cassandra make this possible ▸ Replication ▸ Consistency Level NODE 1 NODE 2 NODE 3 NODE 4

50. ARCHITECTURE PEER TO PEER ▸ With Cassandra there is no Master Slave Hierarchy ▸ Every node is the captain of it’s own ship ▸ Processes within Cassandra make this possible ▸ Replication ▸ Consistency Level NODE 1 NODE 2 NODE 3 NODE 4

51. WHAT DOES THIS GET US?

52. WHAT DOES THIS GET US? LINEAR SCALABILITY

53. WHAT DOES THIS GET US? LINEAR SCALABILITY HIGH AVAILABILITY

54. TOPOLOGY

55. CLIENT TOPOLOGY

56. CLIENT TOPOLOGY OPERATION

60. NODE 3 NODE 4 ▸ Replication factor is the number of replicas/puppies ARCHITECTURE REPLICATION IS HOW CASSANDRA DISTRIBUTES DATA NODE 1 NODE 2

63. NODE 3 NODE 4 ▸ The coordinator talks to the client, sending an ack for the write ARCHITECTURE HOW DO WE ACKNOWLEDGE REPLICATION? NODE 1 NODE 2 COORDINATOR

64. NODE 3 NODE 4 ▸ The coordinator talks to the client, sending an ack for the write ARCHITECTURE HOW DO WE ACKNOWLEDGE REPLICATION? NODE 1 NODE 2 COORDINATOR

65. NODE 3 NODE 4 ▸ The coordinator talks to the client, sending an ack for the write ARCHITECTURE HOW DO WE ACKNOWLEDGE REPLICATION? NODE 1 NODE 2 COORDINATOR ack

66. ARCHITECTURE TUNABLE CONSISTENCY LEVELS NODE 1 NODE 2 NODE 3 NODE 4 ▸ One ▸ Quorum ▸ All

67. ONE ARCHITECTURE NODE 1 NODE 2 NODE 3 NODE 4 ▸ One replica acks adorable puppy data

68. ONE ARCHITECTURE NODE 1 NODE 2 NODE 3 NODE 4 ▸ One replica acks adorable puppy data

69. ▸ All replicas ack adorable puppy data NODE 3 NODE 4 ARCHITECTURE ALL NODE 1 NODE 2

72. ARCHITECTURE QUORUM NODE 1 NODE 2 NODE 3 ▸ Quorum = (sum_of_replication_factors / 2) + 1 ▸ How many nodes get puppies if our replication factor is 3, & we want quorum? NODE 4

73. ARCHITECTURE QUORUM NODE 1 NODE 2 NODE 3 ▸ Quorum = (sum_of_replication_factors / 2) + 1 ▸ How many nodes get puppies if our replication factor is 3, & we want quorum? NODE 4

74. MULTI-DC PARAMETERS ▸ Quorum vs. Local_Quorum ▸ One vs. Local_One US-EAST US-WEST

75. PARTITIONER CONSISTENT HASHING Just how is data actually distributed around the cluster?

80. CASSANDRA DATA MODELING SOUNDS HARD

81. CASSANDRA DATA MODELING SOUNDS HARD NOT REALLY

82. GAIN QUERY POWERS

83. GAIN QUERY POWERS WITH CQL

84. DATA STRUCTURES IN CASSANDRA

85. KEYSPACE DATA STRUCTURES IN CASSANDRA

86. KEYSPACE DATA STRUCTURES IN CASSANDRA TABLE

87. KEYSPACE DATA STRUCTURES IN CASSANDRA ROWS TABLE

88. KEYSPACE DATA STRUCTURES IN CASSANDRA ROWS TABLE

89. KEYSPACE PARTITIONS DATA STRUCTURES IN CASSANDRA ROWS TABLE

92. PRIMARY KEY = PARTITION KEY + CLUSTERING COLUMNS

93. PARTITION KEY

94. PARTITION KEY THIS IS HOW YOU RETRIEVE A PARTITION

95. CLUSTERING COLUMNS

96. CLUSTERING COLUMNS THIS IS HOW YOU GET SORTING, ORDER AND UNIQUE IDENTIFICATION

97. WHY ARE CLUSTERING COLUMNS SO COOL?

98. HOW DO I USE CQL?

99. CQLSH HOW DO I USE CQL?

100. SOME EXAMPLES FROM A MOVIE DB

101. CREATE A KEYSPACE CREATE KEYSPACE movielens_small WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'};

102. CREATE A TABLE CREATE TABLE movies ( id uuid PRIMARY KEY, avg_rating float, genres set<text>, name text, release_date date, url text, video_release_date date ) PRIMARY KEY IN WHITE

103. CREATE A TABLE CREATE TABLE ratings_by_movie ( movie_id uuid, user_id uuid, rating int, ts int, PRIMARY KEY (movie_id, user_id) ) PRIMARY KEY IN WHITE

104. INSERT STATEMENT EXAMPLE insert into movies (id, name, genres) values (976de5da-93ae-4bf0-b127-d19eea1c8ea4, 'My Awesome Movie (2016)', {'Comedy'});

105. THIS ALL LOOKS TOO FAMILIAR, DOESN’T IT?

106. BUT REMEMBER…

107. THIRD NORMAL FORM DOESN’T SCALE ▸ UNPREDICTABLE ▸ DATA > MEMORY? ▸ DISK SEEKS ALL DAY ▸ USERS = ANGRY

108. THIRD NORMAL FORM DOESN’T SCALE AWFUL ▸ UNPREDICTABLE ▸ DATA > MEMORY? ▸ DISK SEEKS ALL DAY ▸ USERS = ANGRY

109. DATA MODELING PRO TIPS

110. DATA MODELING PRO TIPS ▸no joins

111. DATA MODELING PRO TIPS ▸no joins ▸query driven methodology, instead

112. DATA MODELING PRO TIPS ▸no joins ▸query driven methodology, instead ▸denormalize

113. DATA MODELING PRO TIPS ▸no joins ▸query driven methodology, instead ▸denormalize ▸disks are cheap

114. JON & DANI, I’M STARTING TO GET COLD FEET!

115. I MISS THE WARM EMBRACE OF RDBMS I DIDN’T HAVE TO DENORMALIZE BACK THEN

116. CHILL OUT

117. & PREPARE TO BE WOWED

118. & PREPARE TO BE WOWED

119. CDM

120. ROLL UP YOUR SLEEVES TYPE STUFF

121. REMEMBER THAT VM?

122. 1.use movielens_small; 2.desc tables; 3.desc movies; 4.select * from movies limit 10; TRY IT OUT

123. YOU SHOULD GET…

124. YOUR 10 MOVIES

125. ADDING ON 5. select * id, name from movies limit 100; 6. PICK YOUR FAVORITE MOVIE BONUS: CAN YOU FIND THE AVERAGE RATINGS FOR YOUR FAVORITE MOVIE?

126. MOVIE ID LIST

127. SELECT A MOVIE

128. TOP GUN EXAMPLE

129. TOP GUN EXAMPLE

130. FIFTH ELEMENT BECAUSE OBVIOUSLY

131. FIFTH ELEMENT BECAUSE OBVIOUSLY

132. NICE WORK YOU!

133. FRIEND #2: SPARK

134. FRIEND #2: SPARK

135. BATCH PROCESSING LOTS OF DATA?

136. STREAMING & REAL TIME AGGREGATION

137. MACHINE LEARNING FOR THE INEVITABLE END OF TIMES

138. GRAPH ANALYTICS

139. 2 WAYS OF WORKING

140. 1. RDD BASED ON FUNCTIONAL PROGRAMMING

141. blah.map( lambda x : x * 2 )

142. COOL BUT NOT EASY

143. COOL BUT NOT EASY

144. 2. DATAFRAMES

145. PRETTY EASY

146. SPARK-CASSANDRA- CONNECTOR

147. TODAY WE TALK BATCH WITH DATAFRAMES AND PYTHON

148. ROLL UP YOUR SLEEVES OPEN THE OSCON TUTORIAL ON YOUR DESKTOP

149. FRIENDSHIP LEVELS

150. OTHER RESOURCES TO LEARN: 1. free courses - www.academy.datatax.com 2. our blogs - www.rustyrazorblade.com & www.dtrapezoid.com 3. our friend’s blog - https:// lostechies.com/ryansvihla/ 4. datastax blog - http:// www.datastax.com/dev/blog

151. THANK YOU, MAGICAL HUMANS @DTRAPEZOID @RUSTYRAZORBLADE

OSCON TALK: Becoming Friends with Cassandra and Spark

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (7)

Similar to OSCON TALK: Becoming Friends with Cassandra and Spark

Similar to OSCON TALK: Becoming Friends with Cassandra and Spark (20)

More from Dani Traphagen

More from Dani Traphagen (8)

Recently uploaded

Recently uploaded (20)

OSCON TALK: Becoming Friends with Cassandra and Spark