C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan Chan

16,074 views
16,200 views

Published on

This session covers our experience with using the Spark and Shark frameworks for running real-time queries on top of Cassandra data.We will start by surveying the current Cassandra analytics landscape, including Hadoop and HIVE, and touch on the use of custom input formats to extract data from Cassandra. We will then dive into Spark and Shark, two memory-based cluster computing frameworks, and how they enable often dramatic improvements in query speed and productivity, over the standard solutions today.

Published in: Technology, Education
0 Comments
19 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
16,074
On SlideShare
0
From Embeds
0
Number of Embeds
7,423
Actions
Shares
0
Downloads
239
Comments
0
Likes
19
Embeds 0
No embeds

No notes for slide

C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan Chan

  1. 1. Real-time Analytics withCassandra, Spark and Shark
  2. 2. Who is this guy• Staff Engineer, Compute and Data Services, Ooyala• Building multiple web-scale real-time systems on top of C*, Kafka,Storm, etc.• Scala/Akka guy• Very excited by open source, big data projects - share some today• @evanfchan
  3. 3. Agenda• Ooyala and Cassandra• What problem are we trying to solve?• Spark and Shark• Our Spark/Cassandra Architecture• Demo
  4. 4. Cassandra at OoyalaWho is Ooyala, and how we use Cassandra
  5. 5. CONFIDENTIAL—DO NOT DISTRIBUTEOOYALAPowering personalized videoexperiences across all screens.5
  6. 6. CONFIDENTIAL—DO NOT DISTRIBUTE 6CONFIDENTIAL—DO NOT DISTRIBUTEFounded in 2007Commercially launch in 2009230+ employees in Silicon Valley, LA, NYC,London, Paris, Tokyo, Sydney & GuadalajaraGlobal footprint, 200M unique users,110+ countries, and more than 6,000 websitesOver 1 billion videos played per monthand 2 billion analytic events per day25% of U.S. online viewers watch videopowered by OoyalaCOMPANY OVERVIEW
  7. 7. CONFIDENTIAL—DO NOT DISTRIBUTE 7TRUSTED VIDEO PARTNERSTRATEGIC PARTNERSCUSTOMERSCONFIDENTIAL—DO NOT DISTRIBUTE
  8. 8. We are a large Cassandra user• 12 clusters ranging in size from 3 to 115 nodes• Total of 28TB of data managed over ~200 nodes• Largest cluster - 115 nodes, 1.92PB storage, 15TBRAM• Over 2 billion C* column writes per day• Powers all of our analytics infrastructure
  9. 9. What problem are we trying tosolve?Lots of data, complex queries, answered really quickly... but how??
  10. 10. From mountains of useless data...
  11. 11. To nuggets of truth...
  12. 12. To nuggets of truth...• Quickly• Painlessly• At  scale?
  13. 13. Today: Precomputed aggregates• Video metrics computed along several high cardinality dimensions• Very fast lookups, but inflexible, and hard to change• Most computed aggregates are never read• What if we need more dynamic queries?– Top content for mobile users in France– Engagement curves for users who watched recommendations– Data mining, trends, machine learning
  14. 14. The static - dynamic continuum• Super fast lookups• Inflexible, wasteful• Best for 80% mostcommon queries• Always compute resultsfrom raw data• Flexible but slow100% Precomputation 100% Dynamic
  15. 15. Where we want to bePartly dynamic• Pre-aggregate mostcommon queries• Flexible, fast dynamicqueries• Easily generate manymaterialized views
  16. 16. Industry Trends• Fast execution frameworks– Impala• In-memory databases– VoltDB, Druid• Streaming and real-time• Higher-level, productive data frameworks– Cascading, Hive, Pig
  17. 17. Why Spark and Shark?“Lightning-fast in-memory cluster computing”
  18. 18. Introduction to Spark• In-memory distributed computing framework• Created by UC Berkeley AMP Lab in 2010• Targeted problems that MR is bad at:– Iterative algorithms (machine learning)– Interactive data mining• More general purpose than Hadoop MR• Active contributions from ~ 15 companies
  19. 19. HDFSMapReduceMapReducemap()join()cache()transform
  20. 20. Throughput: Memory is king0 37500 75000 112500 150000C*, cold cacheC*, warm cacheSpark RDD6-­‐node  C*/DSE  1.1.9  cluster,Spark  0.7.0
  21. 21. Developers love it• “I wrote my first aggregation job in 30 minutes”• High level “distributed collections” API• No Hadoop cruft• Full power of Scala, Java, Python• Interactive REPL shell• EASY testing!!• Low latency - quick development cycles
  22. 22. Spark word count examplefile = spark.textFile("hdfs://...") file.flatMap(line => line.split(" "))    .map(word => (word, 1))    .reduceByKey(_ + _)1 package org.myorg;23 import java.io.IOException;4 import java.util.*;56 import org.apache.hadoop.fs.Path;7 import org.apache.hadoop.conf.*;8 import org.apache.hadoop.io.*;9 import org.apache.hadoop.mapreduce.*;10 import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;11 import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;12 import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;13 import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;1415 public class WordCount {1617 public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {18 private final static IntWritable one = new IntWritable(1);19 private Text word = new Text();2021 public void map(LongWritable key, Text value, Context context) throws IOException,InterruptedException {22 String line = value.toString();23 StringTokenizer tokenizer = new StringTokenizer(line);24 while (tokenizer.hasMoreTokens()) {25 word.set(tokenizer.nextToken());26 context.write(word, one);27 }28 }29 }3031 public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {3233 public void reduce(Text key, Iterable<IntWritable> values, Context context)34 throws IOException, InterruptedException {35 int sum = 0;36 for (IntWritable val : values) {37 sum += val.get();38 }39 context.write(key, new IntWritable(sum));40 }41 }4243 public static void main(String[] args) throws Exception {44 Configuration conf = new Configuration();4546 Job job = new Job(conf, "wordcount");4748 job.setOutputKeyClass(Text.class);49 job.setOutputValueClass(IntWritable.class);5051 job.setMapperClass(Map.class);52 job.setReducerClass(Reduce.class);5354 job.setInputFormatClass(TextInputFormat.class);55 job.setOutputFormatClass(TextOutputFormat.class);5657 FileInputFormat.addInputPath(job, new Path(args[0]));58 FileOutputFormat.setOutputPath(job, new Path(args[1]));5960 job.waitForCompletion(true);61 }6263 }
  23. 23. The Spark EcosystemBagel  -­‐  Pregel  on  SparkHIVE  on  SparkSpark  Streaming  -­‐  discreRzed  stream  processingSparkTachyon  -­‐  in-­‐memory  caching  DFS
  24. 24. Shark - HIVE on Spark• 100% HiveQL compatible• 10-100x faster than HIVE, answers in seconds• Reuse UDFs, SerDe’s, StorageHandlers• Can use DSE / CassandraFS for Metastore• Easy Scala/Java integration via Spark - easier thanwriting UDFs
  25. 25. Our new analytics architectureHow we integrate Cassandra and Spark/Shark
  26. 26. From raw events to fast queriesIngesRon C*event  storeRaw  EventsRaw  EventsRaw  EventsSparkSparkSparkView  1View  2View  3SparkSharkPredefined  queriesAd-­‐hoc  HiveQL
  27. 27. Our Spark/Shark/Cassandra StackNode1CassandraInputFormatSerDeSpark  WorkerSharkNode2CassandraInputFormatSerDeSpark  WorkerSharkNode3CassandraInputFormatSerDeSpark  WorkerSharkSpark  Master Job  Server
  28. 28. Event Store Cassandra schemat0 t1 t2 t3 t42013-­‐04-­‐05T00:00Z#id1{event0:a0}{event1:a1}{event2:a2}{event3:a3}{event4:a4}ipaddr:10.20.30.40:t1 videoId:45678:t1 providerId:500:t02013-­‐04-­‐05T00:00Z#id1Event  CFEventAfr  CF
  29. 29. Unpacking raw eventst0 t12013-­‐04-­‐05T00:00Z#id1{video: 10,type:5}{video: 11,type:1}2013-­‐04-­‐05T00:00Z#id2{video: 20,type:5}{video: 25,type:9}UserID Video Typeid1 10 5
  30. 30. Unpacking raw eventst0 t12013-­‐04-­‐05T00:00Z#id1{video: 10,type:5}{video: 11,type:1}2013-­‐04-­‐05T00:00Z#id2{video: 20,type:5}{video: 25,type:9}UserID Video Typeid1 10 5id1 11 1
  31. 31. Unpacking raw eventst0 t12013-­‐04-­‐05T00:00Z#id1{video: 10,type:5}{video: 11,type:1}2013-­‐04-­‐05T00:00Z#id2{video: 20,type:5}{video: 25,type:9}UserID Video Typeid1 10 5id1 11 1id2 20 5
  32. 32. Unpacking raw eventst0 t12013-­‐04-­‐05T00:00Z#id1{video: 10,type:5}{video: 11,type:1}2013-­‐04-­‐05T00:00Z#id2{video: 20,type:5}{video: 25,type:9}UserID Video Typeid1 10 5id1 11 1id2 20 5id2 25 9
  33. 33. Tips for InputFormat Development• Know which target platforms you are developing for– Which API to write against? New? Old? Both?• Be prepared to spend time tuning your split computation– Low latency jobs require fast splits• Consider sorting row keys by token for data locality• Implement predicate pushdown for HIVE SerDe’s– Use your indexes to reduce size of dataset
  34. 34. Example: OLAP processingt02013-­‐04-­‐05T00:00Z#id1{video:10,type:5}2013-­‐04-­‐05T00:00Z#id2{video:20,type:5}C*  eventsOLAP  AggregatesOLAP  AggregatesOLAP  AggregatesCached  Materialized  ViewsSparkSparkSparkUnionQuery  1:  Plays  by  ProviderQuery  2:  Top  content  for  mobile
  35. 35. Performance numbersSpark:  C*  -­‐>  OLAP  aggregatescold  cache,  1.4  million  events130  secondsC*  -­‐>  OLAP  aggregateswarmed  cache20-­‐30  secondsOLAP  aggregate  query  via  Spark(56k  records)60  ms6-­‐node  C*/DSE  1.1.9  cluster,Spark  0.7.0
  36. 36. Spark: Under the hoodMap DatasetReduce MapDriver Map DatasetReduce MapMap DatasetReduce MapOne  executor  process  per  nodeDriver
  37. 37. Fault Tolerance• Cached dataset lives in Java Heap only - what if process dies?• Spark lineage - automatic recomputation from source, but this isexpensive!• Can also replicate cached dataset to survive single node failures• Persist materialized views back to C*, then load into cache -- nowrecovery path is much faster• Persistence also enables multiple processes to hold cached dataset
  38. 38. Demo time
  39. 39. Shark Demo• Local shark node, 1 core, MBP• How to create a table from C* using our inputformat• Creating a cached Shark table• Running fast queries
  40. 40. Backup Slides• THE NEXT FEW SLIDES ARE STRICTLY BACKUP IN CASELIVE DEMO DOESN’T WORK
  41. 41. Creating a Shark Table from InputFormat
  42. 42. Creating a cached table
  43. 43. Doing a row count ... don’t try in HIVE!
  44. 44. Top k providerId query
  45. 45. THANK YOU

×