Dean Wampler, profile picture
Uploaded byDean Wampler
PDF, PPTX7,014 views

Spark the next top compute model

The document discusses the limitations of the Hadoop MapReduce model for big data processing and introduces Apache Spark as a more efficient alternative. Spark provides a flexible compute model with resilient distributed datasets (RDDs) that enable faster job execution through in-memory data caching, avoiding the bottleneck of frequent disk flushes inherent in MapReduce. The document illustrates how Spark simplifies the implementation of complex algorithms compared to Hadoop, using the example of creating an inverted index.

Embed presentation

Download as PDF, PPTX
Why Spark Is the Next Top (Compute) Model Philly  ETE  2014 April  22-­‐23,  2014 @deanwampler polyglotprogramming.com/talks Thursday, May 1, 14 Copyright (c) 2014, Dean Wampler, All Rights Reserved, unless otherwise noted. Image: Detail of the London Eye
Dean Wampler dean.wampler@typesafe.com polyglotprogramming.com/talks @deanwampler Thursday, May 1, 14 About me. You can find this presentation and others on Big Data and Scala at polyglotprogramming.com. Programming Scala, 2nd Edition is forthcoming. photo: Dusk at 30,000 ft above the Central Plains of the U.S. on a Winter’s Day.
Or this? Thursday, May 1, 14 photo: https://twitter.com/john_overholt/status/447431985750106112/photo/1
Hadoop circa 2013 Thursday, May 1, 14 The state of Hadoop as of last year. Image: Detail of the London Eye
Hadoop v2.X Cluster node DiskDiskDiskDiskDisk Node Mgr Data Node node DiskDiskDiskDiskDisk Node Mgr Data Node node DiskDiskDiskDiskDisk Node Mgr Data Node master Resource Mgr Name Node master Name Node Thursday, May 1, 14 Schematic view of a Hadoop 2 cluster. For a more precise definition of the services and what they do, see e.g., http://hadoop.apache.org/ docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/YARN.html We aren’t interested in great details at this point, but we’ll call out a few useful things to know.
Hadoop v2.X Cluster node DiskDiskDiskDiskDisk Node Mgr Data Node node DiskDiskDiskDiskDisk Node Mgr Data Node node DiskDiskDiskDiskDisk Node Mgr Data Node master Resource Mgr Name Node master Name Node Resource and Node Managers Thursday, May 1, 14 Hadoop 2 uses YARN to manage resources via the master Resource Manager, which includes a pluggable job scheduler and an Applications Manager. It coordinates with the Node Manager on each node to schedule jobs and provide resources. Other services involved, including application-specific Containers and Application Masters are not shown.
Hadoop v2.X Cluster node DiskDiskDiskDiskDisk Node Mgr Data Node node DiskDiskDiskDiskDisk Node Mgr Data Node node DiskDiskDiskDiskDisk Node Mgr Data Node master Resource Mgr Name Node master Name Node Name Node and Data Nodes Thursday, May 1, 14 Hadoop 2 clusters federate the Name node services that manage the file system, HDFS. They provide horizontal scalability of file-system operations and resiliency when service instances fail. The data node services manage individual blocks for files.
MapReduce The classic compute model for Hadoop Thursday, May 1, 14 Hadoop 2 clusters federate the Name node services that manage the file system, HDFS. They provide horizontal scalability of file-system operations and resiliency when service instances fail. The data node services manage individual blocks for files.
1 map step + 1 reduce step (wash, rinse, repeat) MapReduce Thursday, May 1, 14 You get 1 map step (although there is limited support for chaining mappers) and 1 reduce step. If you can’t implement an algorithm in these two steps, you can chain jobs together, but you’ll pay a tax of flushing the entire data set to disk between these jobs.
Example: Inverted Index MapReduce Thursday, May 1, 14 The inverted index is a classic algorithm needed for building search engines.
Thursday, May 1, 14 Before running MapReduce, crawl teh interwebs, find all the pages, and build a data set of URLs -> doc contents, written to flat files in HDFS or one of the more “sophisticated” formats.
Thursday, May 1, 14 Now we’re running MapReduce. In the map step, a task (JVM process) per file *block* (64MB or larger) reads the rows, tokenizes the text and outputs key-value pairs (“tuples”)...
Map Task (hadoop,(wikipedia.org/hadoop,1)) (mapreduce,(wikipedia.org/hadoop, 1)) (hdfs,(wikipedia.org/hadoop, 1)) (provides,(wikipedia.org/hadoop,1)) (and,(wikipedia.org/hadoop,1)) Thursday, May 1, 14 ... the keys are each word found and the values are themselves tuples, each URL and the count of the word. In our simplified example, there are typically only single occurrences of each work in each document. The real occurrences are interesting because if a word is mentioned a lot in a document, the chances are higher that you would want to find that document in a search for that word.
Thursday, May 1, 14
Thursday, May 1, 14 The output tuples are sorted by key locally in each map task, then “shuffled” over the cluster network to reduce tasks (each a JVM process, too), where we want all occurrences of a given key to land on the same reduce task.
Thursday, May 1, 14 Finally, each reducer just aggregates all the values it receives for each key, then writes out new files to HDFS with the words and a list of (URL-count) tuples (pairs).
Altogether... Thursday, May 1, 14 Finally, each reducer just aggregates all the values it receives for each key, then writes out new files to HDFS with the words and a list of (URL-count) tuples (pairs).
What’s not to like?Thursday, May 1, 14 This seems okay, right? What’s wrong with it?
Most algorithms are much harder to implement in this restrictive map-then-reduce model. Awkward Thursday, May 1, 14 Writing MapReduce jobs requires arcane, specialized skills that few master. For a good overview, see http://lintool.github.io/ MapReduceAlgorithms/.
Lack of flexibility inhibits optimizations, too. Awkward Thursday, May 1, 14 The inflexible compute model leads to complex code to improve performance where hacks are used to work around the limitations. Hence, optimizations are hard to implement. The Spark team has commented on this, see http://databricks.com/blog/2014/03/26/Spark-SQL- manipulating-structured-data-using-Spark.html
Full dump to disk between jobs. Performance Thursday, May 1, 14 Sequencing jobs wouldn’t be so bad if the “system” was smart enough to cache data in memory. Instead, each job dumps everything to disk, then the next job reads it back in again. This makes iterative algorithms particularly painful.
Enter Spark spark.apache.org Thursday, May 1, 14
Can be run in: •YARN (Hadoop 2) •Mesos (Cluster management) •EC2 •Standalone mode Cluster Computing Thursday, May 1, 14 If you have a Hadoop cluster, you can run Spark as a seamless compute engine on YARN. (You can also use pre-YARN Hadoop v1 clusters, but there you have manually allocate resources to the embedded Spark cluster vs Hadoop.) Mesos is a general-purpose cluster resource manager that can also be used to manage Hadoop resources. Scripts for running a Spark cluster in EC2 are available. Standalone just means you run Spark’s built-in support for clustering (or run locally on a single box - e.g., for development). EC2 deployments are usually standalone...
Fine-grained “combinators” for composing algorithms. Compute Model Thursday, May 1, 14 Once you learn the core set of primitives, it’s easy to compose non-trivial algorithms with little code.
RDDs: Resilient, Distributed Datasets Compute Model Thursday, May 1, 14 RDDs shard the data over a cluster, like a virtualized, distributed collection (analogous to HDFS). They support intelligent caching, which means no naive flushes of massive datasets to disk. This feature alone allows Spark jobs to run 10-100x faster than comparable MapReduce jobs! The “resilient” part means they will reconstitute shards lost due to process/server crashes.
Written in Scala, with Java and Python APIs. Compute Model Thursday, May 1, 14 Once you learn the core set of primitives, it’s easy to compose non-trivial algorithms with little code.
Inverted Index in MapReduce (Java). Thursday, May 1, 14 Let’s  see  an  an  actual  implementaEon  of  the  inverted  index.  First,  a  Hadoop  MapReduce  (Java)  version,  adapted  from  hPps:// developer.yahoo.com/hadoop/tutorial/module4.html#soluEon  It’s  about  90  lines  of  code,  but  I  reformaPed  to  fit  bePer. This  is  also  a  slightly  simpler  version  that  the  one  I  diagrammed.  It  doesn’t  record  a  count  of  each  word  in  a  document;  it  just  writes   (word,doc-­‐Etle)  pairs  out  of  the  mappers  and  the  final  (word,list)  output  by  the  reducers  just  has  a  list  of  documentaEons,  hence  repeats.   A  second  job  would  be  necessary  to  count  the  repeats.
import java.io.IOException; import java.util.*; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.*; import org.apache.hadoop.mapred.*; public class LineIndexer { public static void main(String[] args) { JobClient client = new JobClient(); JobConf conf = new JobConf(LineIndexer.class); conf.setJobName("LineIndexer"); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(Text.class); FileInputFormat.addInputPath(conf, new Path("input")); Thursday, May 1, 14 I’ve  shortened  the  original  code  a  bit,  e.g.,  using  *  import  statements  instead  of  separate  imports  for  each  class. I’m  not  going  to  explain  every  line  ...  nor  most  lines. Everything  is  in  one  outer  class.  We  start  with  a  main  rou?ne  that  sets  up  the  job.  LoBa  boilerplate... I  used  yellow  for  method  calls,  because  methods  do  the  real  work!!  But  no?ce  that  the  func?ons  in  this  code  don’t  really  do  a  whole  lot...
JobClient client = new JobClient(); JobConf conf = new JobConf(LineIndexer.class); conf.setJobName("LineIndexer"); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(Text.class); FileInputFormat.addInputPath(conf, new Path("input")); FileOutputFormat.setOutputPath(conf, new Path("output")); conf.setMapperClass( LineIndexMapper.class); conf.setReducerClass( LineIndexReducer.class); client.setConf(conf); try { JobClient.runJob(conf); Thursday, May 1, 14 boilerplate...
LineIndexMapper.class); conf.setReducerClass( LineIndexReducer.class); client.setConf(conf); try { JobClient.runJob(conf); } catch (Exception e) { e.printStackTrace(); } } public static class LineIndexMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text> { private final static Text word = new Text(); private final static Text location = Thursday, May 1, 14 main ends with a try-catch clause to run the job.
public static class LineIndexMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text> { private final static Text word = new Text(); private final static Text location = new Text(); public void map( LongWritable key, Text val, OutputCollector<Text, Text> output, Reporter reporter) throws IOException { FileSplit fileSplit = (FileSplit)reporter.getInputSplit(); String fileName = fileSplit.getPath().getName(); location.set(fileName); Thursday, May 1, 14 This is the LineIndexMapper class for the mapper. The map method does the real work of tokenization and writing the (word, document-name) tuples.
FileSplit fileSplit = (FileSplit)reporter.getInputSplit(); String fileName = fileSplit.getPath().getName(); location.set(fileName); String line = val.toString(); StringTokenizer itr = new StringTokenizer(line.toLowerCase()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); output.collect(word, location); } } } public static class LineIndexReducer extends MapReduceBase implements Reducer<Text, Text, Thursday, May 1, 14 The rest of the LineIndexMapper class and map method.
public static class LineIndexReducer extends MapReduceBase implements Reducer<Text, Text, Text, Text> { public void reduce(Text key, Iterator<Text> values, OutputCollector<Text, Text> output, Reporter reporter) throws IOException { boolean first = true; StringBuilder toReturn = new StringBuilder(); while (values.hasNext()) { if (!first) toReturn.append(", "); first=false; toReturn.append( values.next().toString()); } output.collect(key, new Text(toReturn.toString())); Thursday, May 1, 14 The reducer class, LineIndexReducer, with the reduce method that is called for each key and a list of values for that key. The reducer is stupid; it just reformats the values collection into a long string and writes the final (word,list-string) output.
boolean first = true; StringBuilder toReturn = new StringBuilder(); while (values.hasNext()) { if (!first) toReturn.append(", "); first=false; toReturn.append( values.next().toString()); } output.collect(key, new Text(toReturn.toString())); } } } Thursday, May 1, 14 EOF
Altogether import java.io.IOException; import java.util.*; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.*; import org.apache.hadoop.mapred.*; public class LineIndexer { public static void main(String[] args) { JobClient client = new JobClient(); JobConf conf = new JobConf(LineIndexer.class); conf.setJobName("LineIndexer"); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(Text.class); FileInputFormat.addInputPath(conf, new Path("input")); FileOutputFormat.setOutputPath(conf, new Path("output")); conf.setMapperClass( LineIndexMapper.class); conf.setReducerClass( LineIndexReducer.class); client.setConf(conf); try { JobClient.runJob(conf); } catch (Exception e) { e.printStackTrace(); } } public static class LineIndexMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text> { private final static Text word = new Text(); private final static Text location = new Text(); public void map( LongWritable key, Text val, OutputCollector<Text, Text> output, Reporter reporter) throws IOException { FileSplit fileSplit = (FileSplit)reporter.getInputSplit(); String fileName = fileSplit.getPath().getName(); location.set(fileName); String line = val.toString(); StringTokenizer itr = new StringTokenizer(line.toLowerCase()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); output.collect(word, location); } } } public static class LineIndexReducer extends MapReduceBase implements Reducer<Text, Text, Text, Text> { public void reduce(Text key, Iterator<Text> values, OutputCollector<Text, Text> output, Reporter reporter) throws IOException { boolean first = true; StringBuilder toReturn = new StringBuilder(); while (values.hasNext()) { if (!first) toReturn.append(", "); first=false; toReturn.append( values.next().toString()); } output.collect(key, new Text(toReturn.toString())); } } } Thursday, May 1, 14 The  whole  shebang  (6pt.  font)
Inverted Index in Spark (Scala). Thursday, May 1, 14 This  code  is  approximately  45  lines,  but  it  does  more  than  the  previous  Java  example,  it  implements  the  original  inverted  index  algorithm  I   diagrammed  where  word  counts  are  computed  and  included  in  the  data.
import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ object InvertedIndex { def main(args: Array[String]) = { val sc = new SparkContext( "local", "Inverted Index") sc.textFile("data/crawl") .map { line => val array = line.split("t", 2) (array(0), array(1)) } .flatMap { case (path, text) => text.split("""W+""") map { word => (word, path) } } Thursday, May 1, 14 It  starts  with  imports,  then  declares  a  singleton  object  (a  first-­‐class  concept  in  Scala),  with  a  main  rou?ne  (as  in  Java). The  methods  are  colored  yellow  again.  Note  this  ?me  how  dense  with  meaning  they  are  this  ?me.
val sc = new SparkContext( "local", "Inverted Index") sc.textFile("data/crawl") .map { line => val array = line.split("t", 2) (array(0), array(1)) } .flatMap { case (path, text) => text.split("""W+""") map { word => (word, path) } } .map { case (w, p) => ((w, p), 1) } .reduceByKey { case (n1, n2) => n1 + n2 } Thursday, May 1, 14 You  being  the  workflow  by  declaring  a  SparkContext  (in  “local”  mode,  in  this  case).  The  rest  of  the  program  is  a  sequence  of  func?on  calls,   analogous  to  “pipes”  we  connect  together  to  perform  the  data  flow. Next  we  read  one  or  more  text  files.  If  “data/crawl”  has  1  or  more  Hadoop-­‐style  “part-­‐NNNNN”  files,  Spark  will  process  all  of  them  (in  parallel  if   running  a  distributed  configura?on;  they  will  be  processed  synchronously  in  local  mode).
.map { line => val array = line.split("t", 2) (array(0), array(1)) } .flatMap { case (path, text) => text.split("""W+""") map { word => (word, path) } } .map { case (w, p) => ((w, p), 1) } .reduceByKey { case (n1, n2) => n1 + n2 } .map { case ((w, p), n) => (w, (p, n)) } .groupBy { Thursday, May 1, 14 sc.textFile  returns  an  RDD  with  a  string  for  each  line  of  input  text.  So,  the  first  thing  we  do  is  map  over  these  strings  to  extract  the  original   document  id  (i.e.,  file  name),  followed  by  the  text  in  the  document,  all  on  one  line.  We  assume  tab  is  the  separator.  “(array(0),  array(1))”  returns  a   two-­‐element  “tuple”.  Think  of  the  output  RDD  has  having  a  schema  “String  fileName,  String  text”.    
} .flatMap { case (path, text) => text.split("""W+""") map { word => (word, path) } } .map { case (w, p) => ((w, p), 1) } .reduceByKey { case (n1, n2) => n1 + n2 } .map { case ((w, p), n) => (w, (p, n)) } .groupBy { case (w, (p, n)) => w } .map { Beautiful, no? Thursday, May 1, 14 flatMap  maps  over  each  of  these  2-­‐element  tuples.  We  split  the  text  into  words  on  non-­‐alphanumeric  characters,  then  output  collec?ons  of  word   (our  ul?mate,  final  “key”)  and  the  path.  Each  line  is  converted  to  a  collec?on  of  (word,path)  pairs,  so  flatMap  converts  the  collec?on  of  collec?ons   into  one  long  “flat”  collec?on  of  (word,path)  pairs. Then  we  map  over  these  pairs  and  add  a  single  count  of  1.
Thursday, May 1, 14 Another  example  of  a  beau?ful  and  profound  DSL,  in  this  case  from  the  world  of  Physics:  Maxwell’s  equa?ons:  hBp://upload.wikimedia.org/ wikipedia/commons/c/c4/Maxwell'sEqua?ons.svg
} .reduceByKey { case (n1, n2) => n1 + n2 } .map { case ((w, p), n) => (w, (p, n)) } .groupBy { case (w, (p, n)) => w } .map { case (w, seq) => val seq2 = seq map { case (_, (p, n)) => (p, n) } (w, seq2.mkString(", ")) } .saveAsTextFile(argz.outpath) sc.stop() (word1,  (path1,  n1)) (word2,  (path2,  n2)) ... Thursday, May 1, 14 reduceByKey  does  an  implicit  “group  by”  to  bring  together  all  occurrences  of  the  same  (word,  path)  and  then  sums  up  their  counts. Note  the  input  to  the  next  map  is  now  ((word,  path),  n),  where  n  is  now  >=  1.  We  transform  these  tuples  into  the  form  we  actually  want,  (word,   (path,  n)).
} .groupBy { case (w, (p, n)) => w } .map { case (w, seq) => val seq2 = seq map { case (_, (p, n)) => (p, n) } (w, seq2.mkString(", ")) } .saveAsTextFile(argz.outpath) sc.stop() } } (word,  seq((word,  (path1,  n1)),  (word,  (path2,  n2)),  ...)) (word,  “(path1,  n1),  (path2,  n2),  ...”) Thursday, May 1, 14 Now  we  do  an  explicit  group  by  to  bring  all  the  same  words  together.  The  output    will  be  (word,  (word,  (path1,  n1)),  (word,  (path2,  n2)),  ...). The  last  map  removes  the  redundant  “word”  values  in  the  sequences  of  the  previous  output.  It  outputs  the  sequence  as  a  final  string  of  comma-­‐ separated  (path,n)  pairs. We  finish  by  saving  the  output  as  text  file(s)  and  stopping  the  workflow.  
import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ object InvertedIndex { def main(args: Array[String]) = { val sc = new SparkContext( "local", "Inverted Index") sc.textFile("data/crawl") .map { line => val array = line.split("t", 2) (array(0), array(1)) } .flatMap { case (path, text) => text.split("""W+""") map { word => (word, path) } } .map { case (w, p) => ((w, p), 1) } .reduceByKey { case (n1, n2) => n1 + n2 } .map { case ((w, p), n) => (w, (p, n)) } .groupBy { case (w, (p, n)) => w } .map { case (w, seq) => val seq2 = seq map { case (_, (p, n)) => (p, n) } (w, seq2.mkString(", ")) } .saveAsTextFile(argz.outpath) sc.stop() } } Altogether Thursday, May 1, 14 The  whole  shebang  (12pt.  font,  this  ?me)
sc.textFile("data/crawl") .map { line => val array = line.split("t", 2) (array(0), array(1)) } .flatMap { case (path, text) => text.split("""W+""") map { word => (word, path) } } .map { case (w, p) => ((w, p), 1) } .reduceByKey{ case (n1, n2) => n1 + n2 } .map { case ((w, p), n) => (w, (p, n)) } Powerful Combinators! Thursday, May 1, 14 I’ve  shortened  the  original  code  a  bit,  e.g.,  using  *  import  statements  instead  of  separate  imports  for  each  class. I’m  not  going  to  explain  every  line  ...  nor  most  lines. Everything  is  in  one  outer  class.  We  start  with  a  main  rou?ne  that  sets  up  the  job.  LoBa  boilerplate...
The Spark version took me ~30 minutes to write. Thursday, May 1, 14 Once  you  learn  the  core  primiEves  I  used,  and  a  few  tricks  for  manipulaEng  the  RDD  tuples,  you  can  very  quickly  build  complex  algorithms   for  data  processing! The  Spark  API  allowed  us  to  focus  almost  exclusively  on  the  “domain”  of  data  transformaEons,  while  the  Java  MapReduce  version  (which   does  less),  forced  tedious  aPenEon  to  infrastructure  mechanics.  
Shark: Hive (SQL query tool) ported to Spark. SQL! Thursday, May 1, 14 Use SQL when you can!
Use a SQL query when you can!! Thursday, May 1, 14
CREATE EXTERNAL TABLE stocks ( symbol STRING, ymd STRING, price_open STRING, price_close STRING, shares_traded INT) LOCATION “hdfs://data/stocks”; -- Year-over-year average closing price. SELECT year(ymd), avg(price_close) FROM stocks WHERE symbol = ‘AAPL’ GROUP BY year(ymd); Hive  Query   Language. Thursday, May 1, 14 Hive  and  the  Spark  port,  Shark,  let  you  use  SQL  to  query  and  manipulate  structured  and  semistructured  data.  By  default,  tab-­‐delimited  text  files   will  be  assumed  for  the  files  found  in  “data/stocks”.  Alterna?ves  can  be  configured  as  part  of  the  table  metadata.
~10-100x the performance of Hive, due to in-memory caching of RDDs & better Spark abstractions. Shark Thursday, May 1, 14
Spark SQL: Next generation SQL query tool/API. Did I mention SQL? Thursday, May 1, 14 A new query optimizer. It will become the basis for Shark internals, replacing messing Hive code that is hard to reason about, extend, and debug.
Combine Shark SQL queries with Machine Learning code. Thursday, May 1, 14 We’ll  use  the  Spark  “MLlib”  in  the  example,  then  return  to  it  in  a  moment.
CREATE TABLE Users( userId INT, name STRING, email STRING, age INT, latitude DOUBLE, longitude DOUBLE, subscribed BOOLEAN); CREATE TABLE Events( userId INT, action INT); Hive/Shark  table   definiEons (not  Scala). Thursday, May 1, 14 This  example  adapted  from  the  following  blog  post  announcing  Spark  SQL: hBp://databricks.com/blog/2014/03/26/Spark-­‐SQL-­‐manipula?ng-­‐structured-­‐data-­‐using-­‐Spark.html Assume  we  have  these  Hive/Shark  tables,  with  data  about  our  users  and  events  that  have  occurred.
val trainingDataTable = sql(""" SELECT e.action, u.age, u.latitude, u.longitude FROM Users u JOIN Events e ON u.userId = e.userId""") val trainingData = trainingDataTable map { row => val features = Array[Double](row(1), row(2), row(3)) LabeledPoint(row(0), features) } val model = new LogisticRegressionWithSGD() .run(trainingData) val allCandidates = sql(""" Spark  Scala. Thursday, May 1, 14 Here  is  some  Spark  (Scala)  code  with  an  embedded  SQL/Shark  query  that  joins  the  Users  and  Events  tables.  The  “””...”””  string  allows  embedded   line  feeds. The  “sql”  func?on  returns  an  RDD,  which  we  then  map  over  to  create  LabeledPoints,  an  object  used  in  Spark’s  MLlib  (machine  learning  library)  for   a  recommenda?on  engine.  The  “label”  is  the  kind  of  event  and  the  user’s  age  and  lat/long  coordinates  are  the  “features”  used  for  making   recommenda?ons.  (E.g.,  if  you’re  25  and  near  a  certain  loca?on  in  the  city,  you  might  be  interested  a  nightclub  near  by...)
val model = new LogisticRegressionWithSGD() .run(trainingData) val allCandidates = sql(""" SELECT userId, age, latitude, longitude FROM Users WHERE subscribed = FALSE""") case class Score( userId: Int, score: Double) val scores = allCandidates map { row => val features = Array[Double](row(1), row(2), row(3)) Score(row(0), model.predict(features)) } // Hive table scores.registerAsTable("Scores") Thursday, May 1, 14 Next  we  train  the  recommenda?on  engine,  using  a  “logis?c  regression”  fit  to  the  training  data,  where  “stochas?c  gradient  descent”  (SGD)  is  used   to  train  it.  (This  is  a  standard  tool  set  for  recommenda?on  engines;  see  for  example:  hBp://www.cs.cmu.edu/~wcohen/10-­‐605/assignments/ sgd.pdf)
.run(trainingData) val allCandidates = sql(""" SELECT userId, age, latitude, longitude FROM Users WHERE subscribed = FALSE""") case class Score( userId: Int, score: Double) val scores = allCandidates map { row => val features = Array[Double](row(1), row(2), row(3)) Score(row(0), model.predict(features)) } // Hive table scores.registerAsTable("Scores") val topCandidates = sql(""" SELECT u.name, u.email FROM Scores s Thursday, May 1, 14 Now  run  a  query  against  all  users  who  aren’t  already  subscribed  to  no?fica?ons.
case class Score( userId: Int, score: Double) val scores = allCandidates map { row => val features = Array[Double](row(1), row(2), row(3)) Score(row(0), model.predict(features)) } // Hive table scores.registerAsTable("Scores") val topCandidates = sql(""" SELECT u.name, u.email FROM Scores s JOIN Users u ON s.userId = u.userId ORDER BY score DESC LIMIT 100""")Thursday, May 1, 14 Declare  a  class  to  hold  each  user’s  “score”  as  produced  by  the  recommenda?on  engine  and  map  the  “all”  query  results  to  Scores. Then  “register”  the  scores  RDD  as  a  “Scores”  table  in  Hive’s  metadata  respository.  This  is  equivalent  to  running  a  “CREATE  TABLE  Scores  ...”   command  at  the  Hive/Shark  prompt!
// Hive table scores.registerAsTable("Scores") val topCandidates = sql(""" SELECT u.name, u.email FROM Scores s JOIN Users u ON s.userId = u.userId ORDER BY score DESC LIMIT 100""") Thursday, May 1, 14 Finally,  run  a  new  query  to  find  the  people  with  the  highest  scores  that  aren’t  already  subscribing  to  no?fica?ons.  You  might  send  them  an  email   next  recommending  that  they  subscribe...
Spark Streaming: Use the same abstractions for real-time, event streaming. Cluster Computing Thursday, May 1, 14 Once you learn the core set of primitives, it’s easy to compose non-trivial algorithms with little code.
Spark Job Iteration #1 1 second Data Stream… Spark Job Iteration #2 Thursday, May 1, 14 You can specify the granularity, such as all events in 1 second windows, then your Spark job is patched each window of data for processing.
Very similar code... Thursday, May 1, 14
val sc = new SparkContext(...) val ssc = new StreamingContext( sc, Seconds(1)) // A DStream that will listen to server:port val lines = ssc.socketTextStream(server, port) // Word Count... val words = lines flatMap { line => line.split("""W+""") } val pairs = words map (word => (word, 1)) val wordCounts = pairs reduceByKey ((n1, n2) => n1 + n2) wordCount.print() // print a few counts... Thursday, May 1, 14 This  example  adapted  from  the  following  page  on  the  Spark  website: hBp://spark.apache.org/docs/0.9.0/streaming-­‐programming-­‐guide.html#a-­‐quick-­‐example We  create  a  StreamingContext  that  wraps  a  SparkContext  (there  are  alterna?ve  ways  to  construct  it...).  It  will  “clump”  the  events  into  1-­‐second   intervals.  Next  we  setup  a  socket  to  stream  text  to  us  from  another  server  and  port  (one  of  several  ways  to  ingest  data).
ssc.socketTextStream(server, port) // Word Count... val words = lines flatMap { line => line.split("""W+""") } val pairs = words map (word => (word, 1)) val wordCounts = pairs reduceByKey ((n1, n2) => n1 + n2) wordCount.print() // print a few counts... ssc.start() ssc.awaitTermination() Thursday, May 1, 14 Now  the  “word  count”  happens  over  each  interval  (aggrega?on  across  intervals  is  also  possible),  but  otherwise  it  works  like  before. Once  we  setup  the  flow,  we  start  it  and  wait  for  it  to  terminate  through  some  means,  such  as  the  server  socket  closing.
MLlib: Machine learning library. Cluster Computing Thursday, May 1, 14 Spark implements many machine learning algorithms, although a lot more or needed, compared to more mature tools like Mahout and libraries in Python and R.
•Linear regression •Binary classification •Collaborative filtering •Clustering •Others... MLlib Thursday, May 1, 14 Not as full-featured as more mature toolkits, but the Mahout project has announced they are going to port their algorithms to Spark, which include powerful Mathematics, e.g., Matrix support libraries.
GraphX: Graphical models and algorithms. Cluster Computing Thursday, May 1, 14 Some problems are more naturally represented as graphs. Extends RDDs to support property graphs with directed edges.
Thursday, May 1, 14 Why is Spark so good (and Java MapReduce so bad)? Because fundamentally, data analytics is Mathematics and programming tools inspired by Mathematics - like Functional Programming - are ideal tools for working with data. This is why Spark code is so concise, yet powerful. This is why it is a great platform for performance optimizations. This is why Spark is a great platform for higher-level tools, like SQL, graphs, etc. Interest in FP started growing ~10 years ago as a tool to attack concurrency. I believe that data is now driving FP adoption even faster. I know many Java shops that switched to Scala when they adopted tools like Spark and Scalding (https://github.com/twitter/scalding).
A flexible, scalable distributed compute platform with concise, powerful APIs and higher-order tools. spark.apache.org Spark Thursday, May 1, 14
Why Spark Is the Next Top (Compute) Model @deanwampler polyglotprogramming.com/talks Thursday, May 1, 14 Copyright (c) 2014, Dean Wampler, All Rights Reserved, unless otherwise noted. Image: The London Eye on one side of the Thames, Parliament on the other.

More Related Content

PDF
Why Spark Is the Next Top (Compute) Model
byDean Wampler
 
PDF
MapReduce and Its Discontents
byDean Wampler
 
PDF
Stream all the things
byDean Wampler
 
PPT
The State of Scala
byBoston Area Scala Enthusiasts
 
PDF
Reactive design: languages, and paradigms
byDean Wampler
 
PDF
Hadoop & MapReduce
byNewvewm
 
PPTX
Rapid Prototyping in PySpark Streaming: The Thermodynamics of Docker Containe...
byRichard Seymour
 
PDF
Hive sq lfor-hadoop
byPragati Singh
 
Why Spark Is the Next Top (Compute) Model
byDean Wampler
 
MapReduce and Its Discontents
byDean Wampler
 
Stream all the things
byDean Wampler
 
The State of Scala
byBoston Area Scala Enthusiasts
 
Reactive design: languages, and paradigms
byDean Wampler
 
Hadoop & MapReduce
byNewvewm
 
Rapid Prototyping in PySpark Streaming: The Thermodynamics of Docker Containe...
byRichard Seymour
 
Hive sq lfor-hadoop
byPragati Singh
 

What's hot

PDF
Spark and shark
byDataWorks Summit
 
PPT
Hadoop basics
byAntonio Silveira
 
PPTX
Think Like Spark: Some Spark Concepts and a Use Case
byRachel Warren
 
PDF
Running R on Hadoop - CHUG - 20120815
byChicago Hadoop Users Group
 
PDF
Hadoop pig
bySean Murphy
 
PDF
Practical Problem Solving with Apache Hadoop & Pig
byMilind Bhandarkar
 
PDF
Hadoop-Introduction
bySandeep Deshmukh
 
PPTX
Introduction to MapReduce and Hadoop
byMohamed Elsaka
 
ODP
Hadoop2.2
bySreejith P
 
PPTX
Spark and Shark: Lightning-Fast Analytics over Hadoop and Hive Data
byJetlore
 
KEY
Intro to Hadoop
byjeffturner
 
PDF
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)
byJeffrey Breen
 
PDF
Introduction to Hadoop and MapReduce
byDr Ganesh Iyer
 
PPTX
Frustration-Reduced PySpark: Data engineering with DataFrames
byIlya Ganelin
 
PPTX
Understanding Spark Tuning: Strata New York
byRachel Warren
 
PPTX
Spark autotuning talk final
byRachel Warren
 
PDF
Advanced spark training advanced spark internals and tuning reynold xin
bycaidezhi655
 
PPTX
Hadoop/MapReduce/HDFS
bypraveen bhat
 
PPT
Another Intro To Hadoop
byAdeel Ahmad
 
PPT
An Introduction to Hadoop
byDerrekYoungDotCom
 
Spark and shark
byDataWorks Summit
 
Hadoop basics
byAntonio Silveira
 
Think Like Spark: Some Spark Concepts and a Use Case
byRachel Warren
 
Running R on Hadoop - CHUG - 20120815
byChicago Hadoop Users Group
 
Hadoop pig
bySean Murphy
 
Practical Problem Solving with Apache Hadoop & Pig
byMilind Bhandarkar
 
Hadoop-Introduction
bySandeep Deshmukh
 
Introduction to MapReduce and Hadoop
byMohamed Elsaka
 
Hadoop2.2
bySreejith P
 
Spark and Shark: Lightning-Fast Analytics over Hadoop and Hive Data
byJetlore
 
Intro to Hadoop
byjeffturner
 
Big Data Step-by-Step: Using R & Hadoop (with RHadoop's rmr package)
byJeffrey Breen
 
Introduction to Hadoop and MapReduce
byDr Ganesh Iyer
 
Frustration-Reduced PySpark: Data engineering with DataFrames
byIlya Ganelin
 
Understanding Spark Tuning: Strata New York
byRachel Warren
 
Spark autotuning talk final
byRachel Warren
 
Advanced spark training advanced spark internals and tuning reynold xin
bycaidezhi655
 
Hadoop/MapReduce/HDFS
bypraveen bhat
 
Another Intro To Hadoop
byAdeel Ahmad
 
An Introduction to Hadoop
byDerrekYoungDotCom
 

Viewers also liked

PDF
Why Spark Is the Next Top (Compute) Model
byDean Wampler
 
PDF
Spark at Twitter - Seattle Spark Meetup, April 2014
bySriram Krishnan
 
PDF
Preso spark leadership
bysjoerdluteyn
 
PPTX
Spark - Philly JUG
byBrian O'Neill
 
PDF
Performance
byChristophe Marchal
 
PDF
Spark, the new age of data scientist
byMassimiliano Martella
 
PDF
Spark introduction - In Chinese
bycolorant
 
PDF
Apache Spark with Scala
byFernando Rodriguez
 
PPTX
Intro to Apache Spark
byclairvoyantllc
 
PPTX
The Future of Data Science
bysarith divakar
 
PDF
A Deeper Understanding of Spark Internals (Hadoop Conference Japan 2014)
byHadoop / Spark Conference Japan
 
PPTX
Pixie dust overview
byDavid Taieb
 
PDF
Why dont you_create_new_spark_jl
byShintaro Fukushima
 
PDF
Spark in 15 min
byChristophe Marchal
 
PDF
Spark tutorial py con 2016 part 2
byDavid Taieb
 
PPTX
How Spark is Enabling the New Wave of Converged Applications
byMapR Technologies
 
PDF
Tiny Batches, in the wine: Shiny New Bits in Spark Streaming
byPaco Nathan
 
PDF
Data Platform at Twitter: Enabling Real-time & Batch Analytics at Scale
bySriram Krishnan
 
PDF
Spark tutorial pycon 2016 part 1
byDavid Taieb
 
PDF
What’s New in Spark 2.0: Structured Streaming and Datasets - StampedeCon 2016
byStampedeCon
 
Why Spark Is the Next Top (Compute) Model
byDean Wampler
 
Spark at Twitter - Seattle Spark Meetup, April 2014
bySriram Krishnan
 
Preso spark leadership
bysjoerdluteyn
 
Spark - Philly JUG
byBrian O'Neill
 
Performance
byChristophe Marchal
 
Spark, the new age of data scientist
byMassimiliano Martella
 
Spark introduction - In Chinese
bycolorant
 
Apache Spark with Scala
byFernando Rodriguez
 
Intro to Apache Spark
byclairvoyantllc
 
The Future of Data Science
bysarith divakar
 
A Deeper Understanding of Spark Internals (Hadoop Conference Japan 2014)
byHadoop / Spark Conference Japan
 
Pixie dust overview
byDavid Taieb
 
Why dont you_create_new_spark_jl
byShintaro Fukushima
 
Spark in 15 min
byChristophe Marchal
 
Spark tutorial py con 2016 part 2
byDavid Taieb
 
How Spark is Enabling the New Wave of Converged Applications
byMapR Technologies
 
Tiny Batches, in the wine: Shiny New Bits in Spark Streaming
byPaco Nathan
 
Data Platform at Twitter: Enabling Real-time & Batch Analytics at Scale
bySriram Krishnan
 
Spark tutorial pycon 2016 part 1
byDavid Taieb
 
What’s New in Spark 2.0: Structured Streaming and Datasets - StampedeCon 2016
byStampedeCon
 

Similar to Spark the next top compute model

PDF
Introduction to Spark
byCarol McDonald
 
PDF
Apache Spark Overview
byCarol McDonald
 
PDF
Why Scala Is Taking Over the Big Data World
byDean Wampler
 
PPT
L19CloudMapReduce introduction for cloud computing .ppt
byMaruthiPrasad96
 
PPT
11. From Hadoop to Spark 1:2
byFabio Fumarola
 
PPT
Lecture 4 Parallel and Distributed Systems Fall 2024.ppt
byssusere82d541
 
PPTX
Learn what is Hadoop-and-BigData
byThanusha154
 
PPT
L4.FA16n nm,m,m,,m,m,m,mmnm,n,mnmnmm.ppt
byabdulbasetalselwi
 
PPT
Hadoop - Introduction to HDFS
byVibrant Technologies & Computers
 
PPTX
Introduction to Hadoop and MapReduce
byCsaba Toth
 
PPTX
Hadoop training-in-hyderabad
bysreehari orienit
 
PPT
Large scale computing
byBhupesh Bansal
 
PPT
hadoop
byswatic018
 
PPT
hadoop
byswatic018
 
PPTX
Mapreduce is for Hadoop Ecosystem in Data Science
byDakshGoti2
 
PPT
Hadoop classes in mumbai
byVibrant Technologies & Computers
 
PDF
Hadoop
bydevakalyan143
 
PPTX
Hadoop
byAnil Reddy
 
ODP
Hadoop - Overview
byJay
 
PPT
Hadoop
byGirish Khanzode
 
Introduction to Spark
byCarol McDonald
 
Apache Spark Overview
byCarol McDonald
 
Why Scala Is Taking Over the Big Data World
byDean Wampler
 
L19CloudMapReduce introduction for cloud computing .ppt
byMaruthiPrasad96
 
11. From Hadoop to Spark 1:2
byFabio Fumarola
 
Lecture 4 Parallel and Distributed Systems Fall 2024.ppt
byssusere82d541
 
Learn what is Hadoop-and-BigData
byThanusha154
 
L4.FA16n nm,m,m,,m,m,m,mmnm,n,mnmnmm.ppt
byabdulbasetalselwi
 
Hadoop - Introduction to HDFS
byVibrant Technologies & Computers
 
Introduction to Hadoop and MapReduce
byCsaba Toth
 
Hadoop training-in-hyderabad
bysreehari orienit
 
Large scale computing
byBhupesh Bansal
 
hadoop
byswatic018
 
hadoop
byswatic018
 
Mapreduce is for Hadoop Ecosystem in Data Science
byDakshGoti2
 
Hadoop classes in mumbai
byVibrant Technologies & Computers
 
Hadoop
bydevakalyan143
 
Hadoop
byAnil Reddy
 
Hadoop - Overview
byJay
 
Hadoop
byGirish Khanzode
 

Recently uploaded

PPTX
AI Agent Overview - Why we need AI Agent
byAlokGope
 
PDF
Design and Analysis of Algorithms(DAA): Unit-II Asymptotic Notations and Basi...
byKuvempu University
 
PDF
BCS-FACS Peter Landin Semantics Seminar - Formal Methods: Whence and Whither?
byJonathan Bowen
 
PPTX
SAP FICO Training Agenda for new learners
bysasidhar unnagiri
 
PPTX
Deep Dive into Durable Functions, presented at Cloudbrew 2025
byJoonas Westlin
 
PPTX
Why Your Business Needs Snowflake Consulting_ From Data Silos to AI-Ready Cloud
byChetu
 
PDF
Constraints First - Why Our On-Prem Ticketing System Starts With Limits, Not ...
bySpirit Face
 
PDF
Cloud-Based Underwriting Software for Insurance
byInsurance Tech Services
 
PDF
Virtual Study Circles Innovative Ways to Collaborate Online.pdf
byExplain Learning
 
PDF
What Is A Woman (WIAW) Token – Smart Contract Security Audit Report by EtherA...
byEtherAuthority
 
PDF
Resource-Levelled Critical-Path Analysis Balancing Time, Cost and Constraints
byOrangescrum
 
PDF
Cybersecurity Alert- What Organisations Must Watch Out For This Christmas Fes...
byCybernetic Global Intelligence
 
PDF
TNG_Architecting Green Software - An Introduction_Nils-Oliver Linden&Alexande...
byQAware GmbH
 
PDF
nsfconvertersoftwaretoconvertNSFtoPST.pdf
byNSF Converter Software
 
PPTX
Lecture 3 - Scheduling - Operating System
bynurulalomador
 
PPTX
GDS Integration Solution | GDS Integration Service
byyugababu033
 
PPTX
Modern P&C Insurance Software for Smarter, Faster Operations.pptx
byInsurance Tech Services
 
PPTX
Python-Functions-A-Comprehensive-Overview.pptx
byMuhammad Fahad Bashir
 
PPTX
SNG460-CNG489_Week6_3-7Nov25_Chp6-Part1.pptx
byberayseray382
 
PPTX
Week 7 Introduction to HTML PowerPoint Notes.pptx
bylesandawelgens
 
AI Agent Overview - Why we need AI Agent
byAlokGope
 
Design and Analysis of Algorithms(DAA): Unit-II Asymptotic Notations and Basi...
byKuvempu University
 
BCS-FACS Peter Landin Semantics Seminar - Formal Methods: Whence and Whither?
byJonathan Bowen
 
SAP FICO Training Agenda for new learners
bysasidhar unnagiri
 
Deep Dive into Durable Functions, presented at Cloudbrew 2025
byJoonas Westlin
 
Why Your Business Needs Snowflake Consulting_ From Data Silos to AI-Ready Cloud
byChetu
 
Constraints First - Why Our On-Prem Ticketing System Starts With Limits, Not ...
bySpirit Face
 
Cloud-Based Underwriting Software for Insurance
byInsurance Tech Services
 
Virtual Study Circles Innovative Ways to Collaborate Online.pdf
byExplain Learning
 
What Is A Woman (WIAW) Token – Smart Contract Security Audit Report by EtherA...
byEtherAuthority
 
Resource-Levelled Critical-Path Analysis Balancing Time, Cost and Constraints
byOrangescrum
 
Cybersecurity Alert- What Organisations Must Watch Out For This Christmas Fes...
byCybernetic Global Intelligence
 
TNG_Architecting Green Software - An Introduction_Nils-Oliver Linden&Alexande...
byQAware GmbH
 
nsfconvertersoftwaretoconvertNSFtoPST.pdf
byNSF Converter Software
 
Lecture 3 - Scheduling - Operating System
bynurulalomador
 
GDS Integration Solution | GDS Integration Service
byyugababu033
 
Modern P&C Insurance Software for Smarter, Faster Operations.pptx
byInsurance Tech Services
 
Python-Functions-A-Comprehensive-Overview.pptx
byMuhammad Fahad Bashir
 
SNG460-CNG489_Week6_3-7Nov25_Chp6-Part1.pptx
byberayseray382
 
Week 7 Introduction to HTML PowerPoint Notes.pptx
bylesandawelgens
 

Spark the next top compute model

  • 1.
    Why Spark Is theNext Top (Compute) Model Philly  ETE  2014 April  22-­‐23,  2014 @deanwampler polyglotprogramming.com/talks Thursday, May 1, 14 Copyright (c) 2014, Dean Wampler, All Rights Reserved, unless otherwise noted. Image: Detail of the London Eye
  • 2.
    Dean Wampler dean.wampler@typesafe.com polyglotprogramming.com/talks @deanwampler Thursday, May1, 14 About me. You can find this presentation and others on Big Data and Scala at polyglotprogramming.com. Programming Scala, 2nd Edition is forthcoming. photo: Dusk at 30,000 ft above the Central Plains of the U.S. on a Winter’s Day.
  • 3.
    Or this? Thursday, May 1,14 photo: https://twitter.com/john_overholt/status/447431985750106112/photo/1
  • 4.
    Hadoop circa 2013 Thursday, May1, 14 The state of Hadoop as of last year. Image: Detail of the London Eye
  • 5.
    Hadoop v2.X Cluster node DiskDiskDiskDiskDisk NodeMgr Data Node node DiskDiskDiskDiskDisk Node Mgr Data Node node DiskDiskDiskDiskDisk Node Mgr Data Node master Resource Mgr Name Node master Name Node Thursday, May 1, 14 Schematic view of a Hadoop 2 cluster. For a more precise definition of the services and what they do, see e.g., http://hadoop.apache.org/ docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/YARN.html We aren’t interested in great details at this point, but we’ll call out a few useful things to know.
  • 6.
    Hadoop v2.X Cluster node DiskDiskDiskDiskDisk NodeMgr Data Node node DiskDiskDiskDiskDisk Node Mgr Data Node node DiskDiskDiskDiskDisk Node Mgr Data Node master Resource Mgr Name Node master Name Node Resource and Node Managers Thursday, May 1, 14 Hadoop 2 uses YARN to manage resources via the master Resource Manager, which includes a pluggable job scheduler and an Applications Manager. It coordinates with the Node Manager on each node to schedule jobs and provide resources. Other services involved, including application-specific Containers and Application Masters are not shown.
  • 7.
    Hadoop v2.X Cluster node DiskDiskDiskDiskDisk NodeMgr Data Node node DiskDiskDiskDiskDisk Node Mgr Data Node node DiskDiskDiskDiskDisk Node Mgr Data Node master Resource Mgr Name Node master Name Node Name Node and Data Nodes Thursday, May 1, 14 Hadoop 2 clusters federate the Name node services that manage the file system, HDFS. They provide horizontal scalability of file-system operations and resiliency when service instances fail. The data node services manage individual blocks for files.
  • 8.
    MapReduce The classic computemodel for Hadoop Thursday, May 1, 14 Hadoop 2 clusters federate the Name node services that manage the file system, HDFS. They provide horizontal scalability of file-system operations and resiliency when service instances fail. The data node services manage individual blocks for files.
  • 9.
    1 map step+ 1 reduce step (wash, rinse, repeat) MapReduce Thursday, May 1, 14 You get 1 map step (although there is limited support for chaining mappers) and 1 reduce step. If you can’t implement an algorithm in these two steps, you can chain jobs together, but you’ll pay a tax of flushing the entire data set to disk between these jobs.
  • 10.
    Example: Inverted Index MapReduce Thursday, May1, 14 The inverted index is a classic algorithm needed for building search engines.
  • 11.
    Thursday, May 1,14 Before running MapReduce, crawl teh interwebs, find all the pages, and build a data set of URLs -> doc contents, written to flat files in HDFS or one of the more “sophisticated” formats.
  • 12.
    Thursday, May 1,14 Now we’re running MapReduce. In the map step, a task (JVM process) per file *block* (64MB or larger) reads the rows, tokenizes the text and outputs key-value pairs (“tuples”)...
  • 13.
    Map Task (hadoop,(wikipedia.org/hadoop,1)) (mapreduce,(wikipedia.org/hadoop, 1)) (hdfs,(wikipedia.org/hadoop,1)) (provides,(wikipedia.org/hadoop,1)) (and,(wikipedia.org/hadoop,1)) Thursday, May 1, 14 ... the keys are each word found and the values are themselves tuples, each URL and the count of the word. In our simplified example, there are typically only single occurrences of each work in each document. The real occurrences are interesting because if a word is mentioned a lot in a document, the chances are higher that you would want to find that document in a search for that word.
  • 14.
  • 15.
    Thursday, May 1,14 The output tuples are sorted by key locally in each map task, then “shuffled” over the cluster network to reduce tasks (each a JVM process, too), where we want all occurrences of a given key to land on the same reduce task.
  • 16.
    Thursday, May 1,14 Finally, each reducer just aggregates all the values it receives for each key, then writes out new files to HDFS with the words and a list of (URL-count) tuples (pairs).
  • 17.
    Altogether... Thursday, May 1,14 Finally, each reducer just aggregates all the values it receives for each key, then writes out new files to HDFS with the words and a list of (URL-count) tuples (pairs).
  • 18.
    What’s not to like?Thursday,May 1, 14 This seems okay, right? What’s wrong with it?
  • 19.
    Most algorithms are muchharder to implement in this restrictive map-then-reduce model. Awkward Thursday, May 1, 14 Writing MapReduce jobs requires arcane, specialized skills that few master. For a good overview, see http://lintool.github.io/ MapReduceAlgorithms/.
  • 20.
    Lack of flexibilityinhibits optimizations, too. Awkward Thursday, May 1, 14 The inflexible compute model leads to complex code to improve performance where hacks are used to work around the limitations. Hence, optimizations are hard to implement. The Spark team has commented on this, see http://databricks.com/blog/2014/03/26/Spark-SQL- manipulating-structured-data-using-Spark.html
  • 21.
    Full dump todisk between jobs. Performance Thursday, May 1, 14 Sequencing jobs wouldn’t be so bad if the “system” was smart enough to cache data in memory. Instead, each job dumps everything to disk, then the next job reads it back in again. This makes iterative algorithms particularly painful.
  • 22.
  • 23.
    Can be runin: •YARN (Hadoop 2) •Mesos (Cluster management) •EC2 •Standalone mode Cluster Computing Thursday, May 1, 14 If you have a Hadoop cluster, you can run Spark as a seamless compute engine on YARN. (You can also use pre-YARN Hadoop v1 clusters, but there you have manually allocate resources to the embedded Spark cluster vs Hadoop.) Mesos is a general-purpose cluster resource manager that can also be used to manage Hadoop resources. Scripts for running a Spark cluster in EC2 are available. Standalone just means you run Spark’s built-in support for clustering (or run locally on a single box - e.g., for development). EC2 deployments are usually standalone...
  • 24.
    Fine-grained “combinators” for composingalgorithms. Compute Model Thursday, May 1, 14 Once you learn the core set of primitives, it’s easy to compose non-trivial algorithms with little code.
  • 25.
    RDDs: Resilient, Distributed Datasets Compute Model Thursday, May1, 14 RDDs shard the data over a cluster, like a virtualized, distributed collection (analogous to HDFS). They support intelligent caching, which means no naive flushes of massive datasets to disk. This feature alone allows Spark jobs to run 10-100x faster than comparable MapReduce jobs! The “resilient” part means they will reconstitute shards lost due to process/server crashes.
  • 26.
    Written in Scala, withJava and Python APIs. Compute Model Thursday, May 1, 14 Once you learn the core set of primitives, it’s easy to compose non-trivial algorithms with little code.
  • 27.
    Inverted Index in MapReduce (Java). Thursday,May 1, 14 Let’s  see  an  an  actual  implementaEon  of  the  inverted  index.  First,  a  Hadoop  MapReduce  (Java)  version,  adapted  from  hPps:// developer.yahoo.com/hadoop/tutorial/module4.html#soluEon  It’s  about  90  lines  of  code,  but  I  reformaPed  to  fit  bePer. This  is  also  a  slightly  simpler  version  that  the  one  I  diagrammed.  It  doesn’t  record  a  count  of  each  word  in  a  document;  it  just  writes   (word,doc-­‐Etle)  pairs  out  of  the  mappers  and  the  final  (word,list)  output  by  the  reducers  just  has  a  list  of  documentaEons,  hence  repeats.   A  second  job  would  be  necessary  to  count  the  repeats.
  • 28.
    import java.io.IOException; import java.util.*; importorg.apache.hadoop.fs.Path; import org.apache.hadoop.io.*; import org.apache.hadoop.mapred.*; public class LineIndexer { public static void main(String[] args) { JobClient client = new JobClient(); JobConf conf = new JobConf(LineIndexer.class); conf.setJobName("LineIndexer"); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(Text.class); FileInputFormat.addInputPath(conf, new Path("input")); Thursday, May 1, 14 I’ve  shortened  the  original  code  a  bit,  e.g.,  using  *  import  statements  instead  of  separate  imports  for  each  class. I’m  not  going  to  explain  every  line  ...  nor  most  lines. Everything  is  in  one  outer  class.  We  start  with  a  main  rou?ne  that  sets  up  the  job.  LoBa  boilerplate... I  used  yellow  for  method  calls,  because  methods  do  the  real  work!!  But  no?ce  that  the  func?ons  in  this  code  don’t  really  do  a  whole  lot...
  • 29.
    JobClient client =new JobClient(); JobConf conf = new JobConf(LineIndexer.class); conf.setJobName("LineIndexer"); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(Text.class); FileInputFormat.addInputPath(conf, new Path("input")); FileOutputFormat.setOutputPath(conf, new Path("output")); conf.setMapperClass( LineIndexMapper.class); conf.setReducerClass( LineIndexReducer.class); client.setConf(conf); try { JobClient.runJob(conf); Thursday, May 1, 14 boilerplate...
  • 30.
    LineIndexMapper.class); conf.setReducerClass( LineIndexReducer.class); client.setConf(conf); try { JobClient.runJob(conf); } catch(Exception e) { e.printStackTrace(); } } public static class LineIndexMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text> { private final static Text word = new Text(); private final static Text location = Thursday, May 1, 14 main ends with a try-catch clause to run the job.
  • 31.
    public static classLineIndexMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text> { private final static Text word = new Text(); private final static Text location = new Text(); public void map( LongWritable key, Text val, OutputCollector<Text, Text> output, Reporter reporter) throws IOException { FileSplit fileSplit = (FileSplit)reporter.getInputSplit(); String fileName = fileSplit.getPath().getName(); location.set(fileName); Thursday, May 1, 14 This is the LineIndexMapper class for the mapper. The map method does the real work of tokenization and writing the (word, document-name) tuples.
  • 32.
    FileSplit fileSplit = (FileSplit)reporter.getInputSplit(); StringfileName = fileSplit.getPath().getName(); location.set(fileName); String line = val.toString(); StringTokenizer itr = new StringTokenizer(line.toLowerCase()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); output.collect(word, location); } } } public static class LineIndexReducer extends MapReduceBase implements Reducer<Text, Text, Thursday, May 1, 14 The rest of the LineIndexMapper class and map method.
  • 33.
    public static classLineIndexReducer extends MapReduceBase implements Reducer<Text, Text, Text, Text> { public void reduce(Text key, Iterator<Text> values, OutputCollector<Text, Text> output, Reporter reporter) throws IOException { boolean first = true; StringBuilder toReturn = new StringBuilder(); while (values.hasNext()) { if (!first) toReturn.append(", "); first=false; toReturn.append( values.next().toString()); } output.collect(key, new Text(toReturn.toString())); Thursday, May 1, 14 The reducer class, LineIndexReducer, with the reduce method that is called for each key and a list of values for that key. The reducer is stupid; it just reformats the values collection into a long string and writes the final (word,list-string) output.
  • 34.
    boolean first =true; StringBuilder toReturn = new StringBuilder(); while (values.hasNext()) { if (!first) toReturn.append(", "); first=false; toReturn.append( values.next().toString()); } output.collect(key, new Text(toReturn.toString())); } } } Thursday, May 1, 14 EOF
  • 35.
    Altogether import java.io.IOException; import java.util.*; importorg.apache.hadoop.fs.Path; import org.apache.hadoop.io.*; import org.apache.hadoop.mapred.*; public class LineIndexer { public static void main(String[] args) { JobClient client = new JobClient(); JobConf conf = new JobConf(LineIndexer.class); conf.setJobName("LineIndexer"); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(Text.class); FileInputFormat.addInputPath(conf, new Path("input")); FileOutputFormat.setOutputPath(conf, new Path("output")); conf.setMapperClass( LineIndexMapper.class); conf.setReducerClass( LineIndexReducer.class); client.setConf(conf); try { JobClient.runJob(conf); } catch (Exception e) { e.printStackTrace(); } } public static class LineIndexMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text> { private final static Text word = new Text(); private final static Text location = new Text(); public void map( LongWritable key, Text val, OutputCollector<Text, Text> output, Reporter reporter) throws IOException { FileSplit fileSplit = (FileSplit)reporter.getInputSplit(); String fileName = fileSplit.getPath().getName(); location.set(fileName); String line = val.toString(); StringTokenizer itr = new StringTokenizer(line.toLowerCase()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); output.collect(word, location); } } } public static class LineIndexReducer extends MapReduceBase implements Reducer<Text, Text, Text, Text> { public void reduce(Text key, Iterator<Text> values, OutputCollector<Text, Text> output, Reporter reporter) throws IOException { boolean first = true; StringBuilder toReturn = new StringBuilder(); while (values.hasNext()) { if (!first) toReturn.append(", "); first=false; toReturn.append( values.next().toString()); } output.collect(key, new Text(toReturn.toString())); } } } Thursday, May 1, 14 The  whole  shebang  (6pt.  font)
  • 36.
    Inverted Index in Spark (Scala). Thursday,May 1, 14 This  code  is  approximately  45  lines,  but  it  does  more  than  the  previous  Java  example,  it  implements  the  original  inverted  index  algorithm  I   diagrammed  where  word  counts  are  computed  and  included  in  the  data.
  • 37.
    import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ objectInvertedIndex { def main(args: Array[String]) = { val sc = new SparkContext( "local", "Inverted Index") sc.textFile("data/crawl") .map { line => val array = line.split("t", 2) (array(0), array(1)) } .flatMap { case (path, text) => text.split("""W+""") map { word => (word, path) } } Thursday, May 1, 14 It  starts  with  imports,  then  declares  a  singleton  object  (a  first-­‐class  concept  in  Scala),  with  a  main  rou?ne  (as  in  Java). The  methods  are  colored  yellow  again.  Note  this  ?me  how  dense  with  meaning  they  are  this  ?me.
  • 38.
    val sc =new SparkContext( "local", "Inverted Index") sc.textFile("data/crawl") .map { line => val array = line.split("t", 2) (array(0), array(1)) } .flatMap { case (path, text) => text.split("""W+""") map { word => (word, path) } } .map { case (w, p) => ((w, p), 1) } .reduceByKey { case (n1, n2) => n1 + n2 } Thursday, May 1, 14 You  being  the  workflow  by  declaring  a  SparkContext  (in  “local”  mode,  in  this  case).  The  rest  of  the  program  is  a  sequence  of  func?on  calls,   analogous  to  “pipes”  we  connect  together  to  perform  the  data  flow. Next  we  read  one  or  more  text  files.  If  “data/crawl”  has  1  or  more  Hadoop-­‐style  “part-­‐NNNNN”  files,  Spark  will  process  all  of  them  (in  parallel  if   running  a  distributed  configura?on;  they  will  be  processed  synchronously  in  local  mode).
  • 39.
    .map { line=> val array = line.split("t", 2) (array(0), array(1)) } .flatMap { case (path, text) => text.split("""W+""") map { word => (word, path) } } .map { case (w, p) => ((w, p), 1) } .reduceByKey { case (n1, n2) => n1 + n2 } .map { case ((w, p), n) => (w, (p, n)) } .groupBy { Thursday, May 1, 14 sc.textFile  returns  an  RDD  with  a  string  for  each  line  of  input  text.  So,  the  first  thing  we  do  is  map  over  these  strings  to  extract  the  original   document  id  (i.e.,  file  name),  followed  by  the  text  in  the  document,  all  on  one  line.  We  assume  tab  is  the  separator.  “(array(0),  array(1))”  returns  a   two-­‐element  “tuple”.  Think  of  the  output  RDD  has  having  a  schema  “String  fileName,  String  text”.    
  • 40.
    } .flatMap { case (path,text) => text.split("""W+""") map { word => (word, path) } } .map { case (w, p) => ((w, p), 1) } .reduceByKey { case (n1, n2) => n1 + n2 } .map { case ((w, p), n) => (w, (p, n)) } .groupBy { case (w, (p, n)) => w } .map { Beautiful, no? Thursday, May 1, 14 flatMap  maps  over  each  of  these  2-­‐element  tuples.  We  split  the  text  into  words  on  non-­‐alphanumeric  characters,  then  output  collec?ons  of  word   (our  ul?mate,  final  “key”)  and  the  path.  Each  line  is  converted  to  a  collec?on  of  (word,path)  pairs,  so  flatMap  converts  the  collec?on  of  collec?ons   into  one  long  “flat”  collec?on  of  (word,path)  pairs. Then  we  map  over  these  pairs  and  add  a  single  count  of  1.
  • 41.
    Thursday, May 1,14 Another  example  of  a  beau?ful  and  profound  DSL,  in  this  case  from  the  world  of  Physics:  Maxwell’s  equa?ons:  hBp://upload.wikimedia.org/ wikipedia/commons/c/c4/Maxwell'sEqua?ons.svg
  • 42.
    } .reduceByKey { case (n1,n2) => n1 + n2 } .map { case ((w, p), n) => (w, (p, n)) } .groupBy { case (w, (p, n)) => w } .map { case (w, seq) => val seq2 = seq map { case (_, (p, n)) => (p, n) } (w, seq2.mkString(", ")) } .saveAsTextFile(argz.outpath) sc.stop() (word1,  (path1,  n1)) (word2,  (path2,  n2)) ... Thursday, May 1, 14 reduceByKey  does  an  implicit  “group  by”  to  bring  together  all  occurrences  of  the  same  (word,  path)  and  then  sums  up  their  counts. Note  the  input  to  the  next  map  is  now  ((word,  path),  n),  where  n  is  now  >=  1.  We  transform  these  tuples  into  the  form  we  actually  want,  (word,   (path,  n)).
  • 43.
    } .groupBy { case (w,(p, n)) => w } .map { case (w, seq) => val seq2 = seq map { case (_, (p, n)) => (p, n) } (w, seq2.mkString(", ")) } .saveAsTextFile(argz.outpath) sc.stop() } } (word,  seq((word,  (path1,  n1)),  (word,  (path2,  n2)),  ...)) (word,  “(path1,  n1),  (path2,  n2),  ...”) Thursday, May 1, 14 Now  we  do  an  explicit  group  by  to  bring  all  the  same  words  together.  The  output    will  be  (word,  (word,  (path1,  n1)),  (word,  (path2,  n2)),  ...). The  last  map  removes  the  redundant  “word”  values  in  the  sequences  of  the  previous  output.  It  outputs  the  sequence  as  a  final  string  of  comma-­‐ separated  (path,n)  pairs. We  finish  by  saving  the  output  as  text  file(s)  and  stopping  the  workflow.  
  • 44.
    import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ objectInvertedIndex { def main(args: Array[String]) = { val sc = new SparkContext( "local", "Inverted Index") sc.textFile("data/crawl") .map { line => val array = line.split("t", 2) (array(0), array(1)) } .flatMap { case (path, text) => text.split("""W+""") map { word => (word, path) } } .map { case (w, p) => ((w, p), 1) } .reduceByKey { case (n1, n2) => n1 + n2 } .map { case ((w, p), n) => (w, (p, n)) } .groupBy { case (w, (p, n)) => w } .map { case (w, seq) => val seq2 = seq map { case (_, (p, n)) => (p, n) } (w, seq2.mkString(", ")) } .saveAsTextFile(argz.outpath) sc.stop() } } Altogether Thursday, May 1, 14 The  whole  shebang  (12pt.  font,  this  ?me)
  • 45.
    sc.textFile("data/crawl") .map { line=> val array = line.split("t", 2) (array(0), array(1)) } .flatMap { case (path, text) => text.split("""W+""") map { word => (word, path) } } .map { case (w, p) => ((w, p), 1) } .reduceByKey{ case (n1, n2) => n1 + n2 } .map { case ((w, p), n) => (w, (p, n)) } Powerful Combinators! Thursday, May 1, 14 I’ve  shortened  the  original  code  a  bit,  e.g.,  using  *  import  statements  instead  of  separate  imports  for  each  class. I’m  not  going  to  explain  every  line  ...  nor  most  lines. Everything  is  in  one  outer  class.  We  start  with  a  main  rou?ne  that  sets  up  the  job.  LoBa  boilerplate...
  • 46.
    The Spark version tookme ~30 minutes to write. Thursday, May 1, 14 Once  you  learn  the  core  primiEves  I  used,  and  a  few  tricks  for  manipulaEng  the  RDD  tuples,  you  can  very  quickly  build  complex  algorithms   for  data  processing! The  Spark  API  allowed  us  to  focus  almost  exclusively  on  the  “domain”  of  data  transformaEons,  while  the  Java  MapReduce  version  (which   does  less),  forced  tedious  aPenEon  to  infrastructure  mechanics.  
  • 47.
    Shark: Hive (SQL querytool) ported to Spark. SQL! Thursday, May 1, 14 Use SQL when you can!
  • 48.
    Use a SQLquery when you can!! Thursday, May 1, 14
  • 49.
    CREATE EXTERNAL TABLEstocks ( symbol STRING, ymd STRING, price_open STRING, price_close STRING, shares_traded INT) LOCATION “hdfs://data/stocks”; -- Year-over-year average closing price. SELECT year(ymd), avg(price_close) FROM stocks WHERE symbol = ‘AAPL’ GROUP BY year(ymd); Hive  Query   Language. Thursday, May 1, 14 Hive  and  the  Spark  port,  Shark,  let  you  use  SQL  to  query  and  manipulate  structured  and  semistructured  data.  By  default,  tab-­‐delimited  text  files   will  be  assumed  for  the  files  found  in  “data/stocks”.  Alterna?ves  can  be  configured  as  part  of  the  table  metadata.
  • 50.
    ~10-100x the performanceof Hive, due to in-memory caching of RDDs & better Spark abstractions. Shark Thursday, May 1, 14
  • 51.
    Spark SQL: Next generation SQLquery tool/API. Did I mention SQL? Thursday, May 1, 14 A new query optimizer. It will become the basis for Shark internals, replacing messing Hive code that is hard to reason about, extend, and debug.
  • 52.
    Combine Shark SQL querieswith Machine Learning code. Thursday, May 1, 14 We’ll  use  the  Spark  “MLlib”  in  the  example,  then  return  to  it  in  a  moment.
  • 53.
    CREATE TABLE Users( userIdINT, name STRING, email STRING, age INT, latitude DOUBLE, longitude DOUBLE, subscribed BOOLEAN); CREATE TABLE Events( userId INT, action INT); Hive/Shark  table   definiEons (not  Scala). Thursday, May 1, 14 This  example  adapted  from  the  following  blog  post  announcing  Spark  SQL: hBp://databricks.com/blog/2014/03/26/Spark-­‐SQL-­‐manipula?ng-­‐structured-­‐data-­‐using-­‐Spark.html Assume  we  have  these  Hive/Shark  tables,  with  data  about  our  users  and  events  that  have  occurred.
  • 54.
    val trainingDataTable =sql(""" SELECT e.action, u.age, u.latitude, u.longitude FROM Users u JOIN Events e ON u.userId = e.userId""") val trainingData = trainingDataTable map { row => val features = Array[Double](row(1), row(2), row(3)) LabeledPoint(row(0), features) } val model = new LogisticRegressionWithSGD() .run(trainingData) val allCandidates = sql(""" Spark  Scala. Thursday, May 1, 14 Here  is  some  Spark  (Scala)  code  with  an  embedded  SQL/Shark  query  that  joins  the  Users  and  Events  tables.  The  “””...”””  string  allows  embedded   line  feeds. The  “sql”  func?on  returns  an  RDD,  which  we  then  map  over  to  create  LabeledPoints,  an  object  used  in  Spark’s  MLlib  (machine  learning  library)  for   a  recommenda?on  engine.  The  “label”  is  the  kind  of  event  and  the  user’s  age  and  lat/long  coordinates  are  the  “features”  used  for  making   recommenda?ons.  (E.g.,  if  you’re  25  and  near  a  certain  loca?on  in  the  city,  you  might  be  interested  a  nightclub  near  by...)
  • 55.
    val model = newLogisticRegressionWithSGD() .run(trainingData) val allCandidates = sql(""" SELECT userId, age, latitude, longitude FROM Users WHERE subscribed = FALSE""") case class Score( userId: Int, score: Double) val scores = allCandidates map { row => val features = Array[Double](row(1), row(2), row(3)) Score(row(0), model.predict(features)) } // Hive table scores.registerAsTable("Scores") Thursday, May 1, 14 Next  we  train  the  recommenda?on  engine,  using  a  “logis?c  regression”  fit  to  the  training  data,  where  “stochas?c  gradient  descent”  (SGD)  is  used   to  train  it.  (This  is  a  standard  tool  set  for  recommenda?on  engines;  see  for  example:  hBp://www.cs.cmu.edu/~wcohen/10-­‐605/assignments/ sgd.pdf)
  • 56.
    .run(trainingData) val allCandidates =sql(""" SELECT userId, age, latitude, longitude FROM Users WHERE subscribed = FALSE""") case class Score( userId: Int, score: Double) val scores = allCandidates map { row => val features = Array[Double](row(1), row(2), row(3)) Score(row(0), model.predict(features)) } // Hive table scores.registerAsTable("Scores") val topCandidates = sql(""" SELECT u.name, u.email FROM Scores s Thursday, May 1, 14 Now  run  a  query  against  all  users  who  aren’t  already  subscribed  to  no?fica?ons.
  • 57.
    case class Score( userId:Int, score: Double) val scores = allCandidates map { row => val features = Array[Double](row(1), row(2), row(3)) Score(row(0), model.predict(features)) } // Hive table scores.registerAsTable("Scores") val topCandidates = sql(""" SELECT u.name, u.email FROM Scores s JOIN Users u ON s.userId = u.userId ORDER BY score DESC LIMIT 100""")Thursday, May 1, 14 Declare  a  class  to  hold  each  user’s  “score”  as  produced  by  the  recommenda?on  engine  and  map  the  “all”  query  results  to  Scores. Then  “register”  the  scores  RDD  as  a  “Scores”  table  in  Hive’s  metadata  respository.  This  is  equivalent  to  running  a  “CREATE  TABLE  Scores  ...”   command  at  the  Hive/Shark  prompt!
  • 58.
    // Hive table scores.registerAsTable("Scores") valtopCandidates = sql(""" SELECT u.name, u.email FROM Scores s JOIN Users u ON s.userId = u.userId ORDER BY score DESC LIMIT 100""") Thursday, May 1, 14 Finally,  run  a  new  query  to  find  the  people  with  the  highest  scores  that  aren’t  already  subscribing  to  no?fica?ons.  You  might  send  them  an  email   next  recommending  that  they  subscribe...
  • 59.
    Spark Streaming: Use thesame abstractions for real-time, event streaming. Cluster Computing Thursday, May 1, 14 Once you learn the core set of primitives, it’s easy to compose non-trivial algorithms with little code.
  • 60.
    Spark Job Iteration #1 1second Data Stream… Spark Job Iteration #2 Thursday, May 1, 14 You can specify the granularity, such as all events in 1 second windows, then your Spark job is patched each window of data for processing.
  • 61.
  • 62.
    val sc =new SparkContext(...) val ssc = new StreamingContext( sc, Seconds(1)) // A DStream that will listen to server:port val lines = ssc.socketTextStream(server, port) // Word Count... val words = lines flatMap { line => line.split("""W+""") } val pairs = words map (word => (word, 1)) val wordCounts = pairs reduceByKey ((n1, n2) => n1 + n2) wordCount.print() // print a few counts... Thursday, May 1, 14 This  example  adapted  from  the  following  page  on  the  Spark  website: hBp://spark.apache.org/docs/0.9.0/streaming-­‐programming-­‐guide.html#a-­‐quick-­‐example We  create  a  StreamingContext  that  wraps  a  SparkContext  (there  are  alterna?ve  ways  to  construct  it...).  It  will  “clump”  the  events  into  1-­‐second   intervals.  Next  we  setup  a  socket  to  stream  text  to  us  from  another  server  and  port  (one  of  several  ways  to  ingest  data).
  • 63.
    ssc.socketTextStream(server, port) // WordCount... val words = lines flatMap { line => line.split("""W+""") } val pairs = words map (word => (word, 1)) val wordCounts = pairs reduceByKey ((n1, n2) => n1 + n2) wordCount.print() // print a few counts... ssc.start() ssc.awaitTermination() Thursday, May 1, 14 Now  the  “word  count”  happens  over  each  interval  (aggrega?on  across  intervals  is  also  possible),  but  otherwise  it  works  like  before. Once  we  setup  the  flow,  we  start  it  and  wait  for  it  to  terminate  through  some  means,  such  as  the  server  socket  closing.
  • 64.
    MLlib: Machine learning library. ClusterComputing Thursday, May 1, 14 Spark implements many machine learning algorithms, although a lot more or needed, compared to more mature tools like Mahout and libraries in Python and R.
  • 65.
    •Linear regression •Binary classification •Collaborativefiltering •Clustering •Others... MLlib Thursday, May 1, 14 Not as full-featured as more mature toolkits, but the Mahout project has announced they are going to port their algorithms to Spark, which include powerful Mathematics, e.g., Matrix support libraries.
  • 66.
    GraphX: Graphical models and algorithms. ClusterComputing Thursday, May 1, 14 Some problems are more naturally represented as graphs. Extends RDDs to support property graphs with directed edges.
  • 67.
    Thursday, May 1,14 Why is Spark so good (and Java MapReduce so bad)? Because fundamentally, data analytics is Mathematics and programming tools inspired by Mathematics - like Functional Programming - are ideal tools for working with data. This is why Spark code is so concise, yet powerful. This is why it is a great platform for performance optimizations. This is why Spark is a great platform for higher-level tools, like SQL, graphs, etc. Interest in FP started growing ~10 years ago as a tool to attack concurrency. I believe that data is now driving FP adoption even faster. I know many Java shops that switched to Scala when they adopted tools like Spark and Scalding (https://github.com/twitter/scalding).
  • 68.
    A flexible, scalabledistributed compute platform with concise, powerful APIs and higher-order tools. spark.apache.org Spark Thursday, May 1, 14
  • 69.
    Why Spark Is theNext Top (Compute) Model @deanwampler polyglotprogramming.com/talks Thursday, May 1, 14 Copyright (c) 2014, Dean Wampler, All Rights Reserved, unless otherwise noted. Image: The London Eye on one side of the Thames, Parliament on the other.