1. to
From
Nathan Halko @nhalko
Data Scientist Hadoop day @ Data Summit
NYC May 2015

2. to
From
Trusty old friend
but
slobbers over
everything
Cute and shiny
but
would sell you out for
another bottle of wine

3. At
We turn social noise into
customer revenue
Ingest
Process
Deliver

4. From Hadoop MR job to Spark clone of job
…we can do better
Joins
Caching
Streaming
Counters
Laziness

5. class MyMapper {
override def
map(key, value, context) { ... }
}
class MyReducer {
override def
reduce(key, values, context) { ... }
}
class MyJob {
// bunch of setup
}
object MyJob {
def main(args) { //run the job }
}
def spark {
sc.textFile("input").map {
case (key, value) =>
/ / …
}
.groupByKey()
.map {
case (key, values) =>
/ / …
}
}
Serializability

6. Joins
Map Map
Table 2Table 1
Shuffle Process
Cassandra
Map
Shuffle
Reduce
Process
HDFS
Data 1 Data 2

7. Ca$$hing
Map
“data3”
Transform
Map
res1
Cassandra
HDFS
Map
res2
Broadcast variables
Named RDDs

8. Streaming
bin/hadoop jar hadoop-streaming.jar
-input myInputDirs
-output myOutputDir
-mapper /bin/cat
-reducer /bin/wc
On the fly job creation
Kafka queue
head —
offset —
Listening, always on processing

9. Counters

10. Laziness
Map
Reduce
Reduce
Reduce
Map
Map
Map
Map

11. 1. Set number of cores and heap per job
2. Internal map tasks reuse JVM
3. RDDs feel like Scala collections
4. I can read the source!! (Storm)
5. GraphX
1. Static cluster wide settings
2. JVM startup time costly
3. Reducer values iterator… yuck
4. Time for a beer, or six
5. Giraph
Poor setup docs, mostly high level dev
Serializability, runtime
We at least know how to do this
Most any code can fit in a mapper

12. Thank you!
spotright.com
nathan@spotright.com
Nathan Halko @nhalko
Data Scientist