The document discusses the evolution of in-memory computing with a focus on Tachyon and Apache Spark, highlighting key innovations such as resilient distributed datasets (RDDs) that enhance data processing. It compares Hadoop's traditional MapReduce framework with Spark's capabilities and examines the role of Tachyon as an in-memory data-exchange layer, suggesting it is beneficial for cloud environments. Overall, it emphasizes the shift toward faster, iterative processing in big data applications.

1. Tachyon and Apache Spark:
heralds of in-memory computing era.
Roman Shaposhnik
Director of Open Source @Pivotal
(Twitter: @rhatr)

2. Who’s this guy?
• Director of Open Source @Pivotal
• Apache Software Foundation guy (Member, VP of Apache
Incubator, committer on Hadoop, Giraph, Sqoop, etc)
• Used to be root@Cloudera
• Used to be PHB@Yahoo! (original Hadoop team)

3. Dearly beloved…

4. 20 minute to figure out
Hadoop vs. Spark

5. 20 minute to figure out
Hadoop++ == Spark

6. 20 minute to figure out
Hadoop + Spark

7. But wait! There’s more!
Tachyon

8. Long, long time ago…
HDFS
ASF Projects
FLOSS Projects
Pivotal Products
MapReduce

9. In a blink of an eye
MLib
Shark
GraphX
Streaming
HDFS
Crunch Mahout
Pig
Sqoop Flume
Coordination and
workflow
management
Zookeeper
Command
Center
ASF Projects
FLOSS Projects
Pivotal Products
GemFire XD
Oozie
MapReduce
Hive
Tez
Giraph
Hadoop UI
Hue
SolrCloud
Phoenix
HBase
Spark
Impala
HAWQ
SpringXD
MADlib
Hamster
PivotalR
YARN
Tachyon

10. A Spark view?
HDFS
MLib
Shark
YARN
GraphX
Streaming
Tachyon
Sqoop Flume
Hadoop UI
Hue
Coordination and
workflow
management
Zookeeper
Command
Center
ASF Projects
FLOSS Projects
Pivotal Products
GemFire XD
Oozie
SolrCloud
Phoenix
HBase Spark
SpringXD

11. BDAS

12. Long, long time ago…

13. This is 2014

14. What changed?

15. Your datacenter
…
server 1
server N

16. Hadoop’s view
MapReduce
server 1
server N
HDFS

17. HDFS: decoupled storage
…
MR
HDFS
MR

19. Anatomy of MapReduce
HDFS mappers reducers HDFS
a b c
d a c
a 3
b 1
c 2
a 1
b 1
c 1
a 1
c 1
a 1
a 1 1 1
b 1
c 1 1

20. What’s wrong with MR?
Source: UC Berkeley Spark project (just the image)

21. This looks familiar…
$ grep –R | awk | sort …

22. Spark innovations
• Resilient Distribtued Datasets (RDDs)
• Distributed on a cluster
• Manipulated via parallel operators (map, etc.)
• Automatically rebuilt on failure
• A parallel ecosystem
• A solution to iterative and multi-stage apps

23. RDDs
warnings = textFile(…).filter(_.contains(“warning”))
.map(_.split(‘ ‘)(1))
HadoopRDD
path = hdfs://
FilteredRDD
contains…
MappedRDD
split…

24. Parallel operators
• map, reduce
• sample, filter
• groupBy, reduceByKey
• join, leftOuterJoin, rightOuterJoin
• union, cross

25. What is really happening?
MLib
Shark
GraphX
Streaming
HDFS
Crunch Mahout
Pig
Sqoop Flume
Coordination and
workflow
management
Zookeeper
Command
Center
ASF Projects
FLOSS Projects
Pivotal Products
GemFire XD
Oozie
MapReduce
Hive
Tez
Giraph
Hadoop UI
Hue
SolrCloud
Phoenix
HBase
Spark
Impala
HAWQ
SpringXD
MADlib
Hamster
PivotalR
YARN
Tachyon

26. May be its not so bad
server 1
server N

27. But HDFS/YARN are safe?
HDFS, Ceph, S3, NAS, etc.
New
HDFS
New
YARN

28. Tachyon
• In-memory data-exchange layer
• A set of evolving APIs:
• filesystem
• caching
• RDDs
• Materialized views

29. Tachyon

30. Spark is best for cloud

31. It will be called Hadoop
MLib
Shark
GraphX
Streaming
HDFS
Crunch Mahout
Pig
Sqoop Flume
Coordination and
workflow
management
Zookeeper
Command
Center
ASF Projects
FLOSS Projects
Pivotal Products
GemFire with Tachyon
Oozie
MapReduce
Hive
Tez
Giraph
Hadoop UI
Hue
SolrCloud
Phoenix
HBase
Spark
Impala
HAWQ
SpringXD
MADlib
Hamster
PivotalR
YARN

32. Spark/Tachyon recap
• Is it “Big Data” (Yes)
• Is it “Hadoop” (No)
• It’s one of those “in memory” things, right (Yes)
• JVM, Java, Scala (All)
• Is it Real or just another shiny technology with
a long, but ultimately small tail (Yes and ?)

33. A NEW PLATFORM FOR A NEW
ERA

34. Questions ?