11. Tez
• Hive and Pig can use Tez since version 0.13
• 2-3x performance increase compared to older Hive
and Pig versions
• Tez does performance optimisations and resource
management across the cluster
• Reuses containers and JVMs: effective for short
queries in e.g. Hive.
• Multiple jobs at the same time
14. BlinkDB
• Offline sampling module
• Compute data samples, based on a ‘storage budget’
• Store samples on disk and in memory
• Sample selection module
• Select the right samples for an incoming query
• Query execution in parallel
• Answers are augmented by error and confidence bounds
15. BlinkDB
• BlinkDB has been demonstrated live at VLDB 2012
on a 100 node Amazon EC2 cluster answering a
range of queries on 17 TBs of data in less than 2
seconds (over 200x faster than Hive), within an
error of 2-10%.
16. SummingBird
• Write MapReduce programs that look like native
Java or Scala collection transformations
• Platform-agnostic
• Execute on a number of distributed MapReduce
platforms, like Scalding (Hadoop) or Storm
• The same code can run for batch and streaming
19. Storm (on YARN)
Stream data processing on Hadoop.
Storm recap:
• Processes unbounded streams of tuples.
• Basic primivitives are Spout's and Bolt's
• A spout is a source of streams.
• A bolt processes streams and may emit new streams
24. Mahout
• A scalable machine learning library
The Mahout community decided to move its codebase onto […] systems
that offer a richer programming model and more efficient execution than
Hadoop MapReduce.
!
Mahout will therefore reject new MapReduce algorithm implementations
from now on.
!
We are building our future implementations on top of a DSL […].
Programs written in this DSL are automatically optimized and executed in
parallel on Apache Spark.
https://mahout.apache.org/
