Hadoop Frameworks Panel__HadoopSummit2010

Hadoop Frameworks and Tools Panel
Moderator: Sanjay Radia
Yahoo!

Hadoop Frameworks and Tools Panel
Pig - Alan Gates, Yahoo!
Hive - Ashish Thusoo, Facebook
Cascading – Chris K Wensel, Concurrent, Inc
Elephant Bird – Kevin Weil, Twitter
Voldemort – Jay Kreps, LInkedIn
Hue (Desktop), Philip Zeyliger, Cloudera

Format
Each framework/tool:
The problem space and target audience
Plans for the future enhancements
Questions and discussion

Pig
Alan F. Gates
Yahoo! [email_address] [email_address] [email_address]

Data pipelines
Live in the “data factory” where data is cleansed and transformed
Run at regular time intervals
Research
data is often short lived
data often semi-structured or unstructured
rapid prototyping
Target Use Cases

Turing complete
Add loops, branches, functions, modules
Will enable more complex pipelines, iterative computations, and cleaner programming
Workflow integration
What interfaces does Pig need to provide to enable better workflow integration?
Future Enhancements

What’s Missing: Table Management Service
What we have now
Hive has its own data catalog
Pig, Map Reduce can
Use a InputFormat or loader that knows the schema (e.g. ElephantBird)
Describe the schema in code A = load ‘foo’ as (x:int, y:float)
Still have to know where to read and write files themselves
Must write Loader, and SerDe to read new file type in Pig, and Hive
Workflow systems must poll HDFS to see when data is available

What We Want
Given an InputFormat and OutputFormat only need to write one piece of code to read/write data for all tools
Schema shared across tools
Disk location and storage format abstracted by service
Workflow notified of data availability by service
table mgmt service Pig Hive Map Reduce Streaming RCFile Sequence File Text File

Hive
Ashish Thusoo
Facebook

A system for managing, querying and analyzing structured data stored in Hadoop
Stores metadata in an RDBMS
Stores data in HDFS
Uses Map/Reduce for Computation
Hive – Brief Introduction

Easy to Use
Familiar Data Organization (Tables, Columns and Partitions)
SQL like language for querying this data
Easy to Extend
Interfaces to add User Defined Functions
Language constructs to embed user programs in the data flow
Flexible
Different storage formats
Support for user defined types
Hive – Core Principals

Transparent Optimizations
Optimizations for data skews
Different types of join and group by optimizations
Interoperable
JDBC and ODBC drivers
Thrift interfaces
Hive – Core Principals

Where we are?
Diverse user community
Where we want to be?
Diverse developer community
Hive – Major Future Goals

JDBC / MicroStrategy compatbility
Integration with PIG
Predicate Pushdown to HBase
Cost-based Optimizer
Quotas
Security/ACLs
Indexing
SQL compliance
Unstructured Data
Hive – Things to work on
Statistics
Archival (HAR)
HBase Integration
Improvements to Test Frameworks
Storage Handlers

Cascading
Chris K Wensel
Concurrent, Inc. http://cascading.org/ [email_address] @cwensel

An alternative API to MapReduce for assembling complex data processing applications
Provides implementations of all common MR patterns
Fail fast query planner and topological job scheduler
Integration is first class
Works with structured and unstructured data
Cascading

Cascading

Not a syntax (like PigLatin or SQL)
Allows developers to build tools on top
Cascalog – interactive query language (Backtype)
Bixo – scalable web-mining toolkit (Bixo Labs)
Cascading.JRuby – JRuby based DSL (Etsy)
More >> http://www.cascading.org/modules.html
Is complimentary to alternative tools
Riffle annotations to bridge the gap with Mahout
Cascading

Implement the simplest thing possible
Focus on the problem, not the system
ETL, processing, analytics become logical
Cascading

Log processing
Amazon CloudFront log analyzer
Machine Learning
Predicting flight delays (FlightCaster)
Behavioral ad-targeting
RazorFish (see case study on Amazon site)
Integration
With HBase (StumbleUpon), Hypertable (ZVents), & AsterData (ShareThis)
Social Media Analytics
BackType (see Cascalog)
Cascading – Common Uses

Ships with Karmasphere Studio
Runs great on Appistry CloudIQ Storage
Tested and optimized for Amazon Elastic MapReduce
Cascading - Compatibility

Elephant Bird
Kevin Weil @kevinweil
Twitter

A framework for working with structured data within the Hadoop ecosystem
Elephant Bird

Protocol Buffers
Thrift
JSON
W3C Logs
Elephant Bird

InputFormats
OutputFormats
Hadoop Writables
Pig LoadFuncs
Pig StoreFuncs
Hbase LoadFuncs
Elephant Bird

A framework for working with structured data within the Hadoop ecosystem… plus:
LZO Compression
Code Generation
Hadoop Counter Utilities
Misc Pig UDFs
Elephant Bird

You should only need to specify the data schema
Why?

You should only need to specify the ( flexible, forward-backward compatible, self-documenting ) data schema
Why?

Everything else can be codegen’d.
Why?

Less Code. Efficient Storage. Focus on the Data.
Why?

Underlies 20,000 Hadoop jobs at Twitter every day.
Why?

Underlies 20,000 Hadoop jobs at Twitter every day.
http://github.com/kevinweil/elephant-bird : contributors welcome!
Why?

Project Voldemort
Jay Kreps
LinkedIn

Key-value storage
No single point of failure
Focused on “live serving” not offline analysis
Excellent support for online/offline data cycle
Used for many parts of linkedin.com
Project Voldemort

Online/Offline architecture

Why online/offline split?

Project Voldemort: Hadoop integration
Three key-metrics to balance
Build time
Load time
Live request performance
Meet lots of other needs:
Atomic swap of data sets & rollback
Failover, checksums

Hue (formerly Cloudera Desktop)
Philip Zeyliger
[email_address] @philz42

3 What’s Hue?
a unified web-based UI for interacting with Hadoop
includes applications for looking at running jobs , launching jobs , browsing the file system, interacting with Hive
is an environment for building additional applications near the existing ones

3 Why Hue SDK?
Re-use components for talking to Hadoop
Re-use patterns for developing apps that talk to Hadoop
Centralize Hadoop usage through one interface

Open Source
Apache 2.0 licensed
http://github.com/cloudera/hue
Oh, by the way

Questions for the panel
What is missing in the overall space?
Questions from the audience

Hadoop Frameworks Panel__HadoopSummit2010

by Yahoo! Developer Network on Jun 30, 2010

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Hadoop Frameworks Panel__HadoopSummit2010 Presentation Transcript