Hadoop summit 2010 frameworks panel elephant bird

Hadoop Frameworks
Kevin Weil @kevinweil
Twitter

A framework for working with structured data within the Hadoop ecosystem
Elephant Bird

Protocol Buffers
Thrift
JSON
W3C Logs
Elephant Bird

InputFormats
OutputFormats
Hadoop Writables
Pig LoadFuncs
Pig StoreFuncs
Hbase LoadFuncs
Elephant Bird

A framework for working with structured data within the Hadoop ecosystem… plus:
LZO Compression
Code Generation
Hadoop Counter Utilities
Misc Pig UDFs
Elephant Bird

You should only need to specify the data schema
Why?

You should only need to specify the ( flexible, forward-backward compatible, self-documenting ) data schema
Why?

Everything else can be codegen’d.
Why?

Less Code. Efficient Storage. Focus on the Data.
Why?

Underlies 20,000 Hadoop jobs at Twitter every day.
Why?

Underlies 20,000 Hadoop jobs at Twitter every day.
http://github.com/kevinweil/elephant-bird : contributors welcome!
Why?

Hadoop summit 2010 frameworks panel elephant bird

by Kevin Weil on Jul 03, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 1

Statistics

Hadoop summit 2010 frameworks panel elephant bird — Presentation Transcript