Apache Avro and You

Eric Wendelin Hadooper @returnpath Blog: eriwen.com Twitter: @eriwen

Your data and Hadoop • Traditional Hadoop consumes 2 basic types of input: • Plain text - Cross-platform but inefﬁcient • Java Writables - Efﬁcient but tied to Java platform

Better: Cross-platform data serialization

What about Thrift? • Thrift is expressive and efﬁcient but not dynamic: • Reading/Writing datasets requires code generation and load • Can’t browse arbitrary data

Better than Thrift (in most cases) • At least as efﬁcient as Thrift and Protocol Buffers • Dynamic! • Schema stored with data • Arbitrary data types • All without generating and loading code

Avro is... • A data serialization system • Focused on dynamic access, platform independence and simple schema evolution

Major parts of Avro • Schema language (JSON) • RPC framework • APIs for Python (dynamic), Java (generic and speciﬁc), Ruby, C, C++ and more!

Example Schema { ‘type’: ‘record’, ‘name’:‘Widget’, ‘fields’: [ {‘name’:‘size’,‘type’:‘long’, ‘default’:0}, {‘name’:‘density’,‘type’:‘float’} ] }

Avro RPC • Cross-language access • Permits different versions of services to interoperate

Avro Data • Compressible • Splittable (but that doesn’t mean ﬁles are easy to combine) • Arbitrary metadata • Can include data with different schemas in the same ﬁle and detect dynamically!

Why do you care about Avro?

Future: Hadoop’s RPC mechanism to be replaced by Avro

Using Avro in Hadoop Traditional Class ➡Avro Class Job ➡AvroJob TextInputFormat ➡AvroInputFormat Mapper<K1,V1,K2,V2> ➡AvroMapper<IN,OUT> Reducer<K2,V2,K3,V3> ➡AvroReducer<K,V,OUT>

Future of Avro • Version 1.4 (coming soon) will include: • Classes for Avro input/output • MapReduce connector protocol • PHP API

Apache Avro and You

by Eric Wendelin

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 40

Statistics

Apache Avro and You Presentation Transcript

http://www.sozialpapier.com	37
http://sozialpapier.com	3