Avro is a data serialization system that provides data interchange and interoperability between systems. It allows for efficient encoding of data and schema evolution. Avro defines data using JSON schemas and provides dynamic typing which allows data to be read without code generation. It includes a file format and supports MapReduce workflows. Avro aims to become the standard data format for Hadoop applications by providing rich data types, interoperability between languages, and compatibility between versions.

1. Introduction to
Apache Avro
Doug Cutting
21 July, 2010

2. Avro is...
● data serialization
● file format
● RPC format

3. Existing Serialization Systems:
Protocol Buffers & Thrift
● expressive
● efficient (small & fast)
● but not very dynamic
● cannot browse arbitrary data
● viewing a new datatype
– requires code generation & load
● writing a new datatype
– requires generating schema text
– plus code generation & load

4. Avro Serialization
● spec's a serialization format
● schema language is in JSON
● each lang already has JSON parser
● each lang implements data reader & writer
● in normal code
● code generation is optional
● sometimes useful in statically typed languages
● data is untagged
● schema required to read/write

5. Avro Schema Evolution
● writer's schema always provided to reader
● so reader can compare:
● the schema used to write with
● the schema expected by application
● fields that match (name & type) are read
● fields written that don't match are skipped
● expected fields not written can be identified
● same features as provided by numeric field ids

6. Avro JSON Schemas
// a simple three-element record
{"name": "Block", "type": "record":,
"fields": [
{"name": "id", "type": "string"},
{"name": "length", "type": "integer"},
{"name": "hosts", "type":
{"type": "array:, "items": "string"}}
]
}
// a linked list of strings or ints
{"name": "MyList", "type": "record":,
"fields": [
{"name": "value", "type": ["string", "int"]},
{"name": "next", "type": ["MyList", "null"]}
]
}

7. Avro IDL Schemas
// a simple three-element record
record Block {
string id;
int length;
array<string> hosts;
}
// a linked list of strings or ints
record MyList {
union {string, int} value;
MyList next;
}

8. Hadoop Data Formats
● Today, primarily
● text
– pro: inter­operable
– con: not expressive, inefficient
● Java Writable
– pro: expressive, efficient
– con: platform­specific, fragile

9. Avro Data
● expressive
● small & fast
● dynamic
● schema stored with data
– but factored out of instances
● APIs permit reading & creating
– new datatypes without generating & loading code

10. Avro Data
● includes a file format
● replacement for SequenceFile
● includes a textual encoding
● handles versioning
● if schema changes
● can still process data
● hope Hadoop apps will
● upgrade from text; standardize on Avro for data

11. Avro MapReduce API
● Single-valued inputs and outputs
● key/value pairs only required for intermediate
● map(IN, Collector<OUT>)
● map-only jobs never need to create k/v pairs
● map(IN, Collector<Pair<K,V>>)
● reduce(K, Iterable<V>, Collector<OUT>)
● if IN and OUT are pairs, default is sort
● In Avro trunk today, built on Hadoop 0.20 APIs.
● in Avro1.4.0 release next month

12. Avro MapReduce Example
public void map(Utf8 text, AvroCollector<Pair<Utf8,Long>> c,
Reporter r) throws IOException {
StringTokenizer i = new StringTokenizer(text.toString());
while (i.hasMoreTokens())
c.collect(new Pair<Utf8,Long>(new Utf8(i.nextToken()), 1L));
}
public void reduce(Utf8 word, Iterable<Long> counts,
AvroCollector<Pair<Utf8,Long>> c,
Reporter r) throws IOException {
long sum = 0;
for (long count : counts)
sum += count;
c.collect(new Pair<Utf8,Long>(word, sum));
}

13. Avro RPC
● leverage versioning support
● permit different versions of services to interoperate
● for Hadoop, will
● let apps talk to clusters running different versions
● provide cross­language access

14. Avro IDL Protocol
@namespace("org.apache.avro.test")
protocol HelloWorld {
record Greeting {
string who;
string what;
}
Greeting hello(Greeting greeting);
}

15. Avro Status
● Current
● C, C++, Java, Python & Ruby APIs
● Interoperable RPC and data
● Mapreduce API for Java
● Upcoming
● MapReduce APIs for other languages
– efficient, rich data
● RPC used in Flume, Hbase, Cassandra, Hadoop, etc.
– inter­version compatibility
– non­Java clients

16. Questions?