Apache Avro is a data serialization system and RPC framework that provides rich data structures and compact, fast binary data formats. It uses JSON for schemas and relies on schemas stored with data to allow serialization without per-value overhead. Avro supports Java, C, C++, C#, Python and Ruby and is commonly used in Hadoop for data persistence and communication between nodes. It compares similarly to Protobuf and Thrift but with some differences in implementation, error handling and extensibility.

1. Apache Avro
Zafar Gilani
Muhammad Adnan Khan
Hui Shang

2. Outline
• Overview
• Comparison
• Specification
• SASL profile and usage
• References

3. Overview
• A data serialization system.
• An RPC framework.
• For: storage & comm.
• Purpose:
– Provide rich data structures.
– A compact and fast binary data format.
– Simple integration with dynamic languages.

4. Overview
• Avro uses JSON for Interface Description
Language (IDL).
– To specify data types.
– To specify protocols.
• Review: JavaScript Object Notation is just a
light-weight text-based standard for data
interchange.

5. Why the need for Avro?
• Primary usage in Hadoop, provides standard:
1. Serialization format for persistent data.
2. Wire format for communication ..
• .. among Hadoop nodes.
• .. from client programs to Hadoop services.

6. Overview
• Avro relies on schemas.
– Schema stored with data.
– Each datum written with no per-value overheads.
• Thus serialization is fast and small.
• Avro in RPC:
– Schema exchange during client-server handshake.
– Correspondence in fields can be easily resolved.

7. APIs
• Supporting API for:
– Java
–C
– C++
– C#
– Python
– Ruby

8. Comparison with other systems
• Avro vs. Protobuf and Thrift.
• A quick note about Thrift:
– Initially developed at Facebook by a Google intern.
– Closer to Google’s protobuf.

9. Comparison with other systems
Avro Google protobuf Thrift
Implementation Hmm.. Cleaner  Hmm..
Error handling Complex Simple OK
Extensibility Hmm.. Richer OK
Compatibility Java, C, C++, C#, That and much About the same as
Python and Ruby more such as protobuf
Adobe Actionscript,
Microsoft
Silverlight, etc.

10. Specification
• Schema represented in one of:
– JSON string, naming a defined type.
– JSON object of the form:
• {"type": "typeName" ...attributes...}
– JSON array
• Primitive types: null, boolean, int, long, float,
double, bytes, string
– {"type": "string"}
• Complex types: records, enums, arrays, maps,
unions, fixed

11. Specification, example protocol
{
"namespace": "com.acme",
"protocol": "HelloWorld",
"doc": "Protocol Greetings",
"types": [
{"name": "Greeting", "type": "record", "fields": [
{"name": "message", "type": "string"}]},
{"name": "Curse", "type": "error", "fields": [
{"name": "message", "type": "string"}]}
],
"messages": {
"hello": {
"doc": "Say hello.",
"request": [{"name": "greeting", "type": "Greeting" }],
"response": "Greeting",
"errors": ["Curse"]
}
}
}

12. SASL profile
• Simple Authentication and Security Layer.
• Provides a framework for
– Authentication.
– Security of network protocols.

13. SASL usage
• Negotiation procedure to use connection-
oriented Avro RPC:
– 0: START Used in a client's initial message.
– 1: CONTINUE Used while negotiation is
ongoing.
– 2: FAIL Terminates negotiation unsuccessfully.
– 3: COMPLETE Terminates negotiation
sucessfully.

14. References
1. Apache Avro,
http://avro.apache.org/docs/current/
2. Google protocol buffers vs Apache Avro,
http://www.sammur.com/?p=36
3. Avro vs Thrift,
http://tech.puredanger.com/2011/05/27/serializ
ation-comparison/
4. SASL,
http://avro.apache.org/docs/current/sasl.html

15. Apache Avro
Zafar Gilani
Muhammad Adnan Khan
Hui Shang