Your SlideShare is downloading. ×
0
3 apache-avro
3 apache-avro
3 apache-avro
3 apache-avro
3 apache-avro
3 apache-avro
3 apache-avro
3 apache-avro
3 apache-avro
3 apache-avro
3 apache-avro
3 apache-avro
3 apache-avro
3 apache-avro
3 apache-avro
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

3 apache-avro

7,817

Published on

Published in: Technology
0 Comments
17 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,817
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
87
Comments
0
Likes
17
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Apache Avro Zafar GilaniMuhammad Adnan Khan Hui Shang
  • 2. Outline• Overview• Comparison• Specification• SASL profile and usage• References
  • 3. Overview• A data serialization system.• An RPC framework.• For: storage & comm.• Purpose: – Provide rich data structures. – A compact and fast binary data format. – Simple integration with dynamic languages.
  • 4. Overview• Avro uses JSON for Interface Description Language (IDL). – To specify data types. – To specify protocols.• Review: JavaScript Object Notation is just a light-weight text-based standard for data interchange.
  • 5. Why the need for Avro?• Primary usage in Hadoop, provides standard: 1. Serialization format for persistent data. 2. Wire format for communication .. • .. among Hadoop nodes. • .. from client programs to Hadoop services.
  • 6. Overview• Avro relies on schemas. – Schema stored with data. – Each datum written with no per-value overheads. • Thus serialization is fast and small.• Avro in RPC: – Schema exchange during client-server handshake. – Correspondence in fields can be easily resolved.
  • 7. APIs• Supporting API for: – Java –C – C++ – C# – Python – Ruby
  • 8. Comparison with other systems• Avro vs. Protobuf and Thrift.• A quick note about Thrift: – Initially developed at Facebook by a Google intern. – Closer to Google’s protobuf.
  • 9. Comparison with other systems Avro Google protobuf ThriftImplementation Hmm.. Cleaner  Hmm..Error handling Complex Simple OKExtensibility Hmm.. Richer OKCompatibility Java, C, C++, C#, That and much About the same as Python and Ruby more such as protobuf Adobe Actionscript, Microsoft Silverlight, etc.
  • 10. Specification• Schema represented in one of: – JSON string, naming a defined type. – JSON object of the form: • {"type": "typeName" ...attributes...} – JSON array• Primitive types: null, boolean, int, long, float, double, bytes, string – {"type": "string"}• Complex types: records, enums, arrays, maps, unions, fixed
  • 11. Specification, example protocol{ "namespace": "com.acme", "protocol": "HelloWorld", "doc": "Protocol Greetings", "types": [ {"name": "Greeting", "type": "record", "fields": [ {"name": "message", "type": "string"}]}, {"name": "Curse", "type": "error", "fields": [ {"name": "message", "type": "string"}]} ], "messages": { "hello": { "doc": "Say hello.", "request": [{"name": "greeting", "type": "Greeting" }], "response": "Greeting", "errors": ["Curse"] } }}
  • 12. SASL profile• Simple Authentication and Security Layer.• Provides a framework for – Authentication. – Security of network protocols.
  • 13. SASL usage• Negotiation procedure to use connection- oriented Avro RPC: – 0: START Used in a clients initial message. – 1: CONTINUE Used while negotiation is ongoing. – 2: FAIL Terminates negotiation unsuccessfully. – 3: COMPLETE Terminates negotiation sucessfully.
  • 14. References1. Apache Avro, http://avro.apache.org/docs/current/2. Google protocol buffers vs Apache Avro, http://www.sammur.com/?p=363. Avro vs Thrift, http://tech.puredanger.com/2011/05/27/serializ ation-comparison/4. SASL, http://avro.apache.org/docs/current/sasl.html
  • 15. Apache Avro Zafar GilaniMuhammad Adnan Khan Hui Shang

×