An Introduction to Apache Avro

6,313 views

Published on

An Introduction to Apache Avro, what is it and how is it used ?
Plus some code examples to show JSON etc.

Published in: Technology, Business
1 Comment
2 Likes
Statistics
Notes
  • http://www.dbmanagement.info/Tutorials/Hadoop.htm #Hadoop #Avro #Cassandro #Drill #Flume Tutorial (Videos and Books)at $7.95
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
6,313
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
79
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide

An Introduction to Apache Avro

  1. 1. Apache Avro ● What is it ? ● Language API's ● JSON Example ● Schema Evolution ● Avro Use Example www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  2. 2. Avro – What is it ? ● Data serialization system ● Uses JSON based schemas ● Uses RPC calls to send data ● Schema's sent during data exchange ● Integrated with many languages ● Fast binary data format or encode with JSON www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  3. 3. Avro – Language API's Avro API's exist for the following languages ● Java ● C# ● C ● C++ ● Python ● Ruby www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  4. 4. Avro – JSON Example This text shows an example JSON schema for a customer record { "namespace": "customer.avro", "type": "record", "name": "customer", "fields": [ {"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "address", "type": "string"}, {"name": "phone", "type": "string"}, {"name": "email", "type": "string"}, ] } www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  5. 5. Avro – Schema Evolution In a fast changing environment it is possible to ● Write data to file with a schema ● Change the schema – Add extra fields – Delete fields – Rename fields ● And still read written file with changed schema www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  6. 6. Avro – Use Example // define schema for customer { "namespace": "customer.avro", "type": "record", "name": "Customer", "fields": [ {"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "address", "type": "string"}, {"name": "phone", "type": "string"}, {"name": "email", "type": "string"}, ] } // compile the schema java -jar /....../avro-tools-1.7.4.jar compile schema customer.avsc . // create a couple of customers Customer cust1 = new Customer("Dave", 33, "12 redmont street", "021234675", "davef@somedom@co.nz" ); Customer cust2 = new Customer("Emily", 27, "13a lake cresent", "022876234", "esuanders@somenz@co.nz" ); // now serialize customers to disk File file = new File("customers.avro"); DatumWriter<Customer> userDatumWriter = new SpecificDatumWriter<Customer>(Customer.class); DataFileWriter<Customer> dataFileWriter = new DataFileWriter<Customer>(customerDatumWriter); dataFileWriter.create(cust1.getSchema(), new File("customers.avro")); dataFileWriter.append(cust1); dataFileWriter.append(cust2); dataFileWriter.close(); www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  7. 7. Contact Us ● Feel free to contact us at – www.semtech-solutions.co.nz – info@semtech-solutions.co.nz ● We offer IT project consultancy ● We are happy to hear about your problems ● You can just pay for those hours that you need ● To solve your problems

×